When “working” wasn't good enough anymore.

Different stages of the pipeline did not always finish at the same rate.

If one stage blocked another directly, a temporary slowdown upstream could propagate through the entire pipeline and affect the streamed output.

I introduced multithreading and queues to decouple processing stages. Instead of:

Stage A → Stage B → Stage C

the pipeline could buffer work between stages:

Stage A → Queue → Stage B → Queue → Stage C

The goal was to prevent local processing delays from immediately becoming streaming interruptions.

Queues provided a buffer between components with different processing speeds, while threads allowed independent work to continue concurrently.

This improved decoupling but introduced concurrency complexity.

The system now had to deal with queue growth, synchronization, thread lifecycle, and backpressure.

The production pipeline uses RabbitMQ, multithreading, and queues to separate processing stages — LLM, TTS, video generation, and streaming operate without any stage blocking the entire pipeline.

The system could generate video frames, but raw frame production did not guarantee a stable streaming rate.

Temporary variations in processing speed could cause output FPS to fluctuate — directly affecting the viewing experience.

I introduced an FPS stabilizer and treated approximately 25 FPS as an end-to-end system constraint rather than optimizing each model independently.

The relevant metric was not “How fast can this model generate?”

It was: how consistently can the entire pipeline deliver frames to the stream?

This changed the optimization target from individual component performance to end-to-end runtime behavior.

Maintaining a stable output rate requires buffering and synchronization.

Too much buffering increases latency; too little increases the risk of frame starvation. The system had to balance stability against real-time latency.

The streaming pipeline was designed around ~25 FPS with runtime monitoring of frame delivery. A warning threshold around 22.5 FPS identified degradation before the stream became visibly unstable.

FFmpeg was part of the long-running streaming path.

A failure in the process or FIFO communication could interrupt the stream even though upstream AI components were still operating. A manual restart was not acceptable.

I treated FFmpeg as a component that could fail and built an automatic recovery path around it:

• Process health checks
• FIFO write timeouts
• Process termination
• Automatic restart
• Controlled reinitialization

The goal was not to assume FFmpeg would always remain healthy.

The goal was to make failure recoverable without human intervention — critical for a long-running stream where a small component failure can otherwise end the entire experience.

Automatic restart adds recovery complexity and can introduce a short interruption.

Restarting too aggressively can create restart loops, so the system needed health checks and controlled restart behavior rather than blindly restarting on every error.

The failover mechanism achieved ~2 seconds average recovery time for streaming-process failures, and the system reached ~99.9% uptime.

After stabilizing the pipeline, resource consumption remained high.

The system consisted of multiple AI and video-processing components, so optimizing a single model would not necessarily solve the overall resource problem.

I profiled resource usage across the pipeline and looked for waste at the runtime and system level — memory usage, process behavior, initialization, and how components interacted during execution.

The bottleneck wasn't necessarily where the largest model was.

Memory could be consumed by duplicated data, unnecessary buffers, processes alive longer than needed, or inefficient runtime behavior. Profiling made it possible to optimize the actual resource path rather than assumptions.

Deeper optimization requires more engineering effort and can increase implementation complexity.

Some optimizations make the system more specialized, which may reduce flexibility when requirements change.

The optimization work reduced overall resource usage by approximately 75% — through changes across the runtime pipeline rather than replacing one model with another.

Navigator V2 supported around 30 actions, but its runtime footprint grew significantly as the number of actions increased.

The initialization path also introduced unnecessary resource consumption and startup overhead.

I optimized how actions and their associated resources were initialized and retained during execution — reducing unnecessary VRAM/RAM consumption and avoiding work that did not need to remain resident.

The system did not need every resource to be fully active at all times.

Reducing the state held in memory could lower the runtime footprint without removing the underlying capabilities.

More aggressive resource management increases runtime coordination and may introduce additional loading behavior.

The optimization had to preserve existing action capabilities while reducing their memory footprint.

Across 30 actions, the optimized implementation achieved ~79% reduction in VRAM/RAM and ~63% reduction in initialization time.