Build a Resilient Streaming Platform with FFMPEG

The AI avatar froze mid-sentence. Audio kept playing, but the face on screen was stuck — mouth half-open, eyes locked in an uncanny stare. I checked the logs. No crash. No error. The Python process was still running, top showed CPU usage, and FFmpeg hadn't exited. The system was alive — and completely broken.

The frustrating part? This wasn't a bug in the AI model or the rendering pipeline. It was a deadlock between two named pipes. The audio pipe had filled up, os.write() blocked, and the video thread starved. FFmpeg, waiting for video frames that would never arrive, simply stopped muxing. The stream went silent while every process reported "healthy."

I reproduced the failure across 12-hour streaming sessions, traced it through select.select polling, fcntl buffer tuning, and eventually a full failover architecture. What started as a simple os.write() loop turned into a self-healing system that runs for weeks without intervention.

In this post, I'll walk you through each failure mode I hit, why the obvious fix created the next problem, and how to build an FFmpeg ingestion pipeline that detects its own failures and recovers in under 3 seconds.

1. Starting with the simplest thing that could work

I had audio and video data flowing from an AI pipeline — PCM audio chunks and H.264 encoded frames — and I needed to get them into an RTMP stream. The obvious path: named pipes.

audio_pipe_fp = "/tmp/audio_pipe"
video_pipe_fp = "/tmp/video_pipe"

audio_chunk = b'\x00' * 4096  # Simulated PCM data
video_frame = b'\xFF' * 10000 # Simulated Encoded Frame

if not os.path.exists(audio_pipe_fp):
    os.mkfifo(audio_pipe_fp)
if not os.path.exists(video_pipe_fp):
    os.mkfifo(video_pipe_fp)

audio_fd = os.open(audio_pipe_fp, os.O_RDWR)
video_fd = os.open(video_pipe_fp, os.O_RDWR)

while True:
    os.write(audio_fd, audio_chunk)
    os.write(video_fd, video_frame)
    time.sleep(0.04)  # 25 fps

Paired with a straightforward FFmpeg command:

ffmpeg -re -f s16le -ar 44100 -ac 2 -i /tmp/audio_pipe \
       -f h264 -i /tmp/video_pipe \
       -c:v copy -c:a aac -f flv rtmp://localhost/live/stream

Named pipes were the right call. Unlike physical files, they live entirely in RAM — no disk I/O bottleneck, no filesystem overhead. The OS treats them as files, but data moves at memory speed. For a real-time streaming system pushing 25 FPS with synchronized audio, this was essential.

The blocking I/O also seemed like a feature, not a bug. If the pipe buffer filled up, os.write() would hang until FFmpeg consumed enough data to free space. This created natural backpressure — the producer couldn't outrun the consumer. On paper, it was elegant.

It ran fine for about 20 minutes. Then the avatar froze.

2. The deadlock I didn't see coming

The failure mode was subtle. FFmpeg's muxer needs interleaved audio and video to produce a valid FLV stream. It reads from both pipes in a loop, expecting data to be available on each. But my Python script wrote audio first, then video — sequentially, in a single thread.

Here's what happened under load: the audio pipe's 64KB buffer filled up. os.write(audio_fd, audio_chunk) blocked. The entire Python process froze on that line. Meanwhile, FFmpeg had already consumed the last video frame and was now waiting for the next one — but the Python process couldn't deliver it because it was stuck waiting on audio. FFmpeg couldn't consume more audio because it was waiting for video to continue muxing. Circular deadlock.

No process crashed. No error was logged. The system just... stopped.

The fix was obvious once I saw it: audio and video writes had to be independent. If one pipe blocked, the other had to keep feeding data. But decoupling them created its own set of problems.

3. Decoupling audio and video with threads

I split the write operations into two dedicated threads, each owning its own pipe:

class PipeWriter(threading.Thread):
    def __init__(self, path):
        super().__init__()
        self.path = path
        self.q = Queue(maxsize=100)
        self.running = True
        self.fd = None

    def run(self):
        self.fd = os.open(self.path, os.O_WRONLY)
        while self.running:
            try:
                data = self.q.get(timeout=0.04)

                while True:
                    _, wlist, _ = select.select([], [self.fd], [], 0.01)
                    if not wlist:
                        continue
                    os.write(self.fd, data)
                    break

            except Empty:
                continue
            except BrokenPipeError:
                break

        if self.fd:
            os.close(self.fd)

    def write(self, data):
        while True:
            try:
                self.q.put(data, timeout=0.04)
                break
            except Full:
                continue

    def stop(self):
        self.running = False


audio_writer = PipeWriter(audio_pipe_fp)
video_writer = PipeWriter(video_pipe_fp)

audio_writer.start()
video_writer.start()

while True:
    audio_writer.write(audio_chunk)
    video_writer.write(video_frame)

Two things matter here beyond just threading. First, the internal Queue(maxsize=100) — without it, a fast producer could flood RAM with queued frames that FFmpeg hasn't consumed yet. The queue cap creates a second layer of backpressure at the application level.

Second, select.select([], [self.fd], [], 0.01). Instead of blindly calling os.write() and hoping the pipe is ready, this polls the file descriptor for writability. If the pipe isn't ready, it loops and checks again 10ms later. This eliminates the guesswork — we only write when the OS confirms the pipe can accept data.

The deadlock was gone. The system ran smoothly for hours. I thought I'd solved it.

Then, around the 8-hour mark, the stream dropped. The logs showed BrokenPipeError. The audio pipe had saturated and collapsed.

4. Eight hours wasn't enough — tuning the pipe buffer

The audio pipe was the repeat offender. Audio data is small (4KB chunks) but frequent (44100 samples/sec across 2 channels), so the write rate is relentless. The default Linux pipe buffer is 64KB. At that size, the audio pipe lived in a perpetually full state — zero margin for even a momentary consumption delay.

The fix was a single fcntl call:

import fcntl

F_SETPIPE_SZ = 1031
size = 1024 * 1024  # 1 MB
try:
    fcntl.fcntl(fd, F_SETPIPE_SZ, size)
except OSError as e:
    print(f"Could not set pipe size: {e}")

F_SETPIPE_SZ (constant 1031) is a Linux-specific fcntl operation that resizes a pipe's kernel buffer. The maximum allowed value is controlled by /proc/sys/fs/pipe-max-size, but 1MB is well within default limits on most systems.

The result was dramatic. With the default 64KB buffer, continuous uptime capped at 8–9 hours before a pipe broke. With 1MB, the same system ran for 5–7 days without intervention. A 16x buffer increase bought roughly 15x the uptime.

But "5–7 days" is not "forever." Pipe exhaustion was still inevitable — just deferred. I needed the system to survive it when it happened.

5. Designing for inevitable failure

At this point I accepted a hard truth: in a long-running streaming system, pipes will eventually break. Network jitter, kernel scheduling quirks, memory pressure — something will tip the balance. The goal shifted from "prevent all failures" to "survive any failure."

The architecture needed two capabilities:

• Health monitoring: continuously validate that both pipes are connected and the FFmpeg process is alive.
• Self-healing: when a failure is detected, tear down the broken pipes and FFmpeg instance, then recreate them — all without human intervention.

Here's the implementation:

def health_check(self):
    if not self.process:
        return False

    if self.video_fifo_fp is None:
        return False

    if self.audio_fifo_fp is None:
        return False

    return_code = self.process.poll()
    if return_code is not None:
        return False

    return True

def failover(primary_instance, standby_instance=None):
    primary_instance.terminate()

    if standby_instance is None:
        primary_instance = create_instance()
    elif standby_instance.health_check():
        primary_instance = standby_instance
    else:
        primary_instance = create_instance()
        standby_instance.terminate()

    return primary_instance, standby_instance

The health check is deliberately simple: verify the process exists, verify both pipe paths are set, and poll the FFmpeg subprocess for its return code. A non-None return code means FFmpeg exited — crashed, killed by OOM, or terminated by a broken pipe. Any of these triggers failover.

The failover routine terminates the dead instance, creates a fresh one with new pipes, and returns it. The downtime is roughly 2–3 seconds — the time it takes to os.mkfifo() new pipes and spawn a new FFmpeg process.

I considered a hot standby: keep a second FFmpeg instance running with pre-opened pipes, ready to take over instantly. That would cut downtime to 5–10 frames. But a standby instance consumes GPU memory, CPU cycles, and encoder context — resources that are scarce in a real-time AI streaming pipeline. Given that pipe failures occurred roughly once a week after the buffer tuning, the 2–3 second cold restart was the right tradeoff. Resource efficiency over sub-second recovery.

6. What I'd do differently

The system now runs for weeks unattended. When a pipe breaks, the health check catches it within one polling cycle, the failover fires, and the stream resumes before most viewers notice. But there are edges I'd sharpen in a v2:

Per-pipe health granularity. The current health check is binary — any failure triggers a full teardown. But if only the audio pipe breaks while the video pipe and FFmpeg process are healthy, I'm destroying a working video connection unnecessarily. A finer-grained check that identifies which pipe failed would let me recreate just that pipe and re-attach it to the running FFmpeg instance.

Gradual degradation instead of hard cut. The 2–3 second recovery is a hard drop — the stream goes black, then reconnects. A smoother approach would buffer the last valid frame server-side and display it during the recovery window, so viewers see a still image instead of a disconnect.

Metrics exposure. Right now, failures are logged but not measured. I'd add Prometheus counters for pipe break events, failover triggers, and recovery duration. Without metrics, I can't answer "is this getting worse over time?" — and in production, that's the question that matters.

7. Conclusion

The journey from a naive os.write() loop to a self-healing streaming pipeline taught me something broader: in systems that must run continuously, the interesting problems aren't the ones you anticipate. They're the ones created by your last fix.

Threading solved the deadlock but exposed pipe saturation. Buffer tuning deferred saturation but didn't eliminate it. Failover made failure survivable but introduced a recovery gap. Each solution was correct — and each one revealed the next problem.

If you're building an FFmpeg ingestion pipeline, start with the failover architecture. Don't wait until you've debugged the deadlock and tuned the buffers and hit the 8-hour wall. Build the health check and the recovery routine first, then optimize. A system that knows how to restart itself is worth more than one that runs perfectly until it doesn't.