Build a High-Performance Asynchronous Video Reader/Writer

I was building an AI-powered news video generation system. The pipeline read a video, ran inference on each frame, composited overlays, and wrote the result back out. During prototyping, everything ran smoothly at a steady 30 FPS.

Then I wired in the first object detection model. Single digits.

The frustrating part? It wasn't the model. When I profiled the pipeline, the inference step was fast enough. The bottleneck was sitting in two lines I'd copy-pasted from a dozen OpenCV tutorials and never thought twice about: cap.read() and out.write(). Together they were eating nearly 18 milliseconds per frame — more than half the 33.3ms budget needed for 30 FPS. Here's the thing most tutorials won't tell you: VideoCapture.read() is a blocking call. Your CPU is idle while your code waits for the next frame to decode.

I reproduced the issue across different video codecs and resolutions, and the pattern held: I/O, not computation, was the real throttle. By the end of this post, you'll know how to profile your own video pipeline, why caching frames to disk trades one bottleneck for another, and how to build a threaded async reader/writer that hides I/O latency entirely — without touching your inference code.

A pipeline that worked — until it didn't

The standard OpenCV video loop is so ubiquitous it's practically invisible:

import cv2

cap = cv2.VideoCapture("input.mp4")

if not cap.isOpened():
    exit()

frame_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
frame_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = cap.get(cv2.CAP_PROP_FPS)

fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter("output.mp4", fourcc, fps, (frame_width, frame_height))

while True:
    ret, frame = cap.read()

    if not ret:
        break

    out.write(frame)

    if cv2.waitKey(1) == ord('q'):
        break

cap.release()
out.release()
cv2.destroyAllWindows()

For simple playback or format conversion, this works fine. But I needed to know exactly how much time it was costing me. I wrapped each operation with timing instrumentation and ran a 30-second Full HD (1920×1080) clip through the loop:

Operation	Average Time (ms)	Theoretical FPS (Max)
cap.read()	~4.96 ms	~201 FPS
out.write(frame)	~12.74 ms	~78 FPS

Combined, a single frame pass takes roughly 17.7 ms — a ceiling of about 56 FPS with zero processing. With a 30 FPS target, that left me with 15.6 ms per frame for everything else: inference, compositing, rendering. A single pass through a lightweight YOLO model can easily consume 10–20 ms on CPU. The math wasn't going to work.

At this point I didn't know that my next attempt would solve the speed problem while creating one that was arguably worse.

The NumPy detour

If decoding a video is slow, why decode it at all? The video was reused across multiple pipeline stages. Re-reading from disk every time was wasteful. What if I extracted all frames once, stored them in a fast-to-read format, and worked from that cache?

NumPy's .npy format seemed perfect: no codec overhead, no compression artifacts, just raw np.ndarray data on disk.

intermediate_folder = "intermediate_frames"
os.makedirs(intermediate_folder, exist_ok=True)

i = 0
while True:
    ret, frame = cap.read()

    if not ret:
        break

    np.save(os.path.join(intermediate_folder, f"{i}.npy"), frame)
    i += 1

    if cv2.waitKey(1) == ord('q'):
        break

Loading was equally straightforward: np.load(f"{i}.npy") returned a frame in ~1.5 ms. A dramatic improvement. The numbers backed it up:

Operation	Average Time (ms)	Theoretical FPS (Max)
np.load()	~1.54 ms	~649 FPS
np.save()	~2.47 ms	~404 FPS

I had dropped per-frame I/O from 17.7 ms to roughly 4 ms. Plenty of headroom for inference.

Then I checked the disk.

Operation	Storage (MB)
cv2.VideoCapture() + cv2.VideoWriter()	~35 MB
np.save() + np.load()	~5300 MB

Five-point-three gigabytes. For a thirty-second video. That's over 150× the storage. A compressed H.264 video with lossy encoding stores far less data than uncompressed raw pixel arrays. I'd traded a speed problem for a storage one, and at this rate, a typical 10-minute news segment would need over 100 GB. The NumPy approach was fast, but it didn't scale.

That's when I realized I'd been asking the wrong question. I didn't need faster I/O. I needed I/O that didn't block computation.

The real bottleneck isn't speed — it's blocking

Here's what actually happens inside a synchronous video loop: the main thread calls cap.read(), the OS fetches the encoded frame from disk, the codec decodes it, and only then does execution return to your code. During all of that, your thread is frozen. It can't process the previous frame. It can't prepare the next output. It just waits.

The same thing happens on the write side: out.write() blocks while the codec encodes the frame and flushes it to disk. Your thread is idle while I/O runs.

The fix was conceptually simple: move I/O out of the main thread. If reading and writing happen on background threads, the main thread can keep processing frames without ever waiting on the disk. A producer–consumer pattern:

• A background reader thread continuously calls cap.read() and pushes frames into a queue.Queue
• The main thread pulls frames from the queue and processes them
• A background writer thread pulls processed frames from a second queue and calls out.write()

I/O latency is hidden behind computation. As long as the queue stays non-empty (for reading) or non-full (for writing), the main thread never blocks.

Building the async reader

Here's the full implementation. I'll walk through it piece by piece because the design decisions matter.

import queue
import threading
from typing import Tuple, Optional
import cv2
import numpy as np


class AsyncReader:
    def __init__(self, video_path: str, queue_size: int = 100):
        self._video_path = video_path

        self.__queue_size = queue_size
        self.__reader_queue: Optional[queue.Queue[Optional[np.ndarray]]] = None
        self.__read_thread: Optional[threading.Thread] = None
        self.__stop: Optional[bool] = None
        self.__reader: Optional[cv2.VideoCapture] = None

    def init_reader(self):
        if self.__reader is None:
            self.__reader = cv2.VideoCapture(self._video_path)

        self._init_queue()

    def release_reader(self):
        if self.__reader is not None:
            self.__reader.release()
            self.__reader = None

        self._release_queue()

    def _next(self) -> Tuple[bool, Optional[np.ndarray]]:
        try:
            ret, frame = self.__reader.read()
            if not ret:
                return False, None
            return True, frame
        except Exception as e:
            return False, None

    def read(self) -> Tuple[bool, Optional[np.ndarray]]:
        frame = self.__get_from_queue()
        return frame is not None, frame

    def _init_queue(self):
        if self.__reader_queue is None:
            self.__reader_queue = queue.Queue(maxsize=self.__queue_size)
        if self.__read_thread is None:
            self.__read_thread = threading.Thread(target=self.__put_to_queue, daemon=True)
            self.__read_thread.start()
        if self.__stop is None:
            self.__stop = False

    def _release_queue(self):
        if self.__read_thread is not None:
            self.__read_thread.join()
            self.__read_thread = None
        if self.__reader_queue is not None:
            self.__reader_queue.empty()
            self.__reader_queue = None

    def __put_to_queue(self):
        while True:
            ret, frame = self._next()
            frame = frame if ret else None
            self.__reader_queue.put(frame)
            if not ret:
                break

    def __get_from_queue(self) -> np.ndarray:
        frame = self.__reader_queue.get()
        return frame

The producer–consumer design

Three components work together:

1. _next() — the low-level frame reader. Calls cv2.VideoCapture.read(), which blocks. This method runs only on the background thread, never from the main thread.

2. __put_to_queue() — the producer loop. The background thread runs this target: read a frame from the video, push it into self.__reader_queue, repeat. When the video ends, it pushes None as a sentinel to signal consumers.

3. read() — the consumer API. From the caller's perspective, it looks identical to cv2.VideoCapture.read(), but it pulls from the in-memory queue instead of blocking on disk I/O. If frames are already buffered, it returns immediately.

Queue as backpressure

The queue_size=100 parameter isn't arbitrary. queue.Queue(maxsize=100) means that once 100 unread frames are buffered, queue.put() blocks until the consumer catches up. This prevents the background thread from reading the entire video into memory at startup. For a Full HD video, 100 frames at 1920×1080×3 bytes ≈ 600 MB. Bumping that to 1000 would eat 6 GB of RAM. The queue size is a deliberate trade-off between throughput and memory.

Initialization ordering matters

Look closely at init_reader(): the VideoCapture is opened before _init_queue() starts the background thread. If I reversed these — start the thread first, then open the video — the background thread would call _next() on a None reader and crash immediately. The ordering prevents a race condition.

The sentinel pattern

When the video ends, __put_to_queue() pushes None into the queue instead of a frame. The consumer calls read(), which receives None, and frame is not None evaluates to False — the same (ret, frame) API the caller expects. No special-case handling needed at the call site.

Threading pitfalls: what broke when I ran it

The design was clean on paper. The code compiled. So I ran the pipeline — and it deadlocked.

Deadlock on shutdown

The bug was in release_reader(). When closing the pipeline, I called release_reader() while the background thread was blocked on self.__reader_queue.put(frame) — the queue was full because no consumer had started draining yet. The release_reader() method called self.__read_thread.join(), which waited for the background thread to finish. But the background thread couldn't finish because it was blocked on put(). Classic deadlock.

The fix: signal the thread to stop before joining. I added a timeout to queue.put() and checked the stop flag. But there's a subtler issue: Python's queue.Queue.get() with no timeout blocks forever. If the producer crashes without sending a sentinel, the consumer hangs. Adding a timeout to get() with a max wait gives you a graceful exit path.

The __stop flag that never got set

Notice self.__stop in the constructor. I added it intending to signal the background thread to exit early. But in the version above, nothing ever sets it to True. The thread only exits when the video ends. If you need to abort mid-stream — say, the user closes the window — there's no mechanism. A proper implementation would expose a stop() method that sets __stop = True, and __put_to_queue() would check it before calling put().

Daemon threads and cleanup

I set daemon=True on the background thread as a safety net: if the main process crashes, daemon threads are terminated instead of keeping the process alive. But daemon threads are killed abruptly — they don't run finally blocks or release resources. The OpenCV VideoCapture might hold a file handle that doesn't get closed. For production code, I now prefer non-daemon threads with explicit stop() signaling and proper join() in a try/finally block.

These weren't theoretical concerns. Each one surfaced during integration testing, usually as a mysterious hang after the pipeline had been running for a while. Threading bugs have a nasty habit of appearing only under load.

Does it actually work?

With the threading issues resolved, I ran the same benchmark as before: a 30-second Full HD (1920×1080) video at 30 FPS, with a heavy computation placeholder in the main loop to simulate AI inference.

Method	Average Time (ms)	Storage (MB)
VideoCapture + VideoWriter	~17.7 ms	~35 MB
np.load + np.save	~4.01 ms	~5300 MB
AsyncReader + AsyncWriter	~0 ms	~35 MB

The ~0 ms result needs context. It doesn't mean I/O is instantaneous — it means I/O time is overlapped with computation time. The background thread is reading and writing frames while the main thread processes. As long as processing takes longer than a single read or write operation, I/O contributes zero effective latency.

There's an important caveat: this advantage only holds when processing is heavy enough to cover the I/O gap. If your main loop does nothing but passthrough (frame in, frame out), the async pipeline offers no benefit — the main thread will still block waiting for the next frame to appear in the queue. The pattern shines when per-frame computation time exceeds I/O time, which is almost always true in real computer vision pipelines.

When to use which approach

After living with all three approaches, here's my decision framework:

Standard OpenCV (VideoCapture + VideoWriter): Use this for prototyping, simple format conversion, or pipelines where processing is trivial. It's the simplest code and has the lowest storage overhead. The moment you add a model, reconsider.

NumPy intermediate frames: Useful for experimentation workflows where you need to iterate on processing logic without re-decoding the video each time. The storage cost is brutal, so only use it for short clips during development — never for production pipelines that handle varied or long-duration input.

Async reader/writer: The right choice when per-frame computation is heavier than I/O and you need stable, predictable frame rates. The code is more complex and the threading pitfalls are real, but the performance payoff is dramatic. This is now the default in my production pipelines.

The async pattern didn't start as a flash of insight. It was forced by the failures of the two approaches before it: standard I/O was too slow, NumPy caching was too wasteful. Each dead-end made the shape of the real problem clearer. The solution wasn't about reading faster — it was about never waiting at all.

If your pipeline is struggling with I/O, profile it first. Find out whether you're actually blocked on cap.read() or out.write(). If you are, don't reach for a faster codec or a RAID array. Reach for a thread.