Raspberry Pi video lag wasn't about frames — fixing a hidden 50 ms delay

ran into a stubborn video latency problem while working on a clone of the WiiU gamepad. The project involved receiving the video stream from the console and displaying it on a Raspberry Pi Zero 2W. Everything worked, but there was a persistent 3-frame delay in the video pipeline that simply would not go away. Even substituting dummy frames into the pipeline and ignoring them at the output stage failed to eliminate the lag.

After digging deeper, realized the issue was not tied to a specific number of frames at all — it was tied to time. The delay amounted to roughly 50 milliseconds. Further investigation revealed a flaw in how frames were fed into and retrieved from the decoder. The first frame took about 50 ms to decode, while subsequent frames were processed much faster, sometimes in as little as 5 ms. However, the code was not set up to grab frames the moment they were ready, so the initial lag carried forward through the entire stream.

The breakthrough came when started polling the decoder on a regular basis. This allowed the system to pull each frame as soon as decoding finished, rather than waiting for a fixed schedule. As a result, the accumulated delay disappeared, and the Raspberry Pi Zero 2W could process and display the WiiU video feed with the same responsiveness as the original Nintendo WiiU gamepad.

The full explanation, including how the polling approach solved the problem, is covered in detail in the accompanying video. This case is a useful reminder for anyone building camera or video projects on Raspberry Pi: latency is not always about frame counts, and sometimes the fix is as simple as changing how and when you check for completed frames.