Video Processing Pipeline Showcase
How long does it take to bring an idea to file?
Not so long ago, I was asked how much time did it take to build from ground up an entire video processing pipeline on an FPGA chip (KCU116 Platform). Though somewhat tricky question as it depends on what exactly is in that pipeline, I can finally say, that no matter what – It was worth it. Especially if you can really bring it into life and see it in realtime. Present it, reconfigure it and know that its working as intended.
Sure, one can code up the same blending algorithms that you are about to see in Matlab in a single afternoon, but its probably not as satisfactory, as seeing it in real time on real hardware after “months” of hard work and knowing that it can be reused elsewhere in arbitrary configuration. If you are not from the digital design (ASIC/FPGA) world, then you will probably not understand how much work it requires to receive a video signal, process it and then sent it onto some standartized video interface (HDMI / Display Port).
Bringing the video processing pipeline to life was honestly not an easy task and I would be lying If have said that as it took countless hours of effort.
To make matters a little worse for me, there were not really many viable options how to stream the video data into the KCU116 platform (Except of using FMC Connector and 1080p HDMI expansion cards), but since I have vast experience with data movements and PCIe interface overall, I have decided that I will just read the raw video data from PC memory (IE. RAM) over the PCIe and sent it for processing via DMA Controllers. Besides – I have realized that there is probably not a better way to show up a functionality of an AXI4 Interconnect and AXI4 DMA, than to let them transfer the video images! Therefore, I was able to capture the video mostly by using the hand-written components listed below.
The utilization cells for XCKUP5P device uses 256-bit AXI4 Data Width (PCIe Gen 3.0 x8) and 3 Video Channels (IE 3x DMA in direct register mode) and excludes any 3rd party IPs (PCIe, DDR4) and other configuration and support interfaces (AXI4-Lite infrastructure, HDMI interface, FPGA reconfiguration interface Etc.)
-
- Video Buffer in DDR4 RAM to allow for frame rate conversion “RED“
- Video Processing Pipeline (Position, Cropping, Blending, Etc.) “VIOLET“
- AXI4 Data Interconnect (CrossBar) “GREEN“
- AXI4 Direct Memory Accces (DMA) Controllers “BLUE“
A big thanks! And attribution to people from pexels.com who allowed me to use their videos for the purposes of this post!
Lynx in a Forest | Woman and a Foggy Bridge | Crashing Ocean Waves
I originally wanted to use a slightly different video for the purposes of the demonstration of the mentioned IPs. An interview of someone I deeply admire and a highly inspirational person. I have even asked for the approval to use the interview and wanted to include a full attribution to all the people behind the scenes, but the truth remains that “Who would waste time with a request of some unknown guy?” and “What the hell is FPGA anyway?“. To that part, I have to say that I completely understand that nobody bordered answering me. I just wanted to give it a try as I mostly tend to follow my dreams and hardly ever give up on them (Which might not always be the best option though). If you are curious and want to know who inspired me to use a quote that I widely use and which is among my most favorites ones, I highly encourage you to watch the video by clicking the link below.