Rendering Video with Cubes

Using video to drive animation across led displays and light installations is very common practice. The availability and accessibility of video makes it a great source of content for creating dynamic movement and patterns across light displays.

With this in mind I wanted to use video to drive the colour and position of cubes in Unity. Using GPU Mesh Instantiation techniques to create lots of cubes on the GPU to render the video. Due to performance limitations I was unable to render each pixel of the video (1920 x 1080) as cubes. Although the GPU is very powerful and can handle a lot of instances of cubes my setup struggled to render 2,073,600 cubes. To workaround this I downsampled the the number of cubes relative to the resolution of the video. Downsampling the resolution of the video 8 times seemed to work perfectly. The visual result of the video being rendered at a lower resolution looked great as it a lofi style.

The colour of each cube relates directly to the pixel from the video, however the cubes can be controlled further with Tint colour, Normal Maps and Emissions Maps that use Unity’s Physically Based Rendering to add realtime Global Illuminations and reflections. The movement and scale of each cube is driven by the brightness of each pixel.

Below as two example videos of this in action :