What is Virtual Production

Virtual production is a method of filmmaking in which live action scenes and computer generated imagery is combined to create photo realistic environments in real-time. Recent advancements in graphics processing units (GPUs) and game-engine technology has made real-time photo realistic visual effects (VFX) a reality. The emergence of real-time photo realistic VFX has sparked the beginning of a revolution in the film and television industry. Now with virtual production, the physical and digital worlds can interact with one another seamlessly in photo realistic quality.

By embracing a combination of game-engine technology and fully immersive LED screens into the creative workflow, virtual production offers efficiencies in the creative process resulting in more seamless on-screen experiences. At a high level, virtual production allows historically siloed creative teams to collaborate in real-time and make decisions more quickly as each respective team can see what the final shots will look like while the actual shoot is taking place

Disruptive Technology In Film and Television

Disruptive technology is defined as an innovation that significantly alters the way consumers, industries, and/or businesses operate. For the film and television industry, first came the shift from silent film to audio, then from black and white to colour motion picture, next came television, then home video tapes and DVDs, and most recently streaming services.

Over the years, the methods used to produce film and television series have also seen major technological disruptions. A major shift which sets the tone for the remainder of this article was the transition to modern VFX, arguably pioneered by the films Jurassic Park and Terminator 1. Other milestone movies for VFX include The Matrix, Lord of the Rings, Avatar, and Gravity. The film buffs out there are more than welcome to share your thoughts on which movies served as pioneers/milestones for modern VFX.

Traditionally, movie and television production has been split into three stages; pre-production, production, and post-production.  In the past, VFX was all done in post-production, however, the emerging virtual production method transfers the majority of the VFX process to the pre-production and production phases with post-production done only for specific shots and after shoot fixes.

LED Screens in Creative Workflows

Virtual production is a lot of technology coming together to form an integrated system. Worlds merging where traditionally they had nothing to do with one another. New partnerships, new processes, new technology, the list goes. Virtual production is still very much in the early adoption phase and loads of people are trying to wrap their heads around it.

Anyone who has done the least bit of research on this topic has likely come across the articles posted on FX Guide by Mike Seymour, Art of LED Wall Virtual Production Sets, Parts One and Two. The articles provide a window into the making of The Mandalorian television series, which was filmed in large part on direct view LED screens. Seymour outlines the lessons learned throughout the making of The Mandalorian with respect to how virtual production changed the creative workflow. Part Two reflects on the technical side and challenges faced while implementing in-camera VFX.

Sharing this level of thought leadership has propelled the understanding of recent technological advancements forward for film and television producers. With multiple films and television series having successfully leveraged real-time VFX, the race is on to adopt the newest workflow. Further adoption of virtual production has been in part due to the Covid-19 pandemic shifting the world to remote work and requiring all businesses and organizations to rethink the way things are done.

Designing LED Screens For Virtual Production

Given the scope of technology required for virtual production, getting down to the technical details of how each technology performs and what specifications actually mean requires a combination of experts from the respective worlds at play. This brings us to the true purpose of this article, to write from the perspective of an industry leading direct view LED manufacturer on what goes into designing an LED screen for virtual production.

LED Screen Configuration

The configuration and curvature of an LED volume largely depends on how the virtual backdrop will be captured and how the cameras will be moving while filming. Will the volume be used for broadcast and live stream? If so, will the camera shoot from stationary angles or will it pan around the focus point? Or otherwise will the virtual set be used for full motion video? If so, how will the live action personnel and materials be captured in the volume? These types of considerations allow LED volume designers to determine the appropriate dimensions of the LED screen, whether the screen should be flat or curved and at what angles, ceiling and/or floor requirements. Key factors to manage include providing a large enough canvas to allow for complete viewing frustums and minimizing colour shift, which results from the viewing angles of the LED panels that make up the screen.

Pixel Pitch and Moire Patterns

Moire patterns can be a big issue when filming LED screens. Selecting the correct pixel pitch is the best combatant against moire patterns. If you’re unfamiliar with pixel pitch you can learn more about it here. Moire patterns are caused by the camera picking up high frequency interference patterns caused by the individual pixels of the LED screen. The relationship between pixel pitch and viewing distance for virtual production is not only with respect to where the camera is but the closest focal point for all scenes. If the focal point is within the optimal viewing distance for a respective pixel pitch, moire effect will occur. Depth of field adjustment can further reduce moire effect by shooting the background slightly softer. As a rule of thumb, multiply the pixel pitch by ten in order to get the optimal viewing distance in feet.

Refresh Rate and Flicker

The flicker seen when filming monitors or for our case LED screens, is caused by a mismatch in refresh rate of the display and frame rate of the camera. LED screens require a high refresh rate of 3840hz as it helps to eliminate screen flicker and is absolutely necessary for virtual production application. Ensuring the LED screen has a high refresh rate is step one to avoiding screen flicker while filming, aligning camera shutter speed with refresh rate completes the fix.

Brightness

For LED screens serving off-camera applications, brighter is generally taken as better. However, for virtual production, the brightness of an LED screen is often overpowering and brightness is turned down significantly. When an LED screen’s brightness is reduced, colour performance is sacrificed. The grayscale gets reduced due to there being less intensity levels available per colour. Ensuring the LED screens maximum brightness is aligned with the maximum light output required for sufficient lighting within the LED volume minimizes the amount the LED screen brightness needs to be turned down and minimizes loss of colour performance.

Colour Space, Grayscale, and Contrast Ratio

Colour performance for LED screens is made up of three main components; colour space, grayscale, and contrast ratio. Colour space and grayscale play important roles in virtual production applications, whereas contrast ratio is less critical.

Colour space refers to the specific organization of colours achievable by the screen. Producers should consider the colour space that will be required ahead of time as an LED volume can be designed with different colour spaces if desired.

Grayscale is measured in bits and indicates how many intensity levels are available per colour. Generally speaking, the higher the bits, the more colour available which results in smoother colour transitions and eliminates banding. For virtual production LED screens, grayscale of 12 bits or high is recommended.

Contrast ratio is the difference between the brightest white, and the darkest black. In theory it allows viewers to distinguish content of an image regardless of the brightness of the image. However, the specification is often misrepresented. LED screens capable of higher brightness, by extension have higher contrast ratios. At the other extreme rests fill factor, where using smaller (typically cheaper) LEDs increase the black of the displays leading to higher contrast ratios. Contrast is important, but it’s important to understand the factors in determining the contrast ratio.

Set Visualization

Effectively designing an LED volume for the space and productions it will be built for is the first step to successful implementation of LED technology for virtual production. Given the custom nature of LED screens, virtually building an LED volume in a 3D world is the most effective way to plan the dimensions, curves, mounting, and viewing distances of the LED screen. This allows producers and engineers to visualize the volume and discuss requirements ahead of time while making informed decisions along the way.

Site Preparation

Last but not least, throughout the design process, as teams design and discuss the LED volume important site specific topics will be considered including but not limited to structural, power, data, and ventilation requirements. All of these factors will need to be appropriately considered and made available for proper implementation of the designed LED screen.

For more information on LED screens for virtual production, or to start your project with LIGHTVU, click here.