A live camera to cloud production workflow was the climax of HPA Tech Retreat 2021 and demonstrated both how promising the technology is and also how far there is yet to go.
“Where last year we showed camera to cloud as proof of concept, this year we’ve been working to make the system more robust for field cloud acquisition,” explained Michael Cioni, global SVP innovation for Frame.io at the event.
To underscore the point, Cioni was literally in a field, organizing the live shoot of footage intended to be inserted into an updated version of last’s years HPA Tech Retreat experimental film The Lost Lederhosen. The entire workflow was taken live over the course of three hours from location into post including edit, VFX, audio and color treatments to final deliverable all in the cloud.
The demo shot using a RED Helium 8K camera which automatically triggered the capture of proxy files encoded in H.264 from a Teradek Cube 655. These files, included timecode and “hero” metadata were transmitted instantly to Frame.io’s cloud platform. Simultaneously, audio captured to a Sound Devices 888 portable mixer-recorder was uploaded to cloud via a hotspot thrown by a Net Gear LTE modem. The audio files were uncompressed original .wav files timecode jammed with the camera proxy for synching in post.
Production company Origin Point was next in the chain and the demo showed without smoke and mirrors, the proxy files immediately available on an Adobe Premiere panel for the creative process to begin.
The demonstration, organized by studio-owned thinktank MovieLabs, is a marker on the road to its vision to move production to the cloud by 2030.
“Many people walked away from last year’s Tech Retreat thinking 2020 is the new 2030,” said Leon Silverman, a Senior Advisor to groups including the HPA and MovieLabs. “While on the surface there is a lot of cloud capability, and a lot of work has been done in cloud connected collaborative production there is a lot of vital foundational work that needs to be to created before this vision becomes a practical reality.”
The HPA Supersession highlighted two huge gaps. The first is that while proxy video can be transmitted near instantly to the cloud, the higher bit rate 4K Original Camera Files (OCF) take some time to digest.
“OCF does not transmit from the camera today but this will change,” Cioni said. “By 2031 a media card will be as unfamiliar as arriving today on set with a DV cartridge or DAT tape. You won’t have removeable storage from the camera. Camera tech will transition to become transfer systems to the cloud. It will take a decade but the transition starts here.”
Another piece of the puzzle due for rapid change is mobile broadband. “As it stands to today, in most areas 5G is weaker than 4G,” Cioni said. “That’s a concern for now but 5G’s performance will skyrocket as it is rolled out. In the meantime, a cellular bonded device like (the one used bonds networks from AT&T, Sprint and T-Mobile) delivers great performance but we have to download. For some perspective, uploading 4K OCF to the cloud seemed unobtainable a few years ago and now we’re using cell phone technology to transmit it.”
The HPA demo had to download OCFs to a laptop connected wirelessly to the cellular bonded hotspot and then into the cloud.
The second leapfrog that needs to be made is the ability for a production to access media assets seamlessly regardless of which cloud it is stored on. This would fulfil MovieLabs’ principle of creative applications moving to the data and not the other way around.
Right now though, different facility and technology vendors have a preferred cloud partner. In the HPA model, Frame.io was used to hop between them.
To elaborate, in the HPA demo media was uploaded first to AWS on the U.S East Coast then accessed from two further cloud regions. Avid instances of Media Composer and Protools sharing the same Nexis storage running on Microsoft Azure on the U.S west coast was required by Skywalker Sound. Adobe Premiere was used by Origin Point as a virtual workstation provided by Bebop running on a Google instance located in Amsterdam. Another Google instance in the LA region was uses for VFX (Mr Wolf) and color conform (at Light Iron)
“The workflow essentially means logging into one cloud, downloading the media and uploading it back to Frame.io to pass onto the next stage in the pipe,” said Mark Turner Program Director, Production Technology, MovieLabs.
“It’s hardly efficient and generated a lot of egress but we have made it work without using hard drives.”
Data movement throws up many of the same problems as experienced when physically moving media between facilities, he explained. There’s more chance to introduce errors, media loss or loss of metadata plus it introduces delay. There’s also a chance of confusion – both human and machine – when you move anything, a risk of version duplication and greater gaps in security. When most cloud providers on most tiers of storage charge for egress any data movement incurs a fee.
“The 2030 vision avoids these problems because there is a single source of truth,” Turner said. “We expect workflow to continue to span multiple cloud infrastructure and we encourage that choice — but the crucial point is that sharing work across various clouds should be seamless so it acts like one big cloud.”
MovieLabs may have called its vision 2030 but fully expects the industry to rally round and exceed those expectations.
“There are parts of it we can do today and parts that with cooperation we can do much sooner we think. We should be thinking not 2030 but how can we get it to 2025 or 2022?”
Common Visual Language
As the industry coalesces around cloud, MovieLabs also thinks it a good idea to have a common language to be able to communicate ideas.
“We’re trying to do is come up with a language to express workflow and communicate that to other people,” explained Jim Helman, MovieLabs co-founder and CTO. “This would be used by our member studios and by the industry to do workflow diagrams, dashboards and app development. We will make sure the language is open, flexible and has all the resources needed for widespread adoption.”
MovieLabs’ visual language is a set of shapes, lines, and icons to describe the key concepts of the 2030 vision for film and tv production. The four basic concepts include participants (an actor, a cameraperson or director), task, (filming, setting up some lighting, writing a script), asset (original camera files, script or stills for reference) and context (what it is that is happening – given shot or take for example). The visual language also expresses the relationships between these groups by way of arrows.
It’s not a million miles away from any workflow diagram on any powerpoint but MovieLabs aims to simplify and unify the iconography to make understanding workflows by different parties consistent and easier.
Of course, some systems at a higher level but be general while others will be specific to projects or divisions within projects. That’s why MovieLabs is talking about multiple connected ontologies rather than a single all-encompassing one.
Ongoing work includes creating a sound asset terminology to support a sound file naming specification; and camera metadata mapping so that there can be some consistency in searching camera files in databases.
Common Security Framework
Traditionally, production workflows happen on a facilities infrastructure or on hybrid cloud controlled by the facility and secured by a secure perimeter. When production moves to the cloud, this becomes a cloud resource shared outside of the facility infrastructure and by everyone working on productions.
“This means workflows happen outside of secure perimeter,” warned Spencer Stephens SVP Production Technology & Security at MovieLabs. “You could attempt to throw a secure perimeter around the cloud and every vendor and person working in it – but that would be extraordinarily complex and complexity is the enemy of security.”
Movielabs’ conclusion is that production in the cloud requires a new approach. Among its principles is that security should be intrinsic to workflow not an add-on. It must be designed to secure cloud-based workflows not the infrastructure it runs on (i.e assets are secured, not the storage it lives on).
“Our model describes an authenticated participant carrying out an authorised task on a trusted device using an approved application to a protected asset,” Stephens explains. “It’s a zero-trust architecture. It does not require the infrastructure to be secure and it protects the integrity of the workflow. Nothing can take part in any workflow unless authenticated and no-one can take part in a particular workflow unless authorised to do so. All users, devices, software everything must be authenticated before it they can join the production.
“It is also scalable. A Hollywood movie might turn the dial up to 10 and want individual assets to be encrypted while a reality TV show might be content with security at 3 and use access controlled. It’s really down to a matter of risk tolerance.”
Virtual Production Meets Cloud
Cloud-enabled virtual production was another major plank of the HPA agenda.
“Everything in the future is going to be previs, techvis and virtual production,” said Erik Weaver, a Sr. Consultant Entertainment Technology Center @ USC. “Understanding things like a virtual camera will be a critical component to a cinematographer in the future. They will want to be able to see the magic angle they’ll potentially get on set which is different from not being able to visualize anything with a green screen.”
Solstice Studios CTO Edward Churchward, said, “Over the last year, we’ve seen a real reduction in the cost level. Many small productions on lower budgets are now able to avail themselves of virtual production efficiencies.”
Finished Pixel Virtual Set Finishing
Camera tracking specialist Mo-Sys discussed a workflow that delivers higher graphics quality combined with a real-time virtual production workflow. Technology such as games engines combined with cloud means onset finishing or finished pixel rendering is possible onset, nullifying the guesswork of on set by fixing in post, but also eliminating much of traditional post altogether.
“Realtime rendering is so good it is often good enough for finishing,” said Mike Grieve, Commercial Director at Mo-Sys. “Removing post production compositing from the pipeline in order to save time and money is surely the key driver to using virtual production.”
But there’s a dilemma. “As good as the graphics systems are they are still governed by time, quality and cost,” he said. “How do you increase virtual graphic quality without substantially increasing the cost on set or delaying delivery by using post-production.”
The Mo-Sys solution is a dual workflow where graphics are rendered once on set in realtime and again in the cloud using full quality keys and composites before returning the results back to set within a few minutes for review.
“We are not a post killer,” insisted Grieve. “It will leave post to focus on the creative task not repetitive tasks that should be automated.”
What production ends up with is a final pixel virtual production workflow combined with a post workflow to improve onset quality without incurring extra cost and time.
Uren added that the pipeline will offer the flexibility of deciding if the render is required in five minutes or five hours. “We have a working prototype on the bench and are in the process of deploying it to the cloud,” he said.
This process works with blue and green screen where extracting the key is straightforward but applying this dual render to a LED volume where you are shooting final pixels is not only not easy – “it’s currently not possible to create a simultaneous key while shooting a full composite shot,” said Grieve.
“But we’re working on it for 2022.”
