The classic film “The Wizard of Oz” is often remembered as the first color film, thanks to director Victor Fleming’s strategic use of black-and-white footage for the Kansas scenes. Similarly, while it may not be the first film to be reimagined with AI, it may soon gain that distinction as well.

For several months, a team of thousands of researchers, programmers, visual effects artists, archivists, and producers from Google DeepMind, Google Cloud, Sphere Studios, Magnopus, Warner Bros. Discovery, and other film and technology companies has been working to bring the 1939 classic to a massive screen in a groundbreaking way.

Their work will premiere on August 28 at Sphere, a colossal Las Vegas venue that has been pushing the boundaries of entertainment since its opening in September 2023. The event will feature generative AI, alongside Dorothy, Toto, and a multitude of munchkins, in an immersive sensory experience.

It is fitting that a film that once revolutionized cinema will do so again. “The Wizard of Oz at Sphere” is an equally ambitious project that combines creativity and technology, enveloping the venue’s 17,600-seat spherical space to create an immersive experience.

Just a few years ago, such a project would have been nearly impossible with conventional CGI. However, with the latest advancements in generative AI media models, specifically Imagen and Veo, alongside Gemini, it has become possible. The team must create an all-encompassing experience using only the original material, without adding new dialogue or music.

Jane Rosenthal, an Academy and Emmy Award-nominated producer, notes, “We discussed doing it in different ways, but we realized that we really needed to do it with AI.”

The Visionary Team

The team cannot simply rely on AI prompts to complete the task. Buzz Hays, the global lead for entertainment industry solutions at Google Cloud, points out that this project is about more than just expanding an old film for a new format.

Hays explains, “We’re starting with the original four-by-three image on a 35mm piece of celluloid – it’s actually three separate, grainy film negatives. That obviously won’t work on a 160,000 square-foot screen. So, we’re working with Sphere Studios, Magnopus, and visual effects artists worldwide, alongside our AI models, to bring the original characters and environments to life on a new canvas.”

When the project began, many team members wondered if AI technology was advanced enough to complete the work. However, due to the limitations of traditional CGI and the excitement to break new ground, they proceeded with the project.

Dr. Steven Hickson, a Google DeepMind researcher, comments, “The models are wildly innovative. We’d find something we couldn’t do, think it’s impossible, and then a month later, we’re like, actually, maybe we can do that.”

The challenges were significant, including magnifying the original grainy images for Sphere’s 16K LED screen and accounting for camera cuts in traditional films.

The team had to overcome three major technical hurdles: scaling, outpainting, and performance generation.

The Magic of Fine-Tuning

Using specially tuned versions of Veo, Imagen, and Gemini, the team developed an AI-based “super resolution” tool to turn the original celluloid frames into ultra-high-definition imagery. They then performed AI outpainting to expand the scope of scenes and fill gaps created by camera cuts.

Through performance generation, they incorporated composites of the famous performances into the expanded environments, achieving natural gestures, staging, and fine details that conventional CGI struggles to match.

Dr. Irfan Essa, a principal research scientist at Google DeepMind, comments, “When the request came to us, I was almost jumping up and down. This is the best opportunity to showcase the magic that we develop using AI.”

The team scoured archives to build a vast collection of supplementary material, such as shooting scripts, production illustrations, photographs, set plans, and scores, to fine-tune the AI models.

By uploading these materials to Veo and Gemini, the models can train on specific details of the original characters, environments, and production elements, improving the quality of the outputs.

Following the Yellow Brick Code

As the team continues their journey with this groundbreaking project, many are in awe of what they’ve achieved and excited about what’s to come.

Jim Dolan, Executive Chairman and CEO of Sphere Entertainment, comments, “When you have innovation like this, you don’t always know where it’s going to go. You have to be able to take a leap of faith. What you’re going to see in ‘The Wizard of Oz at Sphere’ is clearly a leap of faith.”