AI lip syncs for Hollywood & Television

Hollywood films have always reflected our enduring fascination with Artificial Intelligence (AI). There have been several films such as Blade Runner, 2001: Space Odyssey, The Matrix, Terminator: Dark Fate, and many more, with AI as their subject. However, slowly but surely, AI’s role has evolved from being the script’s subject to enabling the dubbing of films in multiple languages.

Today, AI is impacting filmmaking in many ways. For instance, the movie division of Warner Bros. is leveraging an AI-driven project management system’s data and predictive analytics to guide decision-making at the greenlight stage. The integrated online platform can assess the value of a star in any territory and how much a film is expected to make in theaters and on other ancillary streams. As a result, executives can spend time on core responsibilities such as packaging, marketing, and distribution decisions, including release dates.

Furthermore, the creators of “Rogue One” used AI to bring back the character Grand Moff Tarkin to life on the screen. Then there is AI-based video editing, which can create hyper-realistic videos using face swaps that leave little trace of manipulation.

AI Takes On the Dubbing Role

Dubbing of films is yet another significant filmmaking aspect being influenced by AI. To gain a broader reach, production houses, film distributors, television channels, and Over the Top (OTT) streaming services leverage AI speech translation engines for dubbing films into diverse languages.

Eros Now is working with Microsoft to use the technology giant’s artificial intelligence speech translation engine to dub Bollywood movies in 10 Indian languages as well as five global ones.

Furthermore, Engadget reports that Cyberpunk 2077 the action-packed RPG will launch with support for dialogue in 10 languages and subtitle options for several others. CD Projekt Red is aiming to add a deeper layer of immersion for players by using artificial intelligence to lip sync the dialogue in multiple languages.

Shortened timeframe and reduced costs are seen as the two key reasons for this substantial interest in using AI to dub films, documentaries, and TV shows into multiple languages instead of the traditional methods. Over and beyond the faster time to market and reduced costs, it is the developments in AI-based language dubbing that is leading to its increased adoption by the film industry across languages and countries.

AI-based Dubbing Face/Off

An AI-based system is trying to change the facial expressions of actors to match dubbed voices accurately. The Automatic Face-to-Face Translation protocol can sync the visual, so the voice style and lip movement match the dubbed language. The protocol can automate the dubbing process at different levels with different trade-offs.

For instance, Face-to-Face Translation can dub a given movie scene in a particular language into a different language without a visual discrepancy between the lip motion and the dubbed audio. When it comes to documentaries, television series, and interviews, Automatic Face-to-Face Translation protocol can potentially allow viewers to access and consume important information from across the globe irrespective of the underlying language.

NMT can play a stellar role in AI-based Dubbing

Neural Machine Translation (NMT) – a fascinating development in AI enriches the dubbing process without altering on-screen content. In NMT, different algorithms work together to enable machines to learn expressions, grammar, and linguistic rules and subsequently predict complex sentences. Furthermore, NMT can learn new languages, and therefore depending on the volume of work done through NMT, dubbing will become more refined as it will also learn the nuances such as pitch and intonation.

Conclusion

It was impossible to imagine that automated dubbing would be anything but robot-like until a few years ago. Today, there is variation in voice delivery, and the AI-dubbed versions are much more life-like. In addition, the process is faster and would have lesser errors than the traditional methods. Significantly, for the production houses, television broadcasters, and distribution companies, AI-based dubbing can help them air the dubbed versions simultaneously with the original version.