Cyberpunk 2077 Uses AI To Lip Sync In Multiple Languages

2
Cyberpunk 2077 Uses AI To Lip Sync In Multiple Languages
Image: Cyberpunk 2077

Cyberpunk 2077 is set in a vast open world populated by hundreds of NPCs. With such a large scope (and the challenges of the pandemic), animating every character in the game by hand is a near-impossible task — but with the help of advanced AI processes, the CD Projekt Red team has found a way to get around the traditional motion capture process.

To build on the sense of realism and quality established in earlier games like The Witcher III, CD Projekt Red is employing JALI, an advanced AI animation tool designed to produce realistic and expressive speech animation with minimal oversight.

Traditionally, this work is completed by actors in motion capture rigs, but the rise of the coronavirus pandemic has meant this process is no longer viable. Instead, JALI will take the reigns and help shape the NPCs of Cyberpunk 2077‘s world.

JALI is a procedural tool that breaks down animation into multiple categories including character model, face shape, muscle movement, speech, speech style, eyes and brows to create more realistic and believable facial patterns during speech. It’s set to be used by every voiced character in the world of Cyberpunk 2077.

It’s also going to be used for the localisation process.

Cyberpunk 2077 is set to have 10 voiced language options including English, French, Japanese, German, Spanish, Polish and Simplified Chinese. JALI is being used to ensure these vocal performances are dubbed accurate and realistic for every language with a focus on correct lip syncing.

Lip syncing is a feature that can really make or break a game’s level of immersion. It might seem inconsequential in the long run, but every detail adds up, particularly in a game world as intriguing as Cyberpunk 2077.

You can hear more about the animation process behind Cyberpunk 2077 in the video below. It’s fairly long, but very interesting and shows just how far animation (and AI) in video games has come over the past decade.

Cyberpunk 2077 is still set for release on November 19. Stay tuned to Kotaku Australia for all the latest news and updates on the game.

Comments

  • Would be incredible to match this with AI voice as well
    Replica Studios is making some super interesting AI voices that are sounding increasingly more natural and expressive!!
    AI voices will make it so much easier to make a lot of voice content in the future, and massively expand the customization and dynamic game content to match any player’s game style. Can’t wait!!

Show more comments

Log in to comment on this story!