Many of the panels at last weekend's RTX Australia offered the opportunity to watch live versions of the group's online content -- with everything from live let's plays to podcast filmings. One of the panels was a little different, however, letting the audience interact with characters Yang and Ruby from the animated web series RWBY in real time.
The panel started out with the two characters on screen in a webcam-like setup, with a bit of dialogue that could have easily been pre-recorded. Soon enough they started interacting with the audience, however -- thanks to a real-time face and motion capture system that had been set up with the voice actors backstage.
The technology was provided by Opaque Media -- the company behind the Earthlight VR demo that caught our interest at PAX last year.
It's not the first time Opaque has worked with Rooster Teeth. Back at PAX in 2014 they used a Kinect to map fans' movements to a model of Ruby Rose, allowing them to dance, gesture or generally move around as Ruby in any way they liked.
This time they're taking it a step further, using software from Faceware Tech for a real-time motion capture performance.
The setup for this technology isn't simple, but there are a few main components needed for it to work. Each actor has a GoPro for their facial capture, along with a lapel mic to broadcast their voices. A single Kinect captures arm and body movement for both actors.
They're also provided with a screen to check their mocap performance and another with a live camera feed from the theater room, so that they can interact with the audience. Both actors can also use certain special expressions (such as sticking their tongue out) by pressing a button to trigger the animation.
While it isn't Opaque's first time providing this kind of service, the stylized RWBY faces can come with their own challenges when mapping from a human face, as the range of expression each one can make is often restricted. Each character has their own unique expressions and facial movements as well.
One of the biggest problems encountered with this technology is the fact that the Kinect isn't great at discerning the palm of a hand from the back of a hand, resulting in some weird glitching while the camera tries to figure out which is which. In the end they decided to lock off hand movement completely, bypassing the problem. It did create a few interesting situations for the actors, however -- especially when they tried to do the moves to the YMCA with restricted hand movement.
Overall the reaction to the panel was incredibly positive, however, with most people I spoke to afterwards being very impressed by the technology. One 'hiccup' that Opaque's Norman Wang mentioned to me was when Rooster Teeth's Burnie Burns decided to mess with the stream, telling Yang's voice actor Barbara Dunkelman to announce that she would do her best Burnie impression -- and then speaking into the mic himself.
Being so close to the actor, Burnie also managed to hijack the mo-cap from the Kinect in the process, but while this could have seemed like a small disaster to the Opaque team, it was also one of the crowd's favourite moments. You can see it here on the official Twitch stream.
After everything, the crowd loved it. The theater room -- like a lot of RTX over the weekend -- was completely packed out. "They went nuts," says Norman. He's not planning on leaving it here, however, with plans for further collaborations with Rooster Teeth in the future.
Now that they've done full body capture and facial capture separately, he wants to one day combine the two -- this time using a proper motion capture suit, rather than relying on cameras. This would allow the RWBY actors to walk around a stage and perform naturally as their characters.
If you want to see the whole panel, check out the RTX Twitch stream or the video below. The visuals come in at around 11:57.