Ever woken up at 8:00 in the morning on a Saturday with a scheme for how to puppeteer a mouth using a Razer Hydra?

Me too!

Workflow went a little something like this:

Prep:

• Before all the things, I rendered out a half-speed version of the “Pulp, Emo!” dialogue snatch and drop it into the Video Sequence Editor. It sounds very 1990s jungle.

Modelling/Rigging:

• I made a slightly open neutral mouth. I added five different keyframe shapes – open, closed, spread lips, rounded lips and bottom lip curled up
• After doing the Hydra controller setup, I set up drivers to the following effect: joystick up makes the mouth closed, joystick down makes the mouth open, joystick left makes the lips spread, joystick right makes the lips round. With the axes set this way, I get complementary mouth shapes at different compass directions. The mouth can’t be simultaneously open and closed and the lips can’t be simultaneously spread and rounded. (That said, a jaw can definitely be varying degrees of open with the lips closed so the system is far from perfect. It’s hacky. I admit it.)
• Rolling the joystick around clockwise makes the lips go mwoaym. Anticlockwise is more like meowm.

Animation:

• I set the Frame Rate to Custom and set / to 2 – this runs the animation at half-speed. This is a trick I learnt from my music days – never record something difficult at playback speed if you don’t have to. Varispeed is your friend.
• I shoved the half-speed file in at about Frame 20.
• I rehearsed the lipsync performance for about an hour. Words turned into gestures: “want” and “what” became like little hadoukens; “pulp!” was like carefully parking a car – going from park to reverse to park again.
• I took a deep breath and recorded a performance, holding down the stick to record it while moving it around is a bit ergonomically hostile, but it’s vaguely doable. What I recorded was definitely in the ballpark but there were a ton of mistakes.. nnh. Easily enough fixed, though.
• I replaced the half-speed sound with the full speed sound and set the frame rate back to 24fps.

Compiling the animation data:

• I made a new Empty object.
• I added a Copy Location constraint to it from the hydra_joystick_left to get the joystick movement, and then used a Transformation constraint to grab the Y movement of hydra_trigger_left (which maps trigger 0-1 to Y Location 0-1) and map it to the Z axis of the empty. All the captured data now lives in one object. Woohoo!
• Cue farting around with drivers for a few minutes to get the shape keys pointing to this new Empty.
• I baked the action to the Empty – this writes the animation resulting from the constraints to editable keyframes in a new action, ready for tweaking.

Clean-up:

• In F-Curve editor, i set all the keyframes to linear because the handles in Bezier mode were getting really irritating.
• I set to work editing the mocap data! A lot of the fixes came down to removing redundant keyframes, smoothing transitions that were unnaturally sharp, getting rid of any jerky motion… mocap data is messy, yo.

Midway through the video you can see both the raw and cleaned up versions playing at once – some of the performance errors are in the timing, some of the errors are making the wrong mouth shape for the sound.. but the final “Pulp!” pretty much stayed as is.

Overall this is an interesting alternative way to get lipsync done. I’m not sure if it’s quicker or better than just manual keying but it’s definitely less of a dry technical exercise. 🙂

Download the .blend file here – to directly play with the mocap, you’ll need the Razer Hydra mocap add-on installed (Windows only) and also a Razer Hydra, but even without it you should still be able to move the control points around to see how the rig works.