Lip sync can make or break your character animation - done well it can take your character's performance to another level - done badly it can turn your shot into a train wreck. This post is for the animator who submitted the offending shot, and the producer or director who can see that it's not working but aren't sure what's wrong with it.
1. The dialog seems too early or late
I've seen it happen more than once where the dialog has accidentally been nudged along the timeline a few frames - so my first take on this would be to slip the dialog track back and forth on the timeline and see if it falls back into sync.
Offsetting the lip sync can actually help - sometimes the mouth shapes may be accurately timed to the dialog sounds, but it still doesn't seem to be in sync when played back. Try offsetting the mouth key poses a frame or two earlier than the corresponding sound - this can improve the readability of lip sync. The exception is the closed mouth sounds B, M and P, which should fall right on the closed mouth pose.
2. The mouth shapes are too big
If the character's mouth moves through BIG shapes for open mouth sounds like A, E and O - this can look pretty grotesque. An animator tackling lip sync for the first time might have referenced a mouth shape chart like Preston Blair's famous 'DIALOGUE' page and created key mouth poses that are oversized. Lip sync mouth shape charts often show exaggerated poses - to make the point, rather than to be followed verbatim. Scale down the size of the extremes of the mouth shape and try it again.
BTW - that single page was the only published document on lip-sync pre-internet days, and a generation of animators owe a debt of gratitude to Preston Blair (RIP) for getting them out of trouble.
3. Too many mouth shapes
If the mouth is moving through too many shapes in the dialog it will looks jittery and unnatural. It's a common mistake to try to hit every single phoneme with a key mouth pose. Identify the important shapes that define the word, and skip over the lesser ones.
Generally you should try to create lip sync with as few mouths shapes as you need to make it convincing - real speech is like that - the mouth doesn't move all that much. An extreme is anime - whose limited animation style typically achieves lip sync with two mouth shapes - open and shut (millions of anime fans never complained about this).
The best way to check what's really going on with speech is to rehearse it in a mirror - and all good animators have one on their desk - right?
4. The wrong mouth shapes
If the mouth is open when it should be closed or wide when it should be narrow, it should be pretty clear that the wrong mouth pose is being used. I know it sounds obvious, but I've seen it many times and unless it was somehow accidental - there's really no excuse for it. As a supervising director once told me - animators who do this should be publicly flogged. That might be a little harsh - but you get my drift.
5. No follow through
It's not a good look when a mouth slams shut - back to the default closed position at the end of a word. Unless the word ends on a closed mouth shape, the last shape of the spoken word should hold for a while.
The final word is that there are no hard and fast rules for lip sync. What works for one character design or animation style, won't necessarily work for another. It may seem bewildering at first, but once you nail it - it's magic. And lip sync is just a subset of facial animation - which is another blog post.
Sherazade The Untold Stories (2017) is an animated TV series initially developed by Hahn Film. Images are from season 1 episode 10 and episode 17, directed by Steve Bristow.
Here's a link to the storm sequence from episode 17
Got any more lips sync tips and tricks? Post 'em as a comment.
Motion Capture (mocap) is the process where actor performances are captured as motion data, then applied to computer generated characters. An 'optical' mocap stage is a large room lined with infrared cameras, which track markers on actors wearing black body suits.
Choice of mocap for an animated TV series
During pre-production of Saladin: The Animated Series the big question was whether to use mocap or to keyframe the animation manually. After much deliberation - it was decided to use mocap. Reasons were - the character designs were humanoid, it was an action genre that would require a lot of complex animation, and there was a belief that mocap would be a time and cost effective solution for the volume of animation needed to make 26 x 24 minute episodes.
Bye bye cartoon physics
The decision to use mocap meant we would have to forego those appealing cartoony qualities like squash, stretch, anticipation, exaggeration and snap. Instead the animated characters would move with realistic human motion. While it's true that mocap'd motion can be edited, in practice it's not that easy to manipulate mocap motion curves (think spaghetti), and in a busy TV series pipeline where artists are under the pump to meet daily quotas, not a practical option. It was a big decision stylistically, that set the tone for the look of the show.
Casting for actors
Not all actors are interested in playing mocap characters. While they will get a screen credit, their face will never be seen, and their voice never heard. Actors will be acting on an empty stage and will often be acting parts alone - unable to play off other actors.
Generally actors will need to be pretty athletic, with good physical performance skills, and if they have other abilities such as martial arts or dance then they may be able to double up and play other characters as well.
Actors are essential - they are trained in performance. And the same goes for stunt performers and dancers - so don't skimp on your mocap talent.
Directing mocap actors
Directing mocap actors is more about choreographing movements in time and space, than extracting sincere emotional performances. With the task of capturing thousands of shots for a TV series - it's a production line.
Before performing a sequence - play the animatic to the actors. Brief them on what's happening in the story at that moment, the emotion in the scene and your vision for the scene. Rehearse the shot. Let the actor interpret your instruction - allow them to have input. Help to locate their starting and ending positions on the stage. Rehearse the shot until you and the actor are ready to go for a take.
Timing of actions is critical - the animatic should have already been cut and timed to dialog recordings before starting any mocap - or at bare minimum the storyboard completed. Sequences are therefore in effect pre-edited - so if a storyboarded shot has the actor taking three steps, turning left and jumping - in three and a half seconds - then that's what has to be performed. There's little room for improvisation - the actors are essentially performing what's already been storyboarded and voiced by vocal actors.
When the mocap technician is ready, she will call 'standby' then 'recording'. Like a film director, the mocap director calls 'action' and 'cut'. This protocol might vary from one studio to the next.
The mocap technician will review each take first. If the take was technically good - without corrupt data caused by occlusion (explained later) or other system limitations, then it will be reviewed by the director and actor to assess whether it's a keeper or not. If not, subsequent takes will be made until a good performance with clean data is achieved. Sometimes a continuous sequence of several shots with different camera angles may be captured in one take, and later in the animation department, the shot cut up into the different camera angles.
Mocap stages will have some basic props and performance aids such as platforms and boxes to simulate stepping or climbing motions; mats for actors to fall or tumble on, and other props such as weapons (typically simple non-reflective wooden or plastic proxies).
The active area of the stage is a performance limitation - actions in the storyboard that need more space than the stage can provide will need to have workarounds figured out - such as cycling a running action or capturing big actions in parts.
Occlusion (in optical mocap systems) is when a marker is obscured from the cameras. Markers need to be visible by at least two cameras to triangulate their location in 3D space. Whether it's self-occlusion or occlusion by other players or props, the result is bad data and retakes. Generally, the more cameras and the fewer players on the stage, the less chance of occlusion. Unless you're on a Hollywood budget, you may be working in a small mocap stage with 12 cameras or less, in which case you can probably put only one or two actors on the stage at a time.
On a mocap stage there is no physical camera to compose the shot, and the actor/s may have no awareness of camera placement or shot composition. So the director has to imagine the performance as it will finally appear on the screen.
However, a well equipped mocap stage will have a virtual camera - a small hand held screen that acts as a viewfinder for the director to frame the shot. The scene is rendered in the screen in real time - and may also include virtual props and set. The director can use the virtual camera to get real time feedback and make composition, staging and lensing decisions. The virtual camera also has markers so it's position and movement can be recorded.
Mocap directors must understand screen direction and continuity if they are capturing scenes for continuity cutting. When all the shots are conformed in the animation timeline later, they must be continuous, otherwise it's back to the mocap stage for a retake, which means time and money (and an anxious producer).
Selected mocap takes are sent for 'clean up' before they can be added to the animation pipeline. Autodesk Motion Builder is the standard app for cleaning mocap data. If you're lucky your data will be pretty clean, if not there will be jerks and bumps caused by occlusion, reflections or other system glitches that need to be smoothed out.
Mocap vs keyframe - conclusion
So did mocap make it easier or faster to make 26 half hours of animation? Well... no. Mocap added a whole other layer of complexity where things could go wrong. The time taken to capture 8,000 shots and the cost of the mocap stage, technicians, stage hands, actors, clean up artists, mocap directors could arguably have been better spent on more animators, resulting in a more traditional animated motion and simpler pipeline. End of the day the realistic motion didn't look out of place on the stylised human characters and the co-production partners were pleased with the outcome.
I'd love to hear about your experience with or thoughts about mocap - leave a comment.
Steve Bristow is an animation producer/director based in Melbourne Australia.