Adaptive Man-Machine Interfaces

MIT9904-15

Start date: 07/99

Project summary

Project description

Prior Work

Much of the previous work in text-to-audio-visual (TTAVS) speech synthesis has focused on integrating physically-based facial models with a particular speech synthesis system in order to give the impression of a "talking face". Some TTAVS systems have also resorted to Cyberware scanning techniques to overlay realistic-looking skin texture on top of the underlying graphics model.

In previous work, we explored an image-based, morphing approach to facial synthesis, in an attempt to bypass the need for any underlying 3D physical models. Our talking facial model was comprised of a collection of viseme imagery and the set of optical flow vectors defining the morph transition paths from every viseme to every other viseme. A many-to-one map was assumed between the set of phonemes and the set of visemes.

Our Approach

While our earlier work made strong strides towards photorealism, it did not address the dynamic aspects of mouth motion. Modelling dynamics requires addressing a phenomenon termed coarticulation, in which the visual manifestation of a particular phone is affected by its preceding and following context.

In order to model lip dynamics, we propose an unsupervised learning framework in order to learn the parameters of a dynamic speech production model. Our new approach is composed of 3 substeps:

1. Morpheable Model of the Lips: we first record a training corpus of a human speaker uttering various sentences naturally. Motivated by recent progress in the creation of statistical shape-appearance models of flexible objects, we build a flexible shape appearance model of the speaker's lips. Then we analyze the entire corpus using the model, yielding a low-dimensional time-series of the lip shape for the entire corpus.
2. HMM Speech Production Model: Next we hypothesize a dynamic speech production model based on HMMs. Each phone is a 3-state left-to-right HMM model with Gaussian emissions. Baum-Welch learning is used to learn the paramters of each phone model from the labeled corpus calculated in step 1 above.
3. Synthesis: Finally we propose a synthesis algorithm to generate novel visual utterances, given input text. The appropriate HMM phone models are concatenated, and smooth set of parameters generated from the HMM using a novel synthesis algorithm that calculates the most likely emission outputs from an HMM given known state sequences.

Demos, movies and other examples

Presentations and posters

For more information