WorldViz User Forum - HD complete characters speaking naturally

WorldViz User Forum (https://forum.worldviz.com/index.php)

- Vizard (https://forum.worldviz.com/forumdisplay.php?f=17)

- - HD complete characters speaking naturally (https://forum.worldviz.com/showthread.php?t=4500)

HD complete characters speaking naturally

We're thinking of buying the HD complete character set. One very interesting features of these characters would be the facial expressions. I'm especially interested in making the characters speak more naturally (the lipflapping of vizact.speak() is far from optimal for conversational agents). Is it possible to let the HD characters speak with phonemes (see http://www.karisproductions.com/wp-c...neme-Chart.jpg)? If so, how would you extract phonemes from a wav-file (or a sound in another format), so that the HD characters gets the information which lip movements to make to speak realistically? Would you need the Microsoft Speech API for this?

Most important question: could we test this with the HD complete character that comes with Vizard4?

Hi Frank,

Our characters are not currently set up with phonemes included, though these could be created either as morph targets or with the included facial bones.

Third-party programs exist for extracting phonemes from audio files, though I don't have any experience with them. One I have seen recommended is CMUSphinx. I also wouldn't be surprised if a python library exists that allows phonemes to be detected in real time.

To clarify, the VCCHD set has expressions and phoneme pose files (both in .bip and .cpy) that can be loaded onto the skeletons in Max. As of this moment they are not currently part of the exported versions of the avatars. It may be worth adding them to the DVD build.

The phonemes included as pose files are:

Quote:

- frame_00: NEUTRAL
- frame_01: A; K; I
- frame_02: Ah
- frame_03: B; M; P
- frame_04: C; D; K; N
- frame_05: Ch; Sh
- frame_06: E
- frame_07: Eh
- frame_08: F; V; W
- frame_09: G; J
- frame_10: G; J; R
- frame_11: L
- frame_12: N
- frame_13: O
- frame_14: S; X; Z
- frame_15: T
- frame_16: Th
- frame_17: U; W; Y

The expressions included as pose files are:

Quote:

- frame_00: NEUTRAL
- frame_01: Angry_01
- frame_02: Angry_02
- frame_03: Angry_03
- frame_04: Angry_04
- frame_05: Bored/Disgusted
- frame_06: Disgusted
- frame_07: Eye_left_closed
- frame_08: Eye_right_closed
- frame_09: Facial_shrug
- frame_10: Friendly
- frame_11: Frightened_01
- frame_12: Frightened_02
- frame_13: Grin
- frame_14: Grin_left
- frame_15: Grin_right
- frame_16: Kiss
- frame_17: Laugh_01
- frame_18: Laugh_02
- frame_19: Laugh_03
- frame_20: Narrow
- frame_21: Pain
- frame_22: Puh
- frame_23: Sad_01
- frame_24: Sad_02
- frame_25: Sad_03
- frame_26: Sad_04
- frame_27: Shout
- frame_28: Smile_01
- frame_29: Smile_02
- frame_30: Smile_03
- frame_31: Surprised

Hi Valeno,

Thanks for the response! It sounds like (no pun intended:)) the VCCHD set has all the ingredients to create a naturally speaking avatar, when phonemes are extracted from a .wav file. Too bad none of them are included with the standard character that comes with Vizard. I noticed that the HD example character does not have any morphs

It would be a good idea to include the different poses in the exported versions of the avatars on the DVD build! I also work with FaceGen, which includes similar morphs for the phonemes, so I will test those. If I can make those phonemes work, I think we can also get the VCCHD characters to speak properly.

Regards,
Frank