Expressive Gesturing

From text to gesture

• The gesture is an important component of the information transfer from the speaker to the listener. To fit with this reality, the idea of the GV-LEx project is to make NAO and Greta perform gestures corresponding to situations and characters mentioned in the story being told.

• The figure below summarizes the process allowing to generate gestures for NAO or Greta starting from the text.

• A first software analyzes the text to automatically extract the informations that may be illustrated by gestures. A "hello!" said by one of the characters is interpretated as a greeting that can be reinforced by a gesture. This analysis is described at the Linguistic Aspects page.
• A further step transforms the semantic annotations extracted from the first analysis into practical functions in order to translate the information into gestures. The Function Markup Language (FML) is used for this. The greeting identified in the text at the previous step is translated into a hand gesture or a head nod.
• The last step transforms the standard gesture into a move for each actuator of the robot or virtual human using the BML formalism (Behaviour Markup Language).

Greta and NAO Gestures

The robot NAO and the 3D avatar Greta, although they both have a humanoid shape, cannot perform the exact same gestures.

• The robot and the virual human do not have the same number of joints and their degrees of freedom differ.
• The avatar has an animated face which is not the case of NAO. Yet his eyes can change colors.
• NAO has physical constraints (auto-collision, balance, limited speed and acceleration) that Greta does not have.

This is why the last step to gesture differs for the robot and for the avatar.