Making Faces
Expressive VR conversations, controlled with a thumbstick.
Making Faces lets someone change their avatar’s facial expression while talking to another person in VR. A radial selector puts six expressions under the right thumbstick. The expression layer runs alongside audio-driven lip sync, so the avatar can look happy, angry or surprised while continuing to speak.
Research context
The related journal study compares face tracking, labelled thumbstick selection and thumbstick selection with a confirmation button during live conversations. This page focuses on the engineering behind the controller-driven interaction.
Read the related publication →From thumb movement to a visible expression
- Right thumbstick X/Y
- Angle + dead zone → expression index
- Animator parameter → facial blendshape clip
- Local feedback + replicated avatar state
Reading and mapping controller input
The controller reads the right-hand thumbstick through OVRInput every frame. An atan2 calculation converts the input direction into one of six angular sectors, with a dead zone around the centre. The menu cursor is smoothed and clamped to an ellipse for readable visual feedback; expression selection uses the raw direction, so the smoothing does not delay the underlying choice.
Driving animation during a live conversation
A selection updates the Animator’s expressionInt parameter and enables the facial-expression layer. The selected state plays an authored facial blendshape animation while speech-driven visemes continue to animate the mouth. A mirror gives the user immediate feedback on the selected expression, helping connect a controller movement with the avatar’s appearance.
Returning cleanly to neutral
A key state-management challenge was avoiding a face becoming stuck in its last expression. The reset coroutine waits for the Neutral0 state before disabling the expression layer. If the user makes a new selection during that reset, the pending reset is cancelled, allowing the new input to take priority.
Keeping remote avatars consistent
In the Unity Netcode prototype, the owning player publishes the expression index, animation-state hash and state time. Remote instances apply that state to their own Animator. Ownership checks prevent remote copies from reading a local controller or attempting to author the same expression state.
Development focus
The practical work was in keeping three systems compatible: immediate controller input, facial Animator transitions and speech animation. Separating the visual selector from selection logic, managing neutral transitions explicitly and gating network updates by ownership made the interaction more predictable.