AI-driven virtual human
Connecting personality, voice and body language in real time.
I built a conversational virtual human by connecting a personality-conditioned language model, Hume’s voice pipeline and a character in Unity. I wanted the character’s words, voice and non-verbal behaviour to respond consistently during a live conversation.
One response, two connected paths
- Unity microphone → Hume EVI
- Recognised speech → Python service → personality-conditioned LLM
- Spoken text → Hume voice → Unity audio + lip sync
- Non-verbal cues → WebSocket → Unity animation + gaze
Giving the model a personality
Each persona starts with a 300-item IPIP-NEO personality profile stored as JSON. An instructor model converts this into a behavioural brief, which is cached. The dialogue model receives the full profile, the brief, conversation history, available affect context and a personality-specific vocabulary of non-verbal cues. Prosocial, neutral and antisocial conditions therefore influence both what the character says and how it behaves.
Wiring the live conversation
Unity sends microphone PCM audio to Hume EVI over a WebSocket. Hume handles the speech input and calls a Python FastAPI custom-language-model endpoint through a public tunnel. That service prepares the personality-conditioned prompt and streams the model’s response back as server-sent events. Hume synthesises the spoken reply, and Unity queues the returned audio for playback through an AudioSource. uLipSync uses the speech audio to drive the mouth.
Separating speech from body language
The model emits spoken dialogue alongside structured gesture, posture and facial-expression cues. The Python service removes those cue tokens from the text sent for speech synthesis and sends them separately to Unity over a local /unity WebSocket. A cue registry and dispatcher map the cue names onto authored animation states, expression controls and gaze behaviour. The LLM chooses behaviours; it does not generate animation curves or a new character rig.
Coordinating animation with speech
A turn_start message provides leading cues and a turn_cues message carries the fuller timeline. Leading actions are deduplicated by turn ID. Mid-turn timing is estimated from word rate, while trailing behaviour can wait for the actual audio queue to drain. Gestures blend in and out; posture can persist across speech; additive idle motion keeps the character from becoming completely static. This is approximate speech–gesture alignment, not phoneme-accurate motion synchronisation.
Making the systems coexist
Lip sync and facial expressions can fight over the same blendshapes. I added mouth-region attenuation during speech so expression can remain visible around the brows and eyes without overwhelming articulation, and addressed a jaw-driver conflict on the imported rig. Interruption handling clears queued audio and cancels pending cues, while repeat guards reduce duplicate behaviour.
Changing personality without reconnecting
The Unity personality selector communicates with the Python service through GET and POST /personality. The next turn uses the selected profile without tearing down the Hume connection. Keeping session transport separate from persona selection makes it possible to compare character behaviour within the same running application.
Built using third-party language, speech and character technologies. My work is the application logic and integration that connects them into a live agent.