Digital avatar systems — AI characters that see, hear, and respond in real time — sit at the intersection of three hard problems: low-latency streaming, GPU resource management, and synchronized multi-modal output. Getting any one of these wrong produces an uncanny, unusable experience.
The pipeline looks deceptively simple on paper: browser captures audio → STT converts to text → LLM generates response → TTS converts to speech → audio plays while avatar lip-syncs. In practice, every step in that chain adds latency, and the human perception threshold for unnatural conversation delay is around 300ms. You don't have much headroom.
WebRTC is the right transport layer for this workload. It's designed for real-time media, handles NAT traversal via STUN/TURN servers, and gives you direct peer-to-peer paths that bypass application servers after the handshake. We used it to stream both the rendered avatar video and the audio track from the rendering server directly to the browser, cutting out an entire relay hop.
GPU instances are expensive and stateful. You can't cold-start a GPU rendering process per request — model loading alone takes several seconds. The solution is a warm pool: maintain N pre-warmed instances with the avatar model loaded, and assign sessions to instances on connection. When a session ends, the instance returns to the pool rather than shutting down. The tricky part is session affinity: all packets for a session must route to the same instance, which means your load balancer needs sticky sessions based on session ID, not round-robin.
Blend shape synchronization is where most teams stumble. LLMs generate text in chunks (streaming completions), TTS converts chunks to audio incrementally, and the avatar must lip-sync to each audio segment as it plays — not after the full response is assembled. This requires a tight feedback loop: TTS returns audio and phoneme timestamps together, and the rendering engine consumes phoneme events to drive blend shape weights frame by frame. Any desync between audio and visual creates the uncanny valley effect immediately.
Latency budget allocation: STT 50-100ms, LLM first-token 200-400ms (streaming), TTS per-chunk 80-150ms, WebRTC transit 20-50ms. Total to first audio: ~400-700ms, which is acceptable. The perceived latency is lower because the avatar shows a thinking animation while the first chunk is being generated — matching user expectations from human conversation.