---
id: ai-voice-realtime-production
title: Voice Agent Realtime — OpenAI / Anthropic / livekit
category: Coding
status: draft
source_trust_level: B
verification_status: conceptual
created_at: 2026-05-09
updated_at: 2026-05-09
tags: [ai, voice, realtime, vibe-coding]
tech_stack: { language: "TS", applicable_to: ["AI"] }
applied_in: []
aliases: [voice agent, realtime API, OpenAI realtime, Anthropic voice, LiveKit, WebRTC, low latency speech]
---

# Voice Agent Realtime

> Voice → LLM → voice 가 < 1 sec. **OpenAI Realtime API, Anthropic, LiveKit Agents**. WebRTC + speech model.

## 📖 핵심 개념
- VAD (Voice Activity Detection).
- Speech-to-text (STT) + text-to-speech (TTS).
- Or speech-to-speech (S2S) native.
- Latency < 800 ms 가 conversational.

## 💻 코드 패턴

### OpenAI Realtime API
```ts
const ws = new WebSocket('wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview', {
  headers: { 'Authorization': `Bearer ${apiKey}`, 'OpenAI-Beta': 'realtime=v1' }
});

ws.on('open', () => {
  ws.send(JSON.stringify({
    type: 'session.update',
    session: {
      voice: 'alloy',
      instructions: 'You are a helpful voice assistant.',
      input_audio_format: 'pcm16',
      output_audio_format: 'pcm16',
      turn_detection: { type: 'server_vad' },
    },
  }));
});

// Send audio chunk
ws.send(JSON.stringify({
  type: 'input_audio_buffer.append',
  audio: base64PcmChunk,
}));

// Receive
ws.on('message', (data) => {
  const event = JSON.parse(data);
  if (event.type === 'response.audio.delta') {
    playAudio(event.delta);  // base64 PCM
  }
});
```

→ Bidi WebSocket. 매우 낮은 latency.

### LiveKit Agents
```python
from livekit.agents import Agent, AgentSession, WorkerOptions, cli
from livekit.plugins import openai, deepgram, elevenlabs

async def entrypoint(ctx: JobContext):
    agent = Agent(
        instructions='You are a helpful agent.',
        stt=deepgram.STT(),
        llm=openai.LLM(model='gpt-4o'),
        tts=elevenlabs.TTS(voice_id='...'),
    )
    
    session = AgentSession()
    await session.start(agent=agent, room=ctx.room)

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
```

→ STT + LLM + TTS pipeline.

### WebRTC + LiveKit
```ts
import { Room } from 'livekit-client';

const room = new Room();
await room.connect(url, token);

// Publish microphone
await room.localParticipant.setMicrophoneEnabled(true);

// Receive agent voice
room.on('trackSubscribed', (track, pub, participant) => {
  if (track.kind === 'audio') {
    track.attach();  // play audio.
  }
});
```

→ Multi-participant + agent.

### VAD (server vs client)
```
Client VAD:
- Microphone 가 silence detect.
- 작은 cost.
- 부정확.

Server VAD (OpenAI):
- 매 chunk send.
- 정확.
- Bandwidth 큰.
```

### Turn detection
```
"User 가 끝났나?"
- Silence 500-1500 ms.
- 또는 LLM 가 자체 (semantic VAD).

→ 너무 짧 = 사용자 가 안 끝남.
너무 길 = 어색.
```

### Interrupt (사용자 가 끼어듦)
```ts
// Agent 가 talking.
// User 가 말 = agent 가 stop + listen.

if (vadDetectsSpeech()) {
  ws.send(JSON.stringify({ type: 'response.cancel' }));
}
```

→ Conversational UX 의 핵심.

### Latency budget
```
End-to-end < 800 ms 가 natural.

Components:
- VAD: 100 ms.
- Network (WebSocket): 50-200 ms.
- STT: 100-300 ms.
- LLM first token: 200-500 ms.
- TTS first chunk: 100-300 ms.

→ Optimize 매 step.
```

### Speech-to-speech (S2S)
```
OpenAI Realtime: native S2S (no STT/TTS separately).
Gemini 2: 비슷.

→ 더 자연스러운 (intonation 보존).
하지만 비싼.
```

### STT (separate)
```
- Deepgram (가장 빠름, low latency).
- AssemblyAI.
- OpenAI Whisper (open).
- Google Speech-to-Text.

→ Streaming STT 가 important.
```

### TTS (separate)
```
- ElevenLabs (가장 자연).
- OpenAI TTS.
- Cartesia (low latency).
- Azure Neural Voices.
- Google Cloud TTS.

→ Voice cloning (ElevenLabs) 가 brand voice.
```

### Use case
```
- Customer support (Twilio + voice agent).
- Phone call automation.
- Voice assistant (Alexa-like).
- Driving (hands-free).
- Real-time translation.
- Game NPC.
- Therapy / coaching.
```

### Twilio integration
```ts
// Twilio call → WebSocket → agent
const ws = new WebSocket(twilioStreamUrl);
ws.on('message', (audio) => {
  // Forward to OpenAI Realtime.
});
```

→ Phone number + AI 가 답.

### Cost
```
OpenAI Realtime:
- $5 / 1M input audio token.
- $20 / 1M output audio token.
- ~$0.30 / minute.

Deepgram + GPT-4 + ElevenLabs:
- ~$0.10 / minute.

LiveKit (self-host):
- Compute cost.

→ Volume = ~$0.10-0.30 / call minute.
```

### Function calling (voice)
```
사용자: "내 잔액 확인".
Agent:
1. STT → "What's my balance?"
2. LLM 가 function call: get_balance().
3. Response: "$1234".
4. TTS → audio.
```

### Memory
```
Long conversation = context grow.
- Sliding window.
- Summary.
- Per-user persistent.

→ [[AI_Memory_Persistence_Deep]].
```

### Background noise
```
- Noise suppression (Krisp, RNNoise).
- Echo cancellation (WebRTC built-in).
- Music handling (filter).

→ Quality 의 큰 lever.
```

### Multi-language
```
- LLM 의 multilingual.
- TTS 의 voice 언어 별.
- STT 가 언어 detect.
```

### Production
- **Sierra** (customer support).
- **PolyAI** (call center).
- **OpenAI ChatGPT voice mode**.
- **Twilio + voice agents**.
- **Hume AI** (emotion).

### 함정
```
- High latency (>1.5 sec) = bad UX.
- Interrupt 안 = user frustrated.
- Memory leak (long call).
- TTS voice 가 robotic.
- Cost (long call = $$).
- Privacy (recording / store).
```

### Mobile
```
WebRTC 가 native iOS / Android.
- LiveKit SDK.
- 매 platform 의 audio session.
- Background audio mode.
```

### Browser
```
WebRTC + getUserMedia.
- HTTPS only.
- User permission.
- Echo cancellation 자동.
```

### Eval
```
- Latency p99.
- Interruption handling.
- Word accuracy (WER).
- User satisfaction.
- Conversation completion rate.

→ Real-world test.
```

## 🤔 의사결정 기준
| 상황 | 추천 |
|---|---|
| Quick prototype | OpenAI Realtime |
| Phone call | Twilio + agent |
| Multi-user | LiveKit Agents |
| Self-host | Deepgram + LLM + ElevenLabs |
| Native voice | OpenAI Realtime / Gemini |
| Voice cloning | ElevenLabs |
| Open / privacy | Whisper + local LLM + Coqui TTS |

## ❌ 안티패턴
- **Latency > 1.5 sec**: bad UX.
- **No interrupt**: frustrating.
- **No VAD**: cost / latency.
- **TTS robot**: ElevenLabs / Cartesia.
- **No tracing**: debug 불가.
- **Privacy 무시**: recording leak.

## 🤖 LLM 활용 힌트
- OpenAI Realtime 가 native S2S.
- LiveKit Agents 가 production framework.
- < 800 ms latency 가 natural.
- Interrupt + VAD 가 핵심.

## 🔗 관련 문서
- [[AI_Voice_Agent_Realtime]]
- [[AI_Voice_Cloning_Synthesis]]
- [[Web_WebRTC_Realtime]]
