AI Voice Agent: Twilio + OpenAI Realtime API Walkthrough

If you have built a chatbot and now need it to answer an actual phone number, this walkthrough is for you. Building an AI voice agent on a real telephony line is a different problem from building a text chatbot, because the caller hears every millisecond of delay and every awkward pause. This guide bridges Twilio Media Streams to the OpenAI Realtime API over WebSockets, in Node.js, with barge-in handling and tool calling included.

Specifically, you will end up with a server that answers an inbound call, streams the caller’s audio to a speech-to-speech model, streams the model’s audio back, and lets the caller interrupt mid-sentence without the agent talking over them. Along the way, this post covers the latency budget, the codec choice that removes an entire transcoding hop, and the production concerns that only surface once real people start calling.

What Is an AI Voice Agent?

An AI voice agent is a program that answers or places phone calls and holds a spoken conversation using a language model instead of a scripted IVR menu. It receives streamed caller audio, generates a spoken reply, and can call backend functions mid-conversation to look up or change real data. Unlike a chatbot, it must respond in under roughly a second to feel natural.

That last constraint drives nearly every architectural decision below. Consequently, the goal throughout this walkthrough is not just “make it work” but “make it work fast enough that a caller does not hang up.”

How the Twilio and OpenAI Realtime Architecture Fits Together

The whole system is two WebSocket connections joined in the middle by your server. Twilio owns the phone side, OpenAI owns the model side, and your process is the bridge that translates between them.

Here is the flow of a single inbound call:

1. A caller dials your Twilio phone number.
2. Twilio sends an HTTP POST to your webhook URL and waits for TwiML in response.
3. Your webhook returns <Connect><Stream>, pointing Twilio at your WebSocket endpoint.
4. Twilio opens a WebSocket to your server and begins streaming the caller’s audio as base64 JSON frames.
5. Your server opens a second WebSocket to the OpenAI Realtime API and configures the session.
6. Caller audio flows to OpenAI; generated audio flows back through your server to Twilio.
7. When the call ends, Twilio closes its socket, and your server closes the OpenAI socket.

Notably, your server never buffers a whole utterance. Instead, it forwards 20-millisecond audio frames as they arrive, which is what keeps the perceived latency low.

Why This Beats a Whisper-Plus-TTS Pipeline

The traditional approach chains three services: speech-to-text, then a chat model, then text-to-speech. That pipeline works, and it gives you more control over each stage, but it stacks three round trips before the caller hears anything.

A speech-to-speech model collapses those three hops into one. As a result, first-audio latency typically drops from a few seconds to a few hundred milliseconds. If you want the cascaded approach instead, our guides on the OpenAI Whisper API for transcription and choosing between OpenAI TTS, ElevenLabs, and Cartesia cover the two ends of that chain in detail.

There is a real trade-off here, though. Speech-to-speech gives you less visibility into the intermediate text, and swapping the voice provider means swapping the whole model. Therefore, pick it when latency matters more than modularity.

Why G.711 μ-law Removes an Entire Transcoding Step

Twilio Media Streams sends and receives audio/x-mulaw at 8000 Hz, mono, base64-encoded. Meanwhile, the OpenAI Realtime API accepts several input and output formats, including G.711 μ-law alongside 24 kHz PCM16.

Because both sides speak μ-law, you can configure the session to use it end to end and forward the base64 payload untouched. In other words, no resampling, no format conversion, and no extra buffer copies in your hot path.

The alternative is real work. Converting 8 kHz μ-law to 24 kHz PCM16 and back means decoding, upsampling, downsampling, and re-encoding every single frame, which adds both CPU load and a few milliseconds of latency per hop. Furthermore, it is a common source of subtle audio artifacts when the resampler is misconfigured.

The cost of matching formats is audio quality. Telephone-grade μ-law at 8 kHz simply carries less detail than 24 kHz PCM, so the model hears a narrower band and the caller hears a slightly duller voice. However, since the audio is going over a phone line that is band-limited anyway, you rarely gain anything by running PCM16 in the middle.

Prerequisites

You need four things before writing code:

• Node.js 20 or newer, for stable native fetch and modern ES module support
• A Twilio account with a voice-capable phone number, which you can get on the trial tier
• An OpenAI API key with Realtime API access
• A public HTTPS URL, since Twilio cannot reach localhost

For local development, ngrok is the standard answer. Start it once your server is running:

# Expose local port 5050 over a public HTTPS URL
ngrok http 5050

# Expected output:
# Forwarding  https://a1b2-203-0-113-42.ngrok-free.app -> http://localhost:5050

Then set the Twilio phone number’s “A call comes in” webhook to https://<your-ngrok-host>/incoming-call using the HTTP POST method.

Install the dependencies next:

npm install fastify @fastify/formbody @fastify/websocket ws dotenv

Fastify handles both the TwiML webhook and the WebSocket upgrade in one process, which keeps the bridge simple. Additionally, ws gives you the client socket for the outbound OpenAI connection.

Step 1: Answer the Call with TwiML

Twilio’s first contact is an ordinary HTTP request. Your job is to reply with TwiML that tells Twilio where to stream the audio.

// server.js
import 'dotenv/config';
import Fastify from 'fastify';
import fastifyFormBody from '@fastify/formbody';
import fastifyWs from '@fastify/websocket';

const fastify = Fastify({ logger: true });

// Twilio posts application/x-www-form-urlencoded, not JSON
await fastify.register(fastifyFormBody);
await fastify.register(fastifyWs);

// Twilio-supplied values land in an XML attribute, so escape them
function escapeXml(value) {
  return String(value).replace(/[<>&'"]/g, (char) => ({
    '<': '&lt;', '>': '&gt;', '&': '&amp;', "'": '&apos;', '"': '&quot;'
  }[char]));
}

fastify.all('/incoming-call', async (request, reply) => {
  // Behind ngrok or a load balancer, host is rewritten — trust the forwarded value
  const host = request.headers['x-forwarded-host'] ?? request.headers.host;
  const callerNumber = escapeXml(request.body?.From ?? 'unknown');

  const twiml = `<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://${host}/media-stream">
      <Parameter name="callerNumber" value="${callerNumber}" />
    </Stream>
  </Connect>
</Response>`;

  return reply.type('text/xml').send(twiml);
});

Why this works: <Connect><Stream> opens a bidirectional stream, which is the only variant that lets you send audio back into the call. Its sibling, <Start><Stream>, forks the audio to you one way only and is meant for transcription or recording.

Two details matter more than they look. First, Twilio blocks on <Connect> until your WebSocket closes, so any TwiML after it is unreachable during the call. Second, <Parameter> values arrive in the WebSocket start message, which is how you pass call context to the socket handler without a shared cache.

Use fastify.all rather than fastify.post here. Twilio defaults to POST, but a misconfigured console setting sends GET, and a 404 at this stage produces a silent failed call that is genuinely annoying to diagnose.

Step 2: Accept the Twilio Media Stream

Twilio now connects to /media-stream and starts sending JSON frames. There are five events you will actually see: connected, start, media, mark, and stop.

The start message carries the identifiers you need for everything else:

{
  "event": "start",
  "sequenceNumber": "1",
  "start": {
    "streamSid": "MZxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "callSid": "CAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "tracks": ["inbound"],
    "mediaFormat": { "encoding": "audio/x-mulaw", "sampleRate": 8000, "channels": 1 },
    "customParameters": { "callerNumber": "+15551234567" }
  },
  "streamSid": "MZxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

You must capture streamSid, because every message you send back to Twilio has to include it. Otherwise Twilio silently drops your audio, which looks exactly like a broken model connection.

fastify.register(async (instance) => {
  instance.get('/media-stream', { websocket: true }, (twilioWs) => {
    let streamSid = null;
    let callerNumber = 'unknown';
    let latestMediaTimestamp = 0;

    twilioWs.on('message', (raw) => {
      let msg;
      try {
        msg = JSON.parse(raw.toString());
      } catch {
        return; // Ignore malformed frames rather than killing the call
      }

      switch (msg.event) {
        case 'start':
          streamSid = msg.start.streamSid;
          callerNumber = msg.start.customParameters?.callerNumber ?? 'unknown';
          fastify.log.info({ streamSid, callerNumber }, 'stream started');
          break;

        case 'media':
          // Twilio stamps each frame with ms since stream start — used for barge-in
          latestMediaTimestamp = Number(msg.media.timestamp);
          break;

        case 'stop':
          fastify.log.info({ streamSid }, 'caller hung up');
          break;
      }
    });
  });
});

Watch the handler signature. In @fastify/websocket v11 and later, the first argument is the WebSocket. In earlier versions it was a wrapper, and you reached the socket through connection.socket. Consequently, most tutorials written before that change will throw connection.on is not a function on a current install.

Step 3: Open the OpenAI Realtime Session

With Twilio connected, open the second socket and configure it. Session configuration is a single session.update event sent right after the connection opens.

import WebSocket from 'ws';

const SYSTEM_PROMPT = `You are the after-hours line for a dental practice.
Confirm, reschedule, or cancel appointments. Keep replies under two sentences.
If the caller describes a medical emergency, tell them to hang up and call 911.
Never guess at appointment details — always use the provided tools.`;

function openRealtimeSocket() {
  return new WebSocket('wss://api.openai.com/v1/realtime?model=gpt-realtime', {
    headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` }
  });
}

function buildSessionUpdate() {
  return {
    type: 'session.update',
    session: {
      type: 'realtime',
      instructions: SYSTEM_PROMPT,
      audio: {
        input: {
          // Matches Twilio's wire format exactly — no transcoding needed
          format: { type: 'audio/pcmu' },
          turn_detection: {
            type: 'server_vad',
            threshold: 0.5,           // Raise toward 0.7 on noisy lines
            prefix_padding_ms: 300,   // Audio kept before detected speech onset
            silence_duration_ms: 500, // Silence before the turn is considered over
            interrupt_response: true  // Cancel model output when the caller speaks
          }
        },
        output: {
          format: { type: 'audio/pcmu' },
          voice: 'marin'
        }
      }
    }
  };
}

Why these settings: server_vad puts voice activity detection on OpenAI’s side, so you do not have to implement endpointing yourself. Moreover, interrupt_response is what makes natural interruption possible at the model layer — though as the next step shows, it is only half of the fix.

Tune silence_duration_ms deliberately. At 500 ms the agent feels snappy but will cut off callers who pause to think; at 900 ms it feels patient but sluggish. In practice, lines that serve older callers or non-native speakers benefit from the higher value.

Note the model and voice names are the parts most likely to drift. Check the current OpenAI Realtime API reference before you ship, since both the model list and the available voices have changed more than once.

Step 4: Bridge Audio in Both Directions

This is the core of the AI voice agent, and it is smaller than most people expect. Because the formats match, both directions are a base64 string handoff.

const openAiWs = openRealtimeSocket();

// Twilio may send audio before the model socket is ready — buffer it
const pendingAudio = [];

openAiWs.on('open', () => {
  openAiWs.send(JSON.stringify(buildSessionUpdate()));

  while (pendingAudio.length > 0) {
    openAiWs.send(JSON.stringify({
      type: 'input_audio_buffer.append',
      audio: pendingAudio.shift()
    }));
  }
});

// Caller -> model
function forwardCallerAudio(payload) {
  if (openAiWs.readyState !== WebSocket.OPEN) {
    pendingAudio.push(payload);
    return;
  }
  openAiWs.send(JSON.stringify({
    type: 'input_audio_buffer.append',
    audio: payload // Already base64 mulaw from Twilio
  }));
}

// Model -> caller
openAiWs.on('message', (raw) => {
  const evt = JSON.parse(raw.toString());

  if (evt.type === 'error') {
    fastify.log.error({ err: evt.error }, 'realtime api error');
    return;
  }

  if (evt.type === 'response.output_audio.delta' && evt.delta) {
    twilioWs.send(JSON.stringify({
      event: 'media',
      streamSid,
      media: { payload: evt.delta }
    }));
  }
});

Why the buffer matters: Twilio starts streaming the moment the call connects, but the OpenAI socket takes a few hundred milliseconds to establish. Without pendingAudio, the caller’s opening “Hi, I’d like to—” is dropped, and the agent responds to a half-sentence.

The event name is worth flagging. Audio deltas arrive as response.output_audio.delta in the current API; the earlier beta used response.audio.delta. If you are following an older sample and hear silence despite a healthy socket, log the raw event types first — the bug is almost always a renamed event rather than a broken bridge.

Greeting the Caller First

By default the agent waits for the caller to speak, which produces an unsettling silent answer. Trigger an opening line explicitly instead:

function sendGreeting() {
  openAiWs.send(JSON.stringify({
    type: 'conversation.item.create',
    item: {
      type: 'message',
      role: 'user',
      content: [{
        type: 'input_text',
        text: 'The caller just connected. Greet them briefly and ask how you can help.'
      }]
    }
  }));
  openAiWs.send(JSON.stringify({ type: 'response.create' }));
}

Call this after session.updated arrives rather than immediately after open. Otherwise the greeting can be generated with the default voice and format, before your configuration takes effect.

Step 5: Handle Barge-In Without Talking Over the Caller

Here is the failure that makes an otherwise working agent feel broken. When the caller interrupts, the model stops generating — but Twilio has already buffered several seconds of audio and keeps playing it. The caller talks, the agent keeps talking, and neither can hear the other.

Fixing it requires three coordinated actions:

1. Tell Twilio to drop its buffer with a clear message.
2. Tell the model how much audio the caller actually heard with conversation.item.truncate.
3. Reset your local playback tracking so the next turn starts clean.

Step two is the one people skip, and skipping it corrupts the conversation history. The model believes it said everything it generated, so it will not repeat the part the caller never heard.

let lastAssistantItem = null;
let responseStartTimestamp = null;
const markQueue = [];

openAiWs.on('message', (raw) => {
  const evt = JSON.parse(raw.toString());

  if (evt.type === 'response.output_audio.delta' && evt.delta) {
    // First delta of a response anchors the playback clock
    if (responseStartTimestamp === null) {
      responseStartTimestamp = latestMediaTimestamp;
    }
    if (evt.item_id) lastAssistantItem = evt.item_id;

    twilioWs.send(JSON.stringify({
      event: 'media', streamSid, media: { payload: evt.delta }
    }));
    sendMark();
  }

  // Server VAD detected the caller starting to speak
  if (evt.type === 'input_audio_buffer.speech_started') {
    handleBargeIn();
  }
});

function sendMark() {
  twilioWs.send(JSON.stringify({
    event: 'mark', streamSid, mark: { name: 'chunk' }
  }));
  markQueue.push('chunk');
}

function handleBargeIn() {
  // Nothing is playing, so there is nothing to interrupt
  if (markQueue.length === 0 || responseStartTimestamp === null) return;

  const heardMs = Math.max(latestMediaTimestamp - responseStartTimestamp, 0);

  if (lastAssistantItem) {
    openAiWs.send(JSON.stringify({
      type: 'conversation.item.truncate',
      item_id: lastAssistantItem,
      content_index: 0,
      audio_end_ms: heardMs // Trim history to what the caller actually heard
    }));
  }

  twilioWs.send(JSON.stringify({ event: 'clear', streamSid }));

  markQueue.length = 0;
  lastAssistantItem = null;
  responseStartTimestamp = null;
}

Why marks are the right signal: Twilio echoes each mark back once the corresponding audio finishes playing. Therefore, a non-empty markQueue means audio is still in flight, which is a far more reliable “is the agent speaking?” check than tracking model events alone.

The latestMediaTimestamp value comes from the inbound media frames you captured in step two. Because Twilio stamps those in milliseconds since stream start, subtracting the response start gives you a close approximation of playback position without any clock of your own.

Test this by interrupting mid-sentence on a real call. If the agent keeps going for two or three more seconds, the clear message is not reaching Twilio — and nine times out of ten, streamSid is null.

Step 6: Give the Agent Tools It Can Actually Call

An AI voice agent that cannot read your database is a very expensive answering machine. Tool calling is what turns it into something useful, and the Realtime API handles it much like the Chat Completions API does.

Declare the tools in your session configuration:

const tools = [
  {
    type: 'function',
    name: 'lookup_appointment',
    description: 'Find an upcoming appointment by the caller phone number.',
    parameters: {
      type: 'object',
      properties: {
        phone_number: { type: 'string', description: 'E.164 format, e.g. +15551234567' }
      },
      required: ['phone_number']
    }
  },
  {
    type: 'function',
    name: 'cancel_appointment',
    description: 'Cancel a confirmed appointment. Confirm with the caller before calling.',
    parameters: {
      type: 'object',
      properties: { appointment_id: { type: 'string' } },
      required: ['appointment_id']
    }
  }
];

Then handle the call-and-return cycle. The model streams argument fragments and signals completion with a done event:

const toolHandlers = {
  lookup_appointment: async ({ phone_number }) => {
    const appointment = await db.appointments.findUpcoming(phone_number);
    return appointment
      ? { found: true, id: appointment.id, startsAt: appointment.startsAt }
      : { found: false };
  },
  cancel_appointment: async ({ appointment_id }) => {
    await db.appointments.cancel(appointment_id);
    return { cancelled: true };
  }
};

async function handleToolCall(evt) {
  const handler = toolHandlers[evt.name];
  let output;

  try {
    output = handler
      ? await handler(JSON.parse(evt.arguments))
      : { error: `Unknown tool: ${evt.name}` };
  } catch (error) {
    // Return the failure to the model so it can apologise, not crash the call
    fastify.log.error({ err: error, tool: evt.name }, 'tool failed');
    output = { error: 'Lookup failed. Ask the caller to try again shortly.' };
  }

  openAiWs.send(JSON.stringify({
    type: 'conversation.item.create',
    item: {
      type: 'function_call_output',
      call_id: evt.call_id,
      output: JSON.stringify(output)
    }
  }));

  // The model does not speak again until you ask it to
  openAiWs.send(JSON.stringify({ type: 'response.create' }));
}

The critical line is the last one. Submitting function_call_output does not automatically produce a reply. Without the follow-up response.create, the tool runs, the result lands, and the caller hears nothing but silence.

Two rules keep tool calls from ruining the call. Keep every handler under about 500 ms, because dead air on a phone line feels much longer than a spinner in a web app. Additionally, always return errors as structured output rather than throwing, so the agent can say “I’m having trouble looking that up” instead of dropping the call.

If you are designing a larger tool surface, our breakdown of how AI agents combine tools, planning, and execution covers the patterns that keep tool sets maintainable as they grow past a handful of functions.

Latency Budget: Where the Milliseconds Actually Go

Perceived responsiveness is the whole product. The table below shows the rough shape of a single turn on a healthy connection, from the caller finishing a sentence to the first audio playing back.

Stage	Typical contribution	Can you reduce it?
PSTN and Twilio ingress	50–100 ms	No — carrier network
End-of-turn detection (server VAD)	300–700 ms	Yes — lower silence_duration_ms
Model time to first audio token	300–600 ms	Partly — shorter system prompt
Tool call round trip, if any	0–500 ms	Yes — cache and index aggressively
Your server’s bridge overhead	Under 10 ms	Already negligible
Twilio egress and playback	50–100 ms	No — carrier network

These are order-of-magnitude figures for a well-placed server, not measurements from a benchmark. Your own numbers will depend on region, network path, and prompt size, so measure before optimising.

Two conclusions follow. First, VAD silence duration is the single largest knob you control, which is why it deserves real tuning rather than a copied default. Second, your bridge code is not the bottleneck, so resist the urge to micro-optimise the WebSocket handler.

Region placement is the other lever worth pulling. Running your bridge in a region far from both Twilio’s edge and the OpenAI endpoint can add 100 ms or more in each direction, and that penalty applies to every single frame.

Production Concerns Before You Point a Real Number at It

Local testing hides most of what breaks in production. These are the items worth handling before real callers arrive.

Reconnects and Cleanup

Close the OpenAI socket whenever the Twilio socket closes, and vice versa. Otherwise, orphaned model sessions accumulate and quietly bill you for connections nobody is listening to.

twilioWs.on('close', () => {
  if (openAiWs.readyState === WebSocket.OPEN) openAiWs.close();
});

openAiWs.on('close', () => {
  if (twilioWs.readyState === WebSocket.OPEN) twilioWs.close();
});

openAiWs.on('error', (error) => {
  fastify.log.error({ err: error }, 'realtime socket error');
  if (twilioWs.readyState === WebSocket.OPEN) twilioWs.close();
});

Reconnecting mid-call is generally not worth it. Since a new session loses the conversation state anyway, failing over to a human or a voicemail prompt gives the caller a better experience than a confused agent restarting from nothing.

Cost Control

Realtime audio is billed per minute of input and output audio, and it is substantially more expensive than text tokens. As a result, an agent that rambles costs materially more than one that does not.

Three controls help. Cap call duration server-side and close the socket with a polite handoff. Instruct the model to keep replies short, because verbose output is billed audio. Finally, log per-call duration so you can spot the outliers, since a handful of stuck calls usually explains a surprising invoice.

Observability

Log the transcript events, not just the audio flow. Enabling input transcription in the session gives you conversation.item.input_audio_transcription.completed events, which are the only readable record of what the caller actually said.

Store four things per call at minimum: callSid, full transcript, every tool call with its arguments and result, and total duration. Consequently, when someone reports “the agent told me the wrong appointment time,” you can actually check.

Security

Validate that inbound webhook requests genuinely come from Twilio using the X-Twilio-Signature header. Without that check, anyone who discovers your endpoint can trigger calls against your OpenAI key.

Treat the caller’s speech as untrusted input, too. A caller can read a prompt injection out loud just as easily as typing it, so tool handlers must enforce their own authorisation rather than trusting that the model asked correctly.

Real-World Scenario: The After-Hours Clinic Line

Consider a small dental practice with two receptionists and a phone line that goes to voicemail after 5 p.m. Roughly a third of after-hours voicemails are simple appointment confirmations or cancellations, and each one still costs a receptionist a callback the next morning.

An AI voice agent handling only those two intents changes the economics without touching anything clinical. The agent looks up the appointment by caller ID, confirms or cancels it, and escalates everything else to voicemail. Importantly, the scope stays narrow enough that the failure modes are predictable.

During a rollout like this, the problems that surface are rarely the ones teams prepare for. Barge-in handling matters more than expected, because callers interrupt constantly once they realise they are talking to a machine. Meanwhile, VAD tuning turns out to be the difference between an agent that feels attentive and one that talks over people, and the right value depends on the caller demographic rather than on any documented default.

The honest limitation is scope creep. Every stakeholder wants “just one more” intent added, and each addition widens the surface where the agent can confidently say something wrong. Keeping the tool list short and the escalation path obvious is what keeps a narrow agent trustworthy over a period of months.

When to Use an AI Voice Agent

• Inbound calls follow a small number of repetitive, well-defined intents
• The caller’s goal maps cleanly onto data you already expose through an API
• Call volume is high enough that deflection saves meaningful staff time
• A wrong answer is inconvenient rather than dangerous or expensive
• You can offer an obvious path to a human within the first few seconds

When NOT to Use an AI Voice Agent

• The conversation involves medical, legal, or financial advice with real liability
• Callers are frequently distressed, since a machine voice makes that worse
• Your backend cannot answer lookups in well under a second
• Regulations in your industry require a disclosed human agent
• The intent space is genuinely open-ended, where a well-built form or callback works better

Common Mistakes with AI Voice Agents

• Ignoring barge-in. Twilio keeps playing buffered audio after the model stops. Without a clear message, the agent talks over the caller.
• Forgetting conversation.item.truncate. The conversation history then contains audio the caller never heard, so the model will not repeat itself when asked.
• Dropping streamSid from outbound messages. Twilio discards them silently, which looks identical to a dead model connection.
• Skipping response.create after a tool result. The tool succeeds, and the caller hears silence.
• Transcoding audio unnecessarily. Configuring the session for μ-law on both sides removes the conversion entirely.
• Buffering whole utterances before forwarding. This defeats the streaming architecture and adds hundreds of milliseconds per turn.
• Leaving silence_duration_ms at the default. The right value depends on your callers, not on the documentation.
• No maximum call duration. One stuck session can run for hours on your account.

Where to Go Next

An AI voice agent built on Twilio Media Streams and the OpenAI Realtime API comes down to two WebSockets, a matched μ-law audio format, and disciplined interruption handling. The bridge code itself is short; the work is in tuning VAD, keeping tool calls fast, and building the escalation path that protects callers when the agent reaches its limits.

Start narrow. Ship one intent, point a test number at it, and call it yourself twenty times before anyone else does — most of the defects above show up in the first five calls.

From here, two directions are worth exploring. If you want to compare architectures before committing, building voice agents with the Gemini Live API in Python covers an equivalent speech-to-speech stack with a different set of trade-offs. If you would rather deepen the conversational layer first, our walkthrough of building an AI customer support bot covers intent handling and escalation design in text, where iteration is far cheaper. For the transport decisions underneath both, streaming LLM responses over SSE versus WebSockets explains why realtime audio leaves you no real choice.