You built a voice agent with Vapi and ElevenLabs, tested it once, and it sounded great. Then it went live and started stumbling: long pauses before it answers, garbled audio, or calls that just drop. None of that means you picked the wrong tools. It almost always means a handful of settings need tuning.
Here's what actually causes a slow or broken Vapi voice agent, and the specific fixes that work.
Why Does My Vapi Voice Agent Feel Slow or Laggy?
Every voice agent turn has to do four things in sequence: hear you (speech-to-text), think of a reply (the LLM), speak it (text-to-speech), and send the audio back over the phone line. Add those up and even a well-built stack can hit 1.5 to 2 seconds of dead air before the agent responds — long enough that callers start talking over it or hang up, assuming the line dropped.
The good news: most of that delay isn't the AI "thinking." It's waiting — waiting for a full transcript before the LLM starts, waiting for the whole reply before the voice starts speaking, waiting on a default pause setting that's longer than it needs to be. Fix the waiting, and the agent feels dramatically faster without changing a single model.
What's a Good Latency Target for a Voice Agent?
Aim for under 500 milliseconds of response time on simple conversational turns, and 700–800 milliseconds on turns where the agent has to call a tool (like checking a calendar or looking up an order). Anything consistently past a second starts to feel unnatural to callers, even if they can't say exactly why.
How Do You Fix Vapi Voice Agent Latency?
These are the changes that make the biggest difference, roughly in order of impact:
- Switch to streaming speech-to-text. Instead of waiting for the caller to finish an entire sentence, streaming STT sends partial transcripts every 100–200ms. Let the LLM start forming a response the moment intent is clear, rather than after the caller stops talking. This alone can save 200–400ms per turn.
- Stream LLM output straight into text-to-speech. Don't wait for the full reply to generate before speaking it — start the voice as soon as the first sentence is ready. This typically saves another 200–500ms.
- Cache your prompt prefix. If your system prompt is long, ask your LLM provider about prompt caching. It avoids reprocessing the same instructions on every single turn, saving 200–400ms on time-to-first-token.
- Make tool calls asynchronous. When the agent needs to hit your server (booking a slot, pulling a record), set that call to run without blocking the spoken response. Vapi can fire the request and keep talking, rather than going silent while it waits.
- Tune the
startSpeakingPlansettings. Vapi's default wait times before the agent starts speaking can add well over a second of unnecessary pause. Shortening these is one of the fastest wins available and takes minutes to change. - Run evaluations and logging asynchronously. Anything that scores or logs the call should happen after the response is sent, never inline before it.
Stacked together, these changes commonly shave 600–900ms off the slowest 5% of your calls — the difference between an agent that feels instant and one that feels like it's thinking too hard.
Common Vapi and ElevenLabs Errors (And How to Fix Them)
Most "broken" voice agents aren't a latency problem at all — they're one of a handful of recurring errors. Here's what each one actually means:
| Symptom | Likely Cause | Fix |
|---|---|---|
| Audio generation fails, but ElevenLabs shows as operational | Voice ID issue — the voice you're calling has been deprecated or removed from your account, or the input text has unusual characters | Re-select the voice inside your ElevenLabs library and re-sync it in Vapi's Integrations tab; strip unusual characters or very long inputs from the prompt |
| 401 Unauthorized error | Your ElevenLabs API key is missing, expired, or was rotated | Generate a fresh key from ElevenLabs' Profile + Keys page and re-enter it under Vapi's ElevenLabs integration settings |
| "Rate limit exceeded" mid-call | You've hit your ElevenLabs character limit for the billing period | Check usage in your ElevenLabs dashboard and upgrade the plan, or cap non-essential test calls |
| Webhook never receives call data | Incorrect endpoint URL, or a local dev server that isn't publicly reachable | Confirm the webhook URL in Vapi points to your live endpoint; if testing locally, use a tunnel (like ngrok) and update the webhook to the tunnel's public URL |
If you're not sure which of these is happening, Vapi's own debugging dashboard logs each call's transcript, tool calls, and timing breakdown — check there first before assuming it's a code problem.
What Does a Vapi Voice Agent Actually Cost Per Minute?
Vapi advertises a $0.05-per-minute starting rate, but that number only covers orchestration — it doesn't include the AI services doing the actual work. You still pay separately for speech-to-text (Deepgram or AssemblyAI), the LLM (OpenAI or Anthropic), text-to-speech (ElevenLabs), and telephony (usually Twilio).
Once you add those up, a lean stack — a smaller LLM, Deepgram, and ElevenLabs' faster Turbo voice model — runs around $0.15 per minute. A premium stack with a larger LLM and ElevenLabs' higher-quality multilingual voices can run $0.35–$0.40 per minute. Budget for the full stack cost, not just Vapi's advertised rate, when you're pricing this out for a client.
The Most Common Mistake When Setting Up a Voice Agent
Testing only in a quiet room on a stable connection, then launching straight to real phone calls. Real callers have background noise, accents your STT wasn't tuned for, and spotty cell signal — all of which surface latency and transcription issues that never show up in a clean demo. Test with a few real, imperfect phone calls before you hand a voice agent to a client or put it live on a business line.
Frequently Asked Questions
Why does my Vapi agent pause for a long time before answering?
This is almost always the startSpeakingPlan default wait settings combined with non-streaming speech-to-text. Switching to streaming STT and shortening the start-speaking wait time typically fixes most of the delay.
Why does my ElevenLabs voice sound garbled or fail to generate?
This usually means the voice ID being called no longer exists in your account, or the API is receiving text with unusual formatting or characters. Re-select and re-sync the voice in Vapi's integration settings, and clean up the input text.
How much does a Vapi and ElevenLabs voice agent cost to run?
Vapi's own orchestration fee starts at $0.05 per minute, but total cost including speech-to-text, the LLM, ElevenLabs voice, and telephony typically lands between $0.15 and $0.40 per minute depending on which models you choose.
What's a normal response latency for a voice agent?
Aim for under 500 milliseconds on simple turns and 700–800 milliseconds on turns that require a tool call, like a calendar lookup. Anything routinely over a second will feel unnatural to callers.
Why isn't my webhook receiving data from Vapi or ElevenLabs?
Usually the endpoint URL is wrong or points to a local server that isn't publicly reachable. If you're testing locally, use a tunneling tool and update the webhook URL to the tunnel's public address.
Do I need to be a developer to fix these issues myself?
No. Most of these fixes — re-entering an API key, re-syncing a voice, adjusting a wait-time setting, updating a webhook URL — are done through Vapi's and ElevenLabs' dashboards, not code. A bit of technical comfort helps, but you don't need to write software to resolve them.
Bottom Line
A laggy or error-prone voice agent is rarely a sign you need a different platform — it's almost always a handful of default settings that were never tuned. Start with streaming STT, trim the start-speaking wait time, and double-check your API keys and webhook URLs. Test with real, imperfect phone calls before launch, and budget for the full per-minute stack cost, not just Vapi's advertised rate.
