Simultaneous Interpretation for Video Calls: How It Actually Works
On a video call, simultaneous interpretation is a short pipeline running continuously in the background: audio in, transcribed speech, translated text, and captions or synthesized speech out — fast enough that it never asks the conversation to pause and wait for it.
The pipeline, step by step
Four things happen, in sequence, for every chunk of speech: the audio is captured from a microphone or the call itself; a speech-recognition model turns that audio into text in the source language; a translation model turns that text into the target language; and the result is displayed as captions or, less commonly, spoken aloud with synthesized speech. Each step adds latency, so the practical engineering problem is keeping the total under a few seconds without sacrificing accuracy at any stage.
This is different from how a call platform's built-in "live captions" usually works, which typically only does the first two steps — speech to text, in the same language. Translation is an extra step with its own latency and its own failure modes (a mistranslated term is a different kind of error than a misheard one).
Two integration models
There are two broad ways this gets attached to a call. The first is a bot that joins the meeting as a participant — visible in the participant list, often with its own video tile — which works on any platform that allows guest participants but means everyone on the call can see an unfamiliar entry join. The second is device-audio capture: the tool runs on one person's device and listens to whatever's playing through their speakers or microphone, without joining the call as a separate participant at all.
The second approach trades a small amount of setup friction (it only sees what one participant's device can hear) for not requiring the call host to admit anything, and for working identically on any platform since it isn't a bot integrating with a specific video API.
What to expect from latency
A useful mental model: under 2 seconds feels like reading fast captions in the same conversation; 2–5 seconds is still very usable for following along, just slightly behind; beyond that, it starts to feel like reading a transcript of something that already happened. Systems tuned for accuracy over raw speed tend to sit in the 2–5 second range deliberately, since committing to a translation too early is the more common source of visibly wrong output.
Be first to try real-time interpretation yourself.
In active testing — join the waitlist and we'll email you when it's ready.
Read next
How Fast Does Real-Time Translation Need to Be? A Latency Guide
Latency in real-time translation isn't just a speed metric — it's the difference between a tool that feels like part of the conversation and one that feels like reading minutes from a meeting that already ended.
Do You Need a Bot on the Call? Device-Audio vs. Bot-Based Translation Tools
A bot that joins your call as a participant needs to be admitted by the host and shows up for everyone; a tool that listens through one person's device audio needs neither, at the cost of only hearing what that one device can hear.
What Is Real-Time Speech Translation, and How Is It Different From an Interpreter App?
Real-time speech translation listens to speech as it's spoken and produces translated text (and often audio) within a few seconds — continuously, not after you finish a sentence and hit a button.