Measurements · August 2026
Not a vendor's press release and not a figure we rounded in our favour. Each row below comes from audio we actually ran, on public datasets anyone can download, with the sample size, the sampling seed and the failure cases printed next to the result. Where we don't have the evidence yet, the row says so instead of guessing.
Run on 8–9 August 2026 and re-run whenever the pipeline changes — when a number here moves, it is because we measured it again, not because we rewrote the copy. Everything is corpus-level word error rate: total word errors over total reference words. Lower is better.
Tier 1 · Measured on real audio
Google FLEURS, Multilingual LibriSpeech and the AMI Meeting Corpus. All three are published under Creative Commons licences, so we can tell you exactly which clips we used and you can fetch the same ones. These are the numbers we are willing to be held to.
| Condition | clips | min | cohere-transcribe | Qwen3-ASR 1.7B | Scribe v2 | whisperX |
|---|---|---|---|---|---|---|
| FLEURS es-419 — Latin American Spanish | 224 | 45.1 | 4.9 | 3.9 | 3.8 | 3.4 |
| FLEURS pt-BR — Brazilian Portuguese beta | 210 | 45.2 | 6.8 | 6.1 | 5.4 | 5.7 |
| FLEURS en-US — English | 277 | 45.1 | 6.0 | 4.9 | 4.8 | 5.7 |
| MLS Spanish — audiobook narration | 177 | 45.1 | 3.4 | 5.2 | 2.8 | 4.1 |
| MLS Portuguese — audiobook, European pt beta | 176 | 45.2 | 11.9 | 7.7 | 3.8 | 6.8 |
| AMI-SDM — English, real meeting room, single far mic | 194 | 18.0 | 12.8 | 18.4 | 19.8 | 23.9 |
Two hosted engines from our routing table did not complete this round; their rows will be filled in when they do. We would rather show you a gap than a placeholder number.
12.8%
AMI-SDM is a real meeting recorded by a single microphone sitting on a table, several people, no headsets — the condition Sinsonte is built for and the one benchmarks usually skip. On 194 clips (18 minutes) cohere-transcribe scores 12.8% word error rate; the other three engines land between 18.4% and 23.9%. That gap, on real room audio, is the clearest result we have.
Caveat that travels with it: AMI is English. We have no real far-field Spanish recording yet, so we make no far-field Spanish claim — see “What we don't claim” below.
A note on reading these: no single engine wins everything. whisperX — the open baseline we hold ourselves against — is the best engine on Spanish read speech. Scribe v2 wins four rows. cohere-transcribe is best in the room and worst on European Portuguese. That is exactly why Sinsonte routes each recording to a different engine by language and condition rather than picking a favourite.
Tier 2 · Measured, in validation
Read this first: the audio behind every number in this section is synthetic — text-to-speech we generated ourselves, because no public dataset of Spanish–English or Portuguese–Spanish conversational speech exists that we are licensed to publish results from. Synthetic speech is cleaner, more evenly paced and more clearly articulated than a real bilingual conversation. Treat these as a best case with the room noise removed, not as what you will get on a Tuesday. They stay in Tier 2 until real recordings replace them.
| Condition | clips | cohere | gemini-3.1-fl | Qwen local | qwen-asr-flash | Scribe v2 | whisperX |
|---|---|---|---|---|---|---|---|
| Spanglish, switching mid-sentence | 3 | 7.4 | 2.7 | 2.7 | 1.9 | 1.7 | 3.1 |
| Spanish/English, switching between sentences | 4 | 62.0 | 8.3 | 35.3 | 18.6 | 7.3 | 49.6 |
| Portuguese + English, mixed beta | 3 | 8.6 | 8.4 | 6.0 | 2.7 | 1.7 | 5.3 |
| Portunhol — Portuguese + Spanish beta | 3 | 17.8 | 14.0 | 18.7 | 13.8 | 8.4 | 20.1 |
Two more honest limits. Scribe v2 is made by the same company whose text-to-speech generated most of these clips, which is a home-field advantage of unknown size — notice it wins every row here and only four of six in Tier 1. And the “between sentences” row is built by joining separate recordings end to end, which punishes engines that lock onto one language per clip; our pipeline transcribes speaker turn by speaker turn, which softens the problem in practice. Neither caveat is a reason to hide the numbers; both are reasons not to headline them.
Spanish and Portuguese sit close enough together that transcription engines quietly “fix” one into the other. Someone speaking portunhol says the status is em rojo — a Spanish word inside a Portuguese sentence — and the engine writes em vermelho, because that is what a well-behaved Portuguese sentence ought to say. Fluent. Plausible. Not what was said.
That costs one word of error rate and changes a fact. Sinsonte writes decisions and action items straight off the transcript, so we built a metric for it: the false-friend error rate. Our fixtures carry 25 marked traps; for each one the engine either kept what was spoken, swapped it for the other language's word — that swap is the FFER — or garbled it, which we deliberately keep out of the numerator so ordinary noise doesn't inflate the score.
| Engine | kept | swapped · FFER | garbled |
|---|---|---|---|
| Scribe v2 | 52% | 20% | 28% |
| granite-speech 4.1 | 32% | 28% | 40% |
| cohere-transcribe | 48% | 40% | 12% |
| qwen3-asr-flash | 48% | 40% | 12% |
| whisperX | 40% | 44% | 16% |
| gemini-3.1-flash-lite | 36% | 48% | 16% |
| Qwen3-ASR 1.7B, local | 24% | 68% | 8% |
Nobody clears 20%. The best engine we measured still rewrites one trap in five, and the worst rewrites seven in ten while barely garbling anything — confidently wrong, which is the dangerous shape of failure. We are publishing this knowing our own local engine comes last on it. As far as we can find, no public leaderboard and no transcription vendor measures this at all; the industry claim is that an engine switches languages, never that it keeps the word it heard. So take the table as a statement about the state of the art, not about who to buy from — and hold us to it as it improves.
Method
Corpus-level word error rate — total word errors divided by total reference words for the condition, not an average of per-clip scores. Text is normalised the way the Open ASR Leaderboard does it, with one deliberate change: accents and ñ are preserved rather than stripped, because a transcript that loses them is not a Spanish transcript. Every hypothesis is stored, so scores can be recomputed later without re-running any audio.
Seed 42 throughout, roughly 45 minutes of audio per Tier-1 condition and 18 for the meeting-room one. FLEURS test splits are smaller than the shuffle buffer, so the sample is an exact uniform shuffle of the whole split. The identifier of every one of the 1 258 clips is written to a manifest, so the same sample can be re-fetched and re-scored by anyone.
One consumer graphics card — an 8 GB RTX 3060 Ti — for the three local engines, one model at a time, one process each. Scribe v2 is a hosted API. Total cost of the round: $1.62, of which $0.89 was actual money; the rest is what the electricity and the card are worth. We mention it because a benchmark you cannot afford to repeat is a benchmark that never gets repeated.
The meeting-room index covers about 92.5% of the available split — seven pages of the catalogue kept returning rate-limit errors while it was being built, and we walked past them rather than retrying forever. The sample is still seeded and reproducible; it is just drawn from that 92.5%, and we would rather write this sentence than let you assume otherwise.
Google FLEURS
Read speech in 102 languages; we use the es-419, pt-BR and en-US test splits.
CC BY 4.0 · huggingface.co/datasets/google/fleurs
Multilingual LibriSpeech
Audiobook narration; we use the Spanish and Portuguese test splits. The Portuguese one is European Portuguese, not Brazilian — which is part of why it scores differently.
CC BY 4.0 · openslr.org/94
AMI Meeting Corpus
Real multi-person meetings recorded in instrumented rooms. We use the single-distant-microphone test split — the closest public audio to a phone left on a table.
CC BY 4.0 · groups.inf.ed.ac.uk/ami/corpus
The harness, the scoring code and the clip-by-clip manifest are not public yet — the repository they live in also holds the product. They are available on request to anyone who wants to check a number: write to [email protected] and we'll send them.