This page is the protocol: the corpora we score against, the metric we score with, the vendors we run, and the failure we most expect to find. It is published before we have run anything, on purpose. A benchmark whose rules appear after its results is a benchmark whose rules were chosen to fit them.
Transcription has something most AI categories do not: an answer key nobody can argue with. A human wrote down what was said, and either the machine matched it or it did not.
| Measure | What it captures | Why it is on the list |
|---|---|---|
Word error rate |
The share of words the system got wrong, after normalising casing, punctuation, and number formatting so that "twenty dollars" and "$20" are not scored as a mistake. | The primary score. It is the one number a buyer actually feels, and it is objective enough that a vendor who dislikes the result has to argue with the audio rather than with us. |
Cost per audio hour |
Retail list price to transcribe one hour, compared like for like against retail list price. | Accuracy without price is half a decision. The cheapest system that clears your accuracy bar is usually the right buy. |
Real-time factor |
How long the system takes to transcribe, relative to the length of the audio. | It separates the batch tools from the ones you can put in front of a live caller. |
Three open corpora, chosen because they fail systems in three different ways. Every clip is playable in your browser with the reference transcript sitting next to it, so you can hear what a system was up against before you judge its score. The rule is simple: if you cannot inspect the input, the score is just a claim.
| Corpus | What it is | What it tests | Licence | Audio |
|---|---|---|---|---|
LibriSpeech 1,000 hrsread speech |
Volunteers reading public-domain audiobooks, cleanly recorded. | The floor. Any system that stumbles on clean read English is out of the conversation, and this is where the field bunches up. | CC BY 4.0 | Play clips |
AMI Meeting Corpus 100 hrsmeetings |
Real meetings, multiple microphones, people talking over each other. | The hard slice. Crosstalk, accents, and far-field microphones are where the marketing accuracy numbers go to die. | CC BY 4.0 | Play clips |
Earnings-22 119 hrsearnings calls |
Earnings calls from companies worldwide, with speakers of many nationalities. | The commercial case. Domain jargon, company names, and accented business English - the audio most people are actually paying to transcribe. | CC BY-SA 4.0 | Play clips |
Audio and reference transcripts are used under their respective Creative Commons licences: LibriSpeech (CC BY 4.0), AMI Meeting Corpus (CC BY 4.0), Earnings-22 (CC BY-SA 4.0). Clips are played from the datasets' own hosted viewers.
The commercial transcription APIs are the ones we run: Deepgram, AssemblyAI, OpenAI, ElevenLabs, Speechmatics, Google, Microsoft Azure, and AWS. A self-hosted open-source model is included as the baseline, because "just run Whisper on a box" is a real option and a buyer deserves to know what it costs them in accuracy.
The open-source models are already well covered by public leaderboards. The question those leaderboards mostly leave unanswered is the one a buyer actually has: how do the paid APIs compare against each other, on audio that resembles the audio in their business, at the prices they will actually be charged. That gap is the reason this category exists.
Our corpora and our scoring harness come from HuggingFace's Open ASR Leaderboard, an excellent piece of open infrastructure, and we use their text normaliser unchanged so that our numbers stay comparable to theirs. What we add is our own runs against the commercial vendors, at our own cost. The scores we publish will be scores we produced.
Word error rate is a blunt instrument. It punishes a system for missing a filler word exactly as hard as for inventing a number that was never said, and the second mistake is far worse in practice. When the results land we will call out hallucinated content separately, because a transcript that reads beautifully and says the wrong thing is the failure mode that costs people money.
