ElevenLabs' Scribe v3 makes 27% fewer word errors in 102 languages

ElevenLabs says in its own tests that its two new speech-to-text models make fewer errors than Scribe v2 and Scribe v2 Realtime. Scribe v3 handles recordings, and Scribe v3 Live handles speech as it happens.

Share
ElevenLabs' Scribe v3 makes 27% fewer word errors in 102 languages
Illustration: AI-generated

A voice system that waits for silence can interrupt people who pause in the middle of a sentence. ElevenLabs says its new Scribe v3 Live uses the context of the conversation as well as the audio, so a caller who pauses halfway through a number is not cut off.

The company has launched two speech-to-text models, which turn spoken words into written text. Scribe v3 works on recorded audio and Scribe v3 Live on realtime audio. The company says both are built for messy real-world audio.

The figures are the company's own. ElevenLabs ran the tests itself, scoring the new models against its older ones and other popular models on the same datasets. It says Scribe v3 makes 27% fewer word errors (words a transcript gets wrong) than Scribe v2 across 102 languages.

For Scribe v3 Live, the company reports 43% fewer errors than Scribe v2 Realtime. That figure uses semantic word error rate, which counts an error only when the meaning of the transcript changes. On standard word error rate, the reduction is 31%.

Multilingual accuracy was measured on an internal version of the FLEURS benchmark, a standard test set of short, clean read speech. Each FLEURS clip has one speaker, so ElevenLabs built its own dataset of real meetings, agent calls and voice notes in 16 languages to test speaker labelling. It says it will release the dataset openly for anyone to use. It cautions that results measured in other ways are not directly comparable with its own.

By language, ElevenLabs reports Scribe v3 word errors down 42% in German, 34% in Swahili and 28% in Thai.

ElevenLabs says Scribe v3 makes 17% fewer errors in labelling who said what than Scribe v2 on recordings with three or more speakers.

ElevenLabs says customers in India wanted Hindi to match English even when callers switch between the two, so Scribe v3 Live adds a multilingual mode for conversations that switch languages mid-sentence.

Eneco, an energy supplier to more than 2 million homes and businesses in the Netherlands, tested Scribe v3 Live on its voice agent before launch. In real customer calls, ElevenLabs reports, the agent recognised Dutch postcodes correctly on the first try 40% more often, and house numbers 15% more often.

The ElevenLabs app uses both models by default, and developers can use them through ElevenAPI. They run in the US by default. Enterprise customers who need their data kept in a particular region can use separate environments in the EU, India and Singapore. A version of Scribe v3 Live adapted for medical conversations is also being released.

Sources