Meet Gladia, a French AI startup that wants to change how companies interact with audio data. The company develops an audio transcription application programming interface (API) that you can integrate with other products and is supposed to work much better than what’s available out there. And this tech foundation unlocks new use cases around audio.

If you’re familiar with audio transcription APIs, you know that big cloud providers already have their own APIs. There’s Google’s speech-to-text API, Amazon Transcribe, Microsoft’s Speech to Text, etc. They work well, but they are expensive, slow and don’t have a ton of features.

Gladia’s co-founder and CEO Jean-Louis Quéguiner, who was the former head of AI for OVHcloud and co-founded the company with Jonathan Soto, told me about some of the limitations with existing APIs. According to him, there are three pain points with existing products. First, when it comes to prices, transcribing an hour of audio generally costs $1.50 to $2 an hour.

Second, the output isn’t always very reliable as some languages work well while others are barely supported. When it comes to advanced features, if people speak in multiple languages, chances are the API simply won’t be able to notice the language change and transcribe the audio in more than one language.

Third, transcription APIs are slow. It can take more than 15 minutes to transcribe an hour of audio. That’s fine if you don’t need transcriptions right away, but it means that you won’t be able to use these APIs in some industries.

Whisper’s whisperer

Gladia is based on Whisper, OpenAI’s open-source transcription model. “We started from Whisper. We haven’t reinvented the wheel, but we listened to our customers and they told us: ‘What I want is something that works as well as Whisper,’” Jean-Louis Quéguiner told me.

But Whisper isn’t perfect. The vanilla version is still quite slow, so Gladia has spent a lot of time turning Whisper into a fast and responsive transcription model. That’s not the only issue.

“Half of Whisper is GPT-2. You’ve seen LLMs and ChatGPT, it tends to hallucinate. We’ve done a lot of work to avoid hallucination problems too,” Quéguiner said.

In particular, he told me that Whisper has been trained on closed captions that you can find on the internet, such as on YouTube. OpenAI’s model tends to hear common phrases that you can hear in online videos, such as “if you enjoyed this video, please like and subscribe.” There is a mathematical overrepresentation of some sentences like this one and Gladia tries to fix those shortcomings.

In addition to these modifications to Whisper and its implementation, Gladia also has some pre-processing and post-processing algorithms that improve the end results.

Gladia promises that it can transcribe an hour of audio for $0.61. And the transcription process takes roughly 60 seconds. Its API can detect when there are multiple speakers, add timestamps, detect languages and switch from one language to another if needed. Gladia also automatically adds punctuation and casing.

Like most APIs, the end result is in JSON format. But Gladia also supports SRT and VTT files for companies that want to generate subtitles.

I created an account and uploaded an audio recording of an interview to see how Gladia works. It took a bit more time than expected but it was definitely much faster than Google’s or Azure’s speech-to-text APIs.

The result wasn’t flawless, but it was extremely good — it understood acronyms and technical terms. I opened the same audio file in Aiko, a Mac app developed by Sindre Sorhus and that lets you transcribe audio file locally using Whisper. As expected, the output was close to Gladia’s output — but Gladia was much faster than running Aiko on my MacBook Pro.

Overall, Gladia was the best transcription API I’ve ever used.

Becoming an audio intelligence API

The company currently works with call center companies, virtual meeting services, and video publishers, including Claap, Livestorm and Selectra.

Gladia raised a $4 million seed round in a funding round led by New Wave. Other investors include Sequoia, Cocoa and business angels, such as Solomon Hykes, Pierre Betouin, Miroslaw Klaba and Alexandre Berriche.

Having a rock-solid transcription API is just step one for Gladia. The company hopes that it can then build features on top of this strong technical foundation.

For instance, after an audio file has been transcribed, Gladia can translate text into another language. Combined with word-level timestamps, it means that a company can upload an audio file and get subtitles in dozens of languages in just a few minutes.

In the future, the company hopes that it can summarize the content of an audio file, categorize content into multiple topic categories, create chapters automatically, conduct sentiment analysis and more.

“Our longer-term vision is to move from 2D to 3D data. Audio is pretty flat, and the idea is to augment it with intelligence,” Quéguiner said. “We think that transcription will become a commodity. But we think that what’s going to matter more is the options we’re going to add.”

Gladia turns any audio into text in near real time by Romain Dillet originally published on TechCrunch

https://techcrunch.com/2023/06/19/gladia-turns-any-audio-into-text-in-near-real-time/