Introduction to OpenAI’s New Developments
OpenAI has introduced new transcription and voice-generating AI models through its API, which the company asserts are improvements over its previous releases.
The Broader Vision
For OpenAI, these models align with its broader “agentic” vision, which involves creating automated systems capable of independently accomplishing tasks on behalf of users. Although the definition of an “agent” might be disputed, OpenAI Head of Product Olivier Godement describes one interpretation as a chatbot that can engage with a business’s customers.
Future Developments
“We’re going to see more and more agents emerge in the coming months,” Godement told TechCrunch during a briefing. “And so, the general theme is about helping customers and developers leverage agents that are useful, available, and accurate.”
Text-to-Speech Model
OpenAI claims its new text-to-speech model, “gpt-4o-mini-tts,” delivers more nuanced and realistic-sounding speech and is more “steerable” than its previous speech-synthesizing models. Developers can instruct gpt-4o-mini-tts using natural language, such as “speak like a mad scientist” or “use a serene voice, like a mindfulness teacher.”
Voice Samples
Here’s an example of a “true crime-style,” weathered voice:
And here’s a sample of a female “professional” voice:
Developer Control
Jeff Harris, a member of the product staff at OpenAI, told TechCrunch that the goal is to let developers tailor both the voice “experience” and “context.”
“In different contexts, you don’t just want a flat, monotonous voice,” Harris said. “If you’re in a customer support experience and you want the voice to be apologetic because it’s made a mistake, you can actually have the voice convey that emotion … Our big belief here is that developers and users want to really control not just what is spoken, but how things are spoken.”
Speech-to-Text Models
OpenAI’s new speech-to-text models, “gpt-4o-transcribe” and “gpt-4o-mini-transcribe,” effectively replace the company’s older Whisper transcription model. Trained on diverse, high-quality audio datasets, these models can better capture accented and varied speech, even in chaotic environments, according to OpenAI.
Improved Accuracy
They’re also less likely to hallucinate, Harris added. Whisper was known to fabricate words and passages in conversations, introducing irrelevant information into transcripts.
“[T]hese models are much improved versus Whisper on that front,” Harris said. “Making sure the models are accurate is completely essential to getting a reliable voice experience, and accurate means that the models are hearing the words precisely and aren’t filling in details they didn’t hear.”
Language-Specific Performance
However, the performance may vary depending on the language being transcribed.
According to OpenAI’s internal benchmarks, gpt-4o-transcribe, the more accurate of the two transcription models, has a word error rate approaching 30% for Indic and Dravidian languages such as Tamil, Telugu, Malayalam, and Kannada. This means three out of every 10 words from the model will differ from a human transcription in those languages.
Availability of New Transcription Models
In a departure from its usual practice, OpenAI doesn’t plan to make its new transcription models openly available. The company had previously released new versions of Whisper for commercial use under an MIT license.
Harris stated that gpt-4o-transcribe and gpt-4o-mini-transcribe are “much bigger than Whisper” and thus not suitable for an open release.
“[T]hey’re not the kind of model that you can just run locally on your laptop, like Whisper,” he continued. “[W]e want to make sure that if we’re releasing things in open source, we’re doing it thoughtfully, and we have a model that’s really honed for that specific need. And we think that end-user devices are one of the most interesting cases for open-source models.”
Update
<p class="wp
Source Link
