Alex Smola, Co-Founder of Boson AI – Interview Series

PUBLISHER ARTICLE PREVIEW

From the original article

Alex Smola is co-founder of Boson AI, where he works on conversational voice AI and multimodal systems. His career spans academic research at Carnegie Mellon University and large-scale AI initiatives at AWS. He brings that combination of research and product experience to building assistants that can understand conversational cues, handle interruptions, and carry out useful work.

Boson AI develops foundation models and tools for voice, audio, and multimodal interaction. Its Higgs Realtime model supports live speech-to-speech conversations, while its broader platform includes expressive speech generation, avatars, and enterprise voice agents. In this interview, Smola discusses conversational intelligence, multilingual performance, inference economics, and the safeguards needed to move voice agents into real business workflows.

Your career has taken you from academic research at Carnegie Mellon University to leading large-scale AI initiatives at AWS, and now to co-founding Boson AI. What inspired you to focus on conversational voice AI, and what fundamental limitations in human-AI interaction are you hoping to overcome?

Humans communicate through much more than words. You can hear uncertainty in someone’s voice or see that they haven’t understood an explanation. A small acknowledgement can tell you to keep talking. We pick up on these cues without thinking, but teaching computers to recognize them is still a work in progress.

That is what makes this area so interesting to me. We now have systems with considerable intelligence, but communicating with them can still feel awkward. I want people to be able to explain what they need in their own words, without having to figure out how to talk to the software.

At AWS, I learned to pay close attention to what customers were trying to do and whether the product helped them do it. We’ve brought that approach to Boson, with the freedom to move quickly as the research advances.

Voice and video give us more to work with. If someone looks confused or hesitates, the assistant has an opportunity to respond before the conversation goes off course. Getting those everyday interactions right is a big part of our ambition.

Boson AI recently introduced Higgs Realtime, designed to enable natural, real-time speech-to-speech interactions. What makes this approach fundamentally different from traditional systems that combine speech recognition, language models, and text-to-speech, and why does that distinction matter for the future of AI assistants?

In a traditional pipeline, speech recognition turns your words into text, a language model generates an answer, and a speech model reads it aloud. Each step depends on the one before it, which can make the conversation feel slow and disjointed.

Higgs Realtime is an audio-native speech-to-speech model, built for live interaction. Its training addresses conversational behavior, including turn-taking and responding to changing requests. That’s very different from putting a voice on a text chatbot.

Think about when you ask someone a difficult question. They might say, “Let me look that up,” while they work on it. You know they’ve heard you and are working on it. An AI assistant needs to do the same while it retrieves information or uses a tool, so you’re not left wondering whether it understood you or stopped working.

That becomes even more important with longer tasks. The assistant needs to keep you informed as it works and respond if you ask a question or change your mind.

One of the biggest challenges in conversational AI is knowing when to speak, when to listen, and how to handle interruptions or changes in direction. How is Boson AI addressing these challenges, and what technical breakthroughs are still needed before AI conversations feel genuinely natural?

An acknowledgment and an interruption can sound quite similar but they need different responses. If someone says “yeah” while you’re talking, they may want you to keep going. If they correct a digit in their phone number, you need to stop and use the right number. An AI assistant should be able to tell the difference.

Higgs Realtime uses the meaning of what someone says to help decide whether they’ve finished speaking or are taking a pause. Responding quickly is important, but so is knowing how to respond.

Getting that right consistently is still quite difficult. People don’t speak in neat turns and the same words can mean different things depending on the conversation. The assistant has to understand what changed and remember what it was doing.

Boson AI has developed benchmarks such as IHBench to measure how voice agents recover from interruptions. What have these evaluations revealed about the weaknesses of today’s leading voice models, and how should the industry rethink the way it measures conversational intelligence?

We built IHBench to test what happens after someone interrupts a voice agent. If you correct something halfway through a booking, the agent should take the correction and carry on from the right point. You should not have to start again, or sit through information it has already given you.