⏭️ Forward Deployed: Voice AI on what works in 2026

Summary

• The discussion’s operational takeaway is that the cascaded STT→LLM→TTS pipeline remains the enterprise default, not speech-to-speech. A voice-platform panelist illustrates the risk with a booking bot that agreed, “You’re right. It is 2030. So let’s schedule this for June 10th, 2030,” while the Smallest AI panelist says enterprises still have “many more cascaded deployments” and that S2S is “the eventual future.” That panelist predicts a hybrid: S2S for the live loop, with cascades delegated for complex lookups.
• Basil frames voice as “one of the most competitive markets in AI right now,” while the host frames customer support as the dominant use case — “hundreds of billions of dollars spent” on call centers that are all bad. The host’s editorial filter is blunt: vendors touting pipelines is “yeah, duh”; the signal is practitioners conceding “the models are not there yet. They may never be.”
• The host says he thinks “voice AI has been one generation ahead of coding agents” — context compaction, now trendy in coding tools, was table stakes “from day one” because early models went off the rails at 10,000 tokens. His rule of thumb even on a 1-million-context model: “at 25% you should become nervous.”
• Latency versus intelligence is called “the hardest problem to solve in voice deployments,” attacked by “shaving off 10 milliseconds at a time” and, counterintuitively, by adding fillers. Zero contextual fillers “actually feels more rigid” than a natural “give me a sec” while a customer’s API takes 5 seconds — the bottleneck becomes how good the fillers sound.
• One execution-engine panelist describes a prompt-architecture pendulum. Seven months ago LLMs were “consistently skipping step 4A1,” forcing situational prompt inclusion; now that relevance layer is sometimes less reliable than the model itself, “so now we’re going back… towards just give the model everything.” The honest method regardless: “test and find out and then test again… it goes on forever.”
• The cost and reliability wedge can favor self-hosted small models over frontier APIs for high-volume voice. A platform panelist notes that calls that “hang up 10 seconds into the call” still incur the input-token cost of a giant prompt; Smallest sees customers swap GPT-4o/4.1 realtime for its fine-tuned SLM “Electron” — cheaper, “way lower” latency, and less exposed to OpenAI API spikes where “you have no control on those latencies.”
• Meta: this episode launches Latent Space’s fourth podcast — Basil’s FDE-focused show — built around the host’s thesis that forward-deployed engineers surface customer ground truth. Researchers “always want bigger models to solve everything”; FDEs field the customer saying “they made this horrible mistake… how can you guarantee me that?”

Deep dive

1. Latent Space adds a fourth podcast: Basil’s FDE show, opening with voice agents

• Basil’s path into forward-deployed engineering: product manager on Credit Karma for a couple of years, then a small venture studio, then the consulting business Exaflop Labs building agents for retailers and insurers two years ago. By January, private-equity clients kept telling him, “I don’t know what’s marketing BS and what’s not,” so he started recording deep-dive panels — the first on agentic engineering with Factory, Cognition, Composio, and Sourcegraph, plus computer-use agents and agents in the enterprise.
• Why voice first: Basil calls it “one of the most competitive markets in AI right now,” citing Sierra; the host’s filter for signal is blunt — pipeline vendors praising pipelines is “yeah, duh,” but support-focused practitioners concluding “the models are not there yet. They may never be” is worth publishing.
• The host’s FDE thesis, explicitly framed by him: researchers “always want bigger models to solve everything,” product engineers try to ship, but FDEs facing customers hear, “This can never happen again. How can you guarantee me that?” Outbound’s brutal reality per Basil: people “just hang up as soon as they realize it’s a bot.”

2. Voice-agent 101: the cascade, and why turn-taking isn’t trivial

• The architecture panelist’s walkthrough: audio arrives over WebRTC, a phone call, or WebSockets; optional noise-removal and voice-isolation models can run around transcription; then turn detection — “it’s not walkie-talkie or push-to-talk” — uses voice activity detection and a Smart Turn model to distinguish a mid-sentence pause from a finished thought before the LLM responds and TTS streams audio back.
• Guardrails differ by call direction: outbound debt collection is a common case and is easier because the bot can drop the call and accept only a finite set of inputs; inbound at Amazon scale means a billion products and different policies that can’t simply live in one prompt, because LLMs “always remember the first 4% and the last 4% and everything in between they kind of forget.”
• The host’s most quotable claim: “voice AI has been one generation ahead of coding agents… we were doing compaction from day one” — early, smaller models derailed at 10,000 tokens, and even with 1-million-token context, “at 25% you should become nervous” and consider compacting.

3. Speech-to-speech dazzles in demos; cascades still win the enterprise

• A voice-platform panelist’s specimen case against S2S: a snappy, emotive demo asked what day to book, he said “next week,” and it replied, “Great, is that June 10th?” — “No, the year is 2030.” “You’re right. It is 2030. So let’s schedule this for June 10th, 2030.” Cascades can route input through supervisor models for prompt injection and social engineering, intent selection, context optimization, and groundedness checks — the price being a conveyor belt the engineering team spent roughly six months parallelizing, “shaving off 10 milliseconds at a time.”
• The Smallest AI panelist’s counter-case: cascades are synchronous, but the brain “is thinking while listening”; “if you ever want to pass the Turing test for how the human brain operates,” you need asynchronous, natively speech-in/speech-out models. Hydra is multimodal — speech and text in, speech and text out — while retaining tool calls and guardrails. The concession is that a dedicated STT model might be more accurate than an S2S encoder, enterprises still have many more cascaded deployments, and S2S is “the eventual future.”
• Smallest AI’s hybrid prediction: keep S2S running the live loop, interrupt and delegate complex questions to a cascade — like humans answering on autopilot for “50% of our time.” As these systems reach high volume, specialized models could be trained for that routine half.
• Model picks and gossip: with thinking turned off, fast models can work; one panelist says, “I still like my Gemini 2.5… the 3.5 that they launched is slower,” and calls Haiku “really, really good.” On Sesame’s vanished demo, a panelist says they understand the company is building hardware, then relays an answer he received that the CEO had already made a lot of money and wanted to play around.

4. Giant prompt vs. workflow graph — one execution-engine panelist’s pendulum swings back

• A platform panelist’s split: inbound calls with dedicated lines and predictable flows suit a node/graph builder; outbound callers can be “erratic” — frustrated, angry, or annoyed — so one prompt as a “central brain” can retrieve components no dedicated flow anticipated, using retrieval and cosine similarity.
• The execution-engine panelist’s change of mind, worth keeping verbatim: seven months ago LLMs were “consistently skipping step 4A1,” so they built situational rule inclusion; now the layer deciding relevance is sometimes worse than the model, “so now we’re going back… towards just give the model everything and more or less it will figure it out.”
• The methodological bottom line for nondeterministic systems: build per customer, run LLM-as-judge, and accept that “the real answer is most likely always just test and find out and then test again and then find out again and it goes on forever.”

5. Latency, cost, localization — and the self-hosted SLM undercut

• Fillers as product, not failure: humans say a few words while thinking, so zero fillers “actually feels more rigid”; a natural “just give me a sec” can cover a 5-second customer-API tool call. The execution-engine panelist’s verdict: the performance-versus-stability trade-off “is the hardest problem to solve in voice deployments.”
• A platform panelist’s tactics: offload steps to constrained sub-agents, such as credit-card capture inside a collections flow; stream in parallel to small classifier models for intent; and make components swappable. In a Japan deployment, voice technology that worked for English, Spanish, and Portuguese “didn’t work at all” for the voice piece, so the approach allowed a custom TTS server; Arabic brand and address pronunciation is similarly hard.
• Another platform panelist’s cost point compounds the latency one: calls predominantly “hang up 10 seconds into the call,” yet the giant prompt’s input tokens have already incurred cost — splitting prompts buys both speed and money.
• Evals and the API-displacement story: the panelist discussing open-source tooling publishes open-turn STT, LLM, and TTS benchmarks that can be run locally against new models. The discussion also mentions that Nemotron 3.5 had just launched and that an NVIDIA ASR benchmark had performed well. FDEs can have Claude “build an eval based on our eval suite” from real customer conversations before going live. Smallest sees customers pair its models with Electron, a fine-tuned SLM, over GPT-4o/4.1 realtime: lower cost, “way lower” latency, and self-hosting that avoids dependence on OpenAI API spikes where “you have no control on those latencies.”