ElevenLabs previews Interaction Engine, Memory, Routines and ElevenAI at its New York Summit

ElevenLabs previewed a system called Interaction Engine at its New York Summit, alongside new ElevenAgents features (Memory and Routines), a new app called ElevenAI, and a healthcare transcription model, Scribe v3 Medical. The company also said that in the coming months it will help businesses fine-tune certain open-source language models and ElevenLabs voice models on their own data. Pricing and general availability dates were not disclosed for any of these items.

Interaction Engine

Interaction Engine is a system previewed at the event to orchestrate a whole conversation. According to ElevenLabs, it starts with the rhythm of the exchange: it handles turn-taking and interruptions so the agent knows when to speak and when to stop. It can also choose the LLM for each part of a conversation. A cheaper model can handle triage to keep token use in check, and a frontier model steps in for more complex tasks.

Domain-specific and fine-tuned models

ElevenLabs introduced Scribe v3 Medical, a transcription model for healthcare. The company claims it recognizes thousands of complex medical terms with much greater accuracy and distinguishes similar drug names such as cycloserine and cyclosporine. CEO Mati Staniszewski said a domain-optimized model for financial services is in the works.

Staniszewski also announced that in the coming months ElevenLabs will help businesses fine-tune certain open-source language models and ElevenLabs voice models. Businesses can bring data from live deployments and work with ElevenLabs researchers, choosing metrics such as customer satisfaction score (CSAT) or reliability. Staniszewski compared the process to each company having its own reinforcement-learning stack, with continuous optimization from each new interaction.

ElevenAgents updates

ElevenAgents is the company's platform for configuring, deploying and improving voice and chat agents across sales, support, recruitment and operations. ElevenLabs named Yahoo, Westgate and KPN as customers and partners using it.

• Memory: Lets an agent build a record of a customer over time and recognize a returning authenticated customer. The CRM remains the primary customer record. Memory captures details shared in conversation that have no CRM field, such as preferences, constraints and upcoming occasions. After each interaction the agent distills the conversation into a structured record and updates it on the next contact. In ElevenLabs' example, a subscriber who mentioned a March move is asked in April whether to transfer service to the new address.
• Routines: A new automation layer that runs multi-step work before, during and after a conversation, connected to tools a team already uses. Examples given include pre-screening a mortgage applicant and calling back within minutes, reminding patients before a visit and offering a canceled slot from the waitlist. Routines can also email a report, write to a CRM or start a call.

ElevenAI

Staniszewski gave a sneak preview of ElevenAI, a new app positioned as an AI teammate for enterprise teams. It listens to spoken conversations, including in-person meetings, and turns them into notes and next steps. It offers dictation and meeting notes powered by Scribe v3, which ElevenLabs describes as its newest and most accurate Speech to Text model. In meetings it takes notes with attendees' consent and delivers a summary and action items for each attendee afterward. It can also offer live assistance during a call.

What this means

The announcements extend ElevenLabs from voice models into the layers around them: orchestration, customer memory, post-call automation and an employee-facing app. Interaction Engine's model selection addresses cost directly, since routing triage to a cheaper model is meant to limit token use while reserving frontier models for harder steps. Fine-tuning on live-deployment data, optimized against a customer's chosen metric, would let deployments adapt to a specific business rather than only an industry. Most items were previewed or described as coming, so scope, pricing and timing remain to be confirmed. Staniszewski's stated view is that conversational agents will become critical infrastructure for every enterprise, as essential as cloud or payments.