AWS SageMaker AI adds bidirectional streaming for real-time speech transcription with vLLM

AWS SageMaker AI adds bidirectional streaming for real-time speech transcription with vLLM

Amazon SageMaker AI launched bidirectional streaming support for real-time inference in November 2025, according to an AWS blog post. The feature enables persistent, full-duplex connections between clients and model containers over HTTP/2, specifically targeting real-time speech-to-text applications.

Technical architecture

The implementation connects three layers:

Client to SageMaker AI: Applications connect to SageMaker AI runtime endpoints on port 8443 using HTTP/2. Each JSON message in vLLM's Realtime protocol is sent inside a RequestPayloadPart with DataType set to "UTF8", instructing SageMaker AI to forward data as WebSocket text frames.

SageMaker AI to container: SageMaker AI automatically bridges HTTP/2 event streams and WebSocket protocols. It establishes a WebSocket connection to containers at ws://localhost:8080/invocations-bidirectional-stream and forwards data frames bidirectionally.

Container layer: A FastAPI bridge listens on port 8080 and forwards connections to vLLM's Realtime API at ws://localhost:8081/v1/realtime. The bridge handles route translation between SageMaker AI's expected path and vLLM's native endpoint.

vLLM Realtime API protocol

vLLM's Realtime API requires audio encoded as base64 PCM16 at 16 kHz sample rate, mono channel. The protocol flow:

1. Client connects to ws://host/v1/realtime
2. Server sends session.created event
3. Client sends input_audio_buffer.commit when ready
4. Client streams input_audio_buffer.append events with base64 audio chunks
5. Server streams transcription.delta events with incremental text
6. Server sends transcription.done with final transcription and usage statistics

The model begins transcribing as soon as it has sufficient audio context, streaming tokens back while the client continues sending audio chunks.

Reference implementation

AWS provides a reference implementation using Mistral AI's Voxtral-Mini-4B-Realtime-2602 model. The example includes:

• Custom Docker container built on SageMaker AI vLLM Deep Learning Container
• Python client using SageMaker AI bidirectional streaming SDK
• Gradio-based live microphone demo
• Full code available in an AWS GitHub repository

vLLM applies piecewise CUDA graph execution to reduce GPU kernel launch overhead, directly reducing per-token latency during streaming transcription.

Infrastructure requirements

SageMaker AI handles connection management with WebSocket ping/pong keepalive frames, container health checks, and CloudWatch monitoring. The service eliminates the need for custom protocol translation layers or GPU server management.

What this means

This release removes a significant infrastructure barrier for deploying production voice AI applications on AWS. The automatic HTTP/2-to-WebSocket bridging and native vLLM integration eliminate the need for custom streaming infrastructure. For enterprises already using SageMaker AI, this provides a direct path to add real-time voice capabilities—voice agents, live captioning, contact center analytics—without migrating to specialized speech platforms. The open-source vLLM foundation prevents vendor lock-in on the serving layer while AWS handles operational complexity.