AWS Ships 13 SageMaker Inference Features in 2026, Cutting Startup Latency 51% and GPT-OSS-20B Throughput 2x

What happened

Amazon Web Services published a mid-year review of Amazon SageMaker AI's inference stack, detailing 13 capabilities shipped between roughly April and mid-2026 across two deployment paths: fully managed SageMaker endpoints and Kubernetes-native SageMaker HyperPod Inference. The post frames the release as an answer to generative AI's inference-specific problems — multi-gigabyte model weights, cold starts that can span minutes, GPU capacity shortages, and token-level performance signals that standard monitoring tools don't surface.

The launches

Four features anchor the managed-endpoint track:

• Inference recommendations (April 2026): Given a model and a target (cost, latency, or throughput), SageMaker automates instance selection and optimization — narrowing candidate instance types, applying techniques like EAGLE 3.0 speculative decoding and tensor parallelism, then benchmarking with NVIDIA AIPerf. AWS says a demonstrated optimization run on GPT-OSS-20B doubled tokens-per-second throughput at the same request latency. The feature itself carries no additional charge; customers still pay for benchmarking compute unless they use ML Reservations.

• Capacity-aware instance pools (May 2026): Customers can now define up to five prioritized instance types per endpoint. SageMaker automatically falls back through the list during endpoint creation and scale-out, and retracts toward preferred hardware during scale-in, addressing the prior single point of failure where one unavailable instance type could block an endpoint from serving any requests. Available in all commercial AWS Regions.

• OpenAI-compatible APIs (May 2026): SageMaker endpoints now expose an /openai/v1 Chat Completions path with streaming, so applications built on the OpenAI SDK, LangChain, or Strands Agents can point at SageMaker by changing only the endpoint URL. Bearer tokens generated from AWS credentials last up to 12 hours, removing SigV4 signing requirements. Live in 14 AWS Regions, with vLLM and SGLang Deep Learning Containers supported alongside custom containers implementing /v1/chat/completions.

• Container caching (June 2026): Pre-pulls container images to new instances ahead of scale-out, eliminating the multi-minute delay large serving containers (10GB+) previously caused. AWS reports a test case with Qwen3-8B on an ml.g6.2xlarge instance using a 17.7GB LMI container: end-to-end startup latency fell from 525 seconds to 258 seconds, a 51% reduction, with model download time dropping from 168 to 77 seconds. Early-access customers reportedly saw gains between 38% and 65%. The feature requires no configuration and activates automatically on supported accelerator instances.

AWS also lists async inference inline payloads, an observability integration with CloudWatch, and — on the HyperPod Inference side — prefix-aware routing, a simplified operator, tiered KV caching, data capture, disaggregated prefill/decode, and model caching, though the source content cuts off before detailing all of these.

What this means

This is infrastructure tooling, not a model release — no new weights, no new benchmark leaderboard entry. The significance is operational: AWS is systematically closing gaps between raw GPU capacity and usable inference throughput, targeting the specific pain points (cold starts, capacity shortages, SDK lock-in) that make self-hosted LLM serving expensive and fragile at scale. The OpenAI-compatible API is the most consequential single item here — it lowers switching costs for any team currently locked into OpenAI's SDK ecosystem but wanting to run open-weight models on AWS-owned GPUs. The performance numbers (51% startup reduction, 2x throughput) are AWS-reported from specific test configurations and should be treated as upper-bound claims rather than guaranteed results across all model/instance combinations.