AWS Adds Model Caching to SageMaker HyperPod, Cuts Inference Cold Starts from 30 Minutes to Seconds

AWS has launched model caching for Amazon SageMaker Inference on HyperPod, a feature designed to eliminate the download delays that stall inference pods before they can serve traffic. According to AWS, pods can now typically start serving requests in seconds rather than tens of minutes.

The Problem

Deploying an LLM on SageMaker HyperPod involves two sequential downloads before a pod can serve a single request: the inference server container image from Amazon ECR (5–7 minutes for multi-gigabyte images bundling GPU drivers, CUDA libraries, and serving frameworks), and the model weights from Amazon S3, FSx for Lustre, or Hugging Face Hub. For a 145 GB model, weight downloads alone can take 20+ minutes. For DeepSeek-R1, at over 600 GB, the full process takes upwards of 30 minutes.

This delay compounds during autoscaling. If a HorizontalPodAutoscaler requests five new pods in response to a traffic spike, each pod independently repeats the full download cycle. AWS notes that while the autoscaling policy may react within seconds, actual capacity to serve additional traffic can lag by 25–30+ minutes.

How Model Caching Works

The feature introduces two independent capabilities — a weights cache and an image cache — that can be enabled together or separately via a new modelCacheConfig field on the InferenceEndpointConfig or JumpStartModel resource.

The weights cache pre-downloads model weights to local NVMe storage on target nodes before pods are scheduled. The HyperPod Inference Operator creates a ModelDataCacheConfig resource, downloads weights from the configured source, and labels nodes as "cache-ready" once complete. The operator waits for all target nodes to reach cache-ready status before creating the inference deployment. Pods then read from local NVMe at approximately 7 GB/s instead of downloading over the network, and the cache persists across pod restarts on the same node.

The image cache pre-pulls the container image via a DaemonSet onto target nodes, tracked through a new ModelImageCache CRD. Unlike the weights cache, it does not block deployment creation — the deployment starts immediately while the image pull happens in parallel, saving 5–7 minutes when the cache completes before a pod needs it.

Both mechanisms use preferred rather than required scheduling. If a pod lands on a node without a warm cache — for example, during rapid scale-out that exceeds the number of pre-cached nodes — it falls back to standard network downloads with no failure or manual intervention required.

When a model source changes (a new S3 path with updated weights, for instance), the operator creates a new cache, rolls out the updated deployment, and removes the old cache afterward, which AWS says ensures zero-downtime transitions.

Implementation

Enabling the feature requires no additional infrastructure — users add a modelCacheConfig block with weightsCache and imageCache flags to existing resource definitions. Cache state can be inspected via kubectl get modeldatacacheconfig and kubectl get inferenceimagecache, which report per-node readiness and target counts.

What This Means

Cold-start latency has been a persistent operational tax on GPU-based inference infrastructure, particularly as models grow past hundreds of gigabytes. AWS's fix is straightforward infrastructure engineering — pre-stage data locally rather than optimize the transfer path — but it directly addresses a bottleneck that made autoscaling effectively meaningless for large models: a policy that reacts in seconds is worthless if pods still need 30 minutes to become useful.

The practical impact will scale with model size. Teams running smaller models that already start in a few minutes will see modest gains. Teams running frontier-scale open models like DeepSeek-R1 on HyperPod get the most significant benefit, since the gap between autoscaler response time and actual capacity increase closes almost entirely. This is a HyperPod-specific, Kubernetes-native feature — it does not extend to SageMaker's fully managed inference endpoints — so its relevance is limited to customers already operating on HyperPod's more hands-on infrastructure layer.