Kubernetes

AWS has launched model caching for Amazon SageMaker Inference on HyperPod, pre-loading model weights and container images onto cluster nodes to cut inference pod startup times from tens of minutes to seconds. The feature targets large models like DeepSeek-R1, which previously took 30+ minutes to become servable during scale-out events.