inference optimization

7 articles tagged with inference optimization

November 17, 2026
researchTogether AI

Together AI says CUDA Graphs and PDL recover 98% of megakernel performance without custom kernels

Together AI reports that CUDA Graphs and PDL on top of default Torch Inductor kernels recover 98% of its earlier megakernel performance. With "multi-kernels" and agent loops, it claims up to 1.7x over its old megakernels, reaching over 1100 TPS for Llama 1B on H100. The work led to Together Compiler, a framework for continual AI inference optimization.

November 9, 2026
research

Baidu discloses ERNIE Evolve, an RSI research system already used to optimize ERNIE 5.5 Flash inference

Baidu ERNIE disclosed its first recursive self-improvement (RSI) progress on November 9: ERNIE Evolve, an automated research system now applied to inference optimization for the upcoming ERNIE 5.5 Flash. Baidu claims state-of-the-art results on NanoChat AutoResearch and NanoGPT Speedrun.

October 5, 2026
product updateAmazon Web Services

AWS ships aws-ai-ml skill so Kiro, Claude Code and Codex can benchmark SageMaker inference endpoints

Amazon SageMaker AI has released the aws-ai-ml skill, distributed through the Agent Toolkit for AWS. It lets MCP-compatible coding agents such as Kiro, Claude Code and Codex benchmark endpoints, recommend deployment configurations and generate executable SageMaker Python SDK v3 code.

September 24, 2026
analysisLiquid AI

Liquid AI Releases DSpark Draft Model for LFM2.5-VL-3B, Claims Up to 3.13x Decode Speedup

Liquid AI has released LFM2.5-VL-3B-DSpark, a 280M-parameter speculative decoding drafter for its LFM2.5-VL-3B vision-language model. The company claims decode speedups up to 3.13x on Apple silicon and 2.66x on H100 GPUs, with day-one support for llama.cpp, MLX-VLM, and SGLang.

August 20, 2026
changelogLiquid AI

Liquid AI Ships DSpark Draft Models, Cutting LFM2.5 Inference Latency Up to 3.18x on GPU

Liquid AI released DSpark draft model checkpoints for three LFM2.5 models, enabling speculative decoding that speeds up inference by up to 3.18x on H100 GPUs and 2.87x on-device, with day-one support for llama.cpp and SGLang.

July 9, 2026
model releaseNVIDIA

NVIDIA releases Nemotron-Labs-3-Puzzle-75B, compressed from 120B to 75B parameters with 2× throughput

NVIDIA has released Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super that reduces the model from 120.7B total/12.8B active parameters to 75.3B total/9.3B active parameters. According to NVIDIA, the model achieves approximately 2× higher server throughput on a single 8×B200 node and increases sustainable 1M-token single-H100 concurrency from 1 request to 8 requests while maintaining strong accuracy across benchmarks.

May 22, 2026
model releaseNVIDIA

NVIDIA releases Nemotron-Labs-Diffusion-14B with tri-mode decoding achieving 3.3x speed-up on GB200

NVIDIA released Nemotron-Labs-Diffusion-14B, a 14-billion parameter language model that supports three decoding modes by switching attention patterns during inference. The model achieves 850 tokens per second on GB200 hardware at concurrency 1, representing a 3.3x speed-up over standard autoregressive decoding and outperforming Qwen3-8B-Eagle3 by 2.2x in self-speculation mode.