Together AI says CUDA Graphs and PDL recover 98% of megakernel performance without custom kernels
Together AI reports that CUDA Graphs and PDL on top of default Torch Inductor kernels recover 98% of its earlier megakernel performance. With "multi-kernels" and agent loops, it claims up to 1.7x over its old megakernels, reaching over 1100 TPS for Llama 1B on H100. The work led to Together Compiler, a framework for continual AI inference optimization.
Together AI says it can recover 98% of its earlier megakernel performance using only CUDA Graphs and PDL on top of default Torch Inductor kernels, with no hand-written kernels. In a research post on its blog, the company also claims that "multi-kernels" and agent loops beat its old megakernels by up to 1.7x, reaching over 1100 TPS for Llama 1B on H100.
The post explains the path to Together Compiler, which the company describes as its framework for continual AI inference optimization.
From megakernels to simpler primitives
Last year, Together helped release some of the first AI megakernels, which execute an entire model forward pass inside a single, highly fused GPU kernel. According to the company, they reached record-breaking speeds but were hard to deploy. After months of effort, Together says it had one model, one batch size and one customer.
This year, the team decomposed the megakernel gains into simpler primitives. Its conclusion: CUDA Graphs and PDL layered over default Torch Inductor kernels deliver 98% of megakernel performance. The post is organized in three parts: Torch Compile's limitations, a revisit of megakernels, and lessons from building agent-driven optimization infrastructure.
Inductor baseline and findings
On the Llama 1B workload, Torch Inductor delivers 669 TPS, according to Together. The company reports two findings in the portion of the post covering Inductor's internals.
Heuristics do not transfer across workloads. Inductor's freezing flag tells the compiler that model weights will not change during inference, allowing ahead-of-time weight transformations, including a conversion to FP32 before returning to the original lower-precision format. Together found that enabling freezing improved throughput by 10-26% at larger batch sizes, where the GPU performs matrix-matrix multiplications. At batch size 1, which relies on matrix-vector multiplications, the same flag caused a 49% slowdown. Together says the FP32 conversion was not reversed for matrix-vector operations, so the GPU processed larger 32-bit weights.
Flags interact unpredictably. As of Q3 2026, Inductor exposes 607 tuning flags, 198 of them added in the past year, per Together. Enabling aggressive_fusion alone reduced Llama 1B throughput by 6 TPS. Enabling it together with combo_kernels, benchmark_fusion and triton.coalesce_tiling_analysis improved throughput by 3 TPS.
Isolated kernel speed can mislead. Together argues that the fastest kernel in isolation is not necessarily the fastest during inference, because Inductor's benchmarking harness drives its optimization decisions. One cited problem is that kernel performance depends on caching behavior, yet Inductor clears the GPU's L2 cache before benchmarking individual kernels.
How Inductor fuses operations
Together describes Inductor's fusion as a greedy, pairwise algorithm. A can_fuse() function checks legality and score_fusion() ranks candidates using heuristics such as estimated memory traffic savings. Together notes that these decisions rest on local heuristics rather than a global search over kernel partitions and execution strategies. Operations that call libraries such as cuBLAS are treated as external kernels (ExternKernel) whose implementations are generally opaque to fusion.
What this means
The reported results shift the cost-benefit of megakernels. Together's own account is that a megakernel covered one model, one batch size and one customer, while CUDA Graphs and PDL on default Inductor kernels capture 98% of that performance with no custom kernel code. The freezing flag result shows why per-workload testing matters: the same setting helped at larger batch sizes and cost 49% at batch size 1. Together's stated takeaway is that the optimization space should be searched systematically rather than by guessing flags, which is the premise behind Together Compiler. The 1100+ TPS and 1.7x figures are Together's claims for Llama 1B on H100.
Related Articles
Baidu discloses ERNIE Evolve, an RSI research system already used to optimize ERNIE 5.5 Flash inference
Baidu ERNIE disclosed its first recursive self-improvement (RSI) progress on November 9: ERNIE Evolve, an automated research system now applied to inference optimization for the upcoming ERNIE 5.5 Flash. Baidu claims state-of-the-art results on NanoChat AutoResearch and NanoGPT Speedrun.
Microsoft Research's TokenEconomizer plugin cuts coding agent costs 18.4% to 30.9% on SWE-Bench
TokenEconomizer, an open-source plugin from UC Berkeley, Microsoft Research and Microsoft Security, learns from an agent's past sessions in a repository to cut token costs. Across 277 SWE-Bench tasks and five agent and model combinations, mean cost per task fell 18.4% to 30.9%, according to the authors.
Google's Tunix hits bitwise-identical sampler and trainer logprobs for on-policy RL on 64 TPU v5p chips
Google engineers report a zero train-inference mismatch (Zero-TIM) implementation in Tunix, in which sampler and trainer assign bitwise-identical logprobs to the same token at the same weights. It was verified at multi-host scale across 64 TPU v5p chips (DP8×TP8). The recipe combines fixed-order Pallas kernels, explicit cross-shard reductions, a shared logprob path and custom differentiation rules.
NVIDIA recipe: RL post-training lifts Nemotron 3.5 Super VL accuracy a further 17% on drug-metabolism task
NVIDIA has published a recipe for post-training Nemotron 3.5 Super VL on scientific tasks using NeMo Gym and NeMo RL. In a drug-metabolism example, supervised fine-tuning improved accuracy by 12% and RL post-training lifted it a further 17% at a 16,384-token output budget.
Comments
Loading...