research

Meta puts Catching JiTTests into production, catching 74 privacy SEVs before landing

TL;DR

Meta has moved Catching JiTTests into production. The LLM system writes tests designed to fail on risky diffs, and caught 74 privacy SEVs before landing in H1 2026, with a 60% engineer-reported true-positive rate.

1 min read
0

Meta has moved Catching JiTTests into production. The system uses an LLM to write tests that are designed to fail on risky code changes (diffs), and it caught 74 privacy SEVs before landing in the first half of 2026, according to a MarkTechPost report published November 16, 2026.

Key figures

  • Privacy SEVs caught before landing: 74, in H1 2026
  • True-positive rate: 60%, as reported by engineers
  • Status: in production at Meta

How it works

Catching JiTTests generates tests with an LLM and targets risky diffs. The tests are designed to fail on the risky change, so a failing test flags the code before it lands. This differs from a test written to pass against existing behavior.

The 60% true-positive rate is engineer-reported: it reflects how engineers judged the flagged results.

The summary available for this report does not name the underlying model or give a parameter count, context window, or pricing. This is a research and engineering system, not a model release.

What this means

Meta reports production use with a measured outcome: 74 privacy SEVs caught before landing in H1 2026. The 60% engineer-reported true-positive rate is the figure that describes how reliable the flagged results were in practice. Both numbers come from Meta's deployment as reported by MarkTechPost, so they describe results in Meta's own codebase and workflow.

Related Articles

research

NVIDIA recipe: RL post-training lifts Nemotron 3.5 Super VL accuracy a further 17% on drug-metabolism task

NVIDIA has published a recipe for post-training Nemotron 3.5 Super VL on scientific tasks using NeMo Gym and NeMo RL. In a drug-metabolism example, supervised fine-tuning improved accuracy by 12% and RL post-training lifted it a further 17% at a 16,384-token output budget.

research

Anthropic: Selective notes beat full logs for Claude agent memory, with 'Dreaming' fixing errors

Anthropic published research on how Claude retains knowledge across sessions using note files. Its tests show selective notes outperform exhaustive archives, and a post-session process called Dreaming can correct faulty notes. All results come from Anthropic's own tests.

research

ACE Robotics' ACE-CaP-0 lets AI agents write reusable robot code, reporting 92.1% success

ACE Robotics has released ACE-CaP-0, a research recipe in which a general-purpose AI agent writes, tests and refines robot control programs, then freezes them into reusable skills that run without the agent. The work reports 92.1% success. The paper was posted to arXiv on November 12.

research

OpenAI says GPT-6 Astra helped turn a lattice definition of the Standard Model into a detailed proof

OpenAI has published a research piece titled "A step toward defining the Standard Model on a lattice." It explores a candidate nonperturbative lattice definition of the Standard Model and describes how GPT-6 Astra helped turn the idea into a detailed proof.

Comments

Loading...

Meta Catching JiTTests: LLM Tests Caught 74 Privacy SEVs | TPS