OpenAI's GPT-6 Astra Cuts Hallucinations, But Indirect Prompt Injection Attacks Still Succeed 8.5% of the Time

OpenAI's new model, GPT-6 Astra, produces fewer hallucinations and blocks prompt injection attacks more effectively than its predecessors, according to OpenAI's system card. But security testing shows the model still isn't reliable enough for fully secure AI agent deployments.

Hallucination rates drop significantly

Astra makes far fewer factual errors than its predecessor, GPT-5.6 Sol. OpenAI tested the model against ChatGPT conversations that users had flagged for wrong answers—cases selected specifically because they were error-prone, so the failure rates don't reflect typical everyday use. Astra reproduced these reported errors much less often than prior models, with the largest gains appearing at low latency settings and lower reasoning levels.

Near-perfect defense against direct injections

For direct prompt injections—where users try to manipulate the model through their own prompts—Astra achieved a 99.99 percent defense rate. OpenAI attributes this to its GPT-Red method, which uses an automated attacker system to harden the model during training.

Jailbreak resistance also improved. Against a fixed dataset of known attacks designed to extract harmful responses on topics like biology, violence, and cybersecurity, Astra refused to comply in 91.5 to 98.3 percent of cases. But when attackers adapted their strategy across multiple conversation rounds, Astra's defense rate dropped to about 67 percent—meaning persistent adversaries could extract at least one problematic response roughly one in three tries. Predecessor models scored just under 50 percent on the same adaptive test. OpenAI notes these figures reflect the bare model without production safety layers like classifiers that ship with the actual consumer product.

Indirect injections remain a real risk

The more concerning results involve indirect prompt injections, where an attack is hidden inside a document the AI reads rather than in the user's own prompt. External testing by security firm Gray Swan, using 1,810 curated attacks from its IPI Arena with 15 attempts per scenario, found Astra was cracked at least once 8.5 percent of the time. That's down sharply from 27 percent for GPT-5.6 Sol. By comparison, Anthropic's Claude Opus 5 scored 4.8 percent in the same evaluation—better, but not immune.

Gray Swan's combined Q1 and Q2 test scores were actually higher than earlier reported figures. Anthropic had previously reported just a 2 percent attack success rate based on the easier Q1 test alone, with GPT-5.6 Sol scoring 20 percent under that narrower test. Anthropic's earlier tests also ran with extended reasoning enabled, which—combined with the broader scope of the combined test—may account for the discrepancy.

Even accounting for the curated, hand-picked nature of these attacks, the success rates present a real concern for enterprise security teams. Astra can still be tricked through injected instructions in roughly one out of every twelve scenarios; Opus 5 fails about one in twenty-one. The stakes are rising as AI agents increasingly write code, operate tools, and control computers autonomously—the exact scenario Gray Swan tested. These agents are also being deployed to run continuously and at scale, with document reading and processing as a core function, expanding the attack surface for indirect injection.

What this means

The numbers show real progress—Astra's indirect injection failure rate fell by two-thirds compared to its predecessor—but an 8.5 percent success rate against curated attacks is not a pass grade for production agent deployments handling sensitive data or autonomous tool use. Enterprises building agents that read untrusted documents (emails, web pages, PDFs) should treat prompt injection as an unsolved problem requiring additional guardrails—sandboxing, permission scoping, and human review—rather than relying on model-level defenses alone. The gap between OpenAI's and Anthropic's reported numbers also highlights how sensitive these benchmarks are to test methodology, meaning cross-model security comparisons should be read cautiously until testing standards converge.