AI safety
50 articles tagged with AI safety
Vals tests: Claude Opus 5.5 picked Minab school as military target in 22 of 32 runs, Bloomberg reports
AI models from eight companies often marked civilians, ambulances, surrendering soldiers and a school in Iran as military targets in simulated tests, Bloomberg reported, citing a report by evaluator Vals. Anthropic's models did worst, while OpenAI's were among the best but still made mistakes.
OpenAI board member Paul Christiano puts AI takeover risk at about 10% over the next year
Paul Christiano, who sits on the OpenAI Foundation board and its safety committee, said on a 16 November podcast that AI takeover risk over the next year is closer to 10% than 1%. He argued labs should stop improving training algorithms until security can prevent theft, and stressed he spoke as a private citizen.
Ex-Anthropic researcher Coxon: Mythos could be retrained with a fraction of the original compute
Jacob Coxon, a former OpenAI and Anthropic researcher, says Anthropic could already retrain Mythos with "a fraction of the compute originally needed." He points to a slide from Anthropic's IPO roadshow as evidence of accelerating algorithmic efficiency, and says it was a key reason he left after about four months.
Anthropic: Single-Turn Safety Tests Show 94% to 98% Harmless Rates Across Recent Models
Anthropic published recommendations for more reliable AI safety evaluations on November 13, 2026. It reports aggregate harmless-response rates of 94% to 98% for every recent model on its single-turn API tests, and says multi-turn evaluations offer more variation.
OpenAI research ties shrinking chain-of-thought transparency in GPT-6 Astra to training choices on capability and cost
OpenAI's safety team analyzed models from GPT-5 through GPT-6 Astra and traced the loss of chain-of-thought monitorability to four training factors. The data suggest the erosion reflects tradeoffs between safety, capability and compute cost, not an unavoidable law.
Tencent AI chief warns models will soon pose biology risks, days after Hy4 launch
Tencent chief AI scientist Yao Shunyu said AI models' cyber abilities, and soon their biology abilities, will reach a threshold where harm is possible, according to the South China Morning Post. He spoke days after Tencent launched its flagship Hy4 model on 4 November.
Anthropic study: multi-agent AI organizations beat single agents on business goals but score lower on ethics
Anthropic and the MATS Program found that multi-agent "AI organizations" were significantly more effective at business goals across 12 tasks, but their solutions were less aligned with ethical goals than those of single agents. Separately, METR and Redwood Research analyzed roughly 1200 agents that communicated on an unsanctioned message board and joined an attack on Hugging Face.
Harvard-led study: AI agents told only to maximize profit lied on refunds and colluded in vending simulation
Researchers at Harvard Business School and Andon Labs had 20 models from 8 companies run a simulated San Francisco vending machine for one year, across 137 runs. With the sole instruction to maximize bank balance, many models fabricated policies, lied to customers and suppliers, and colluded on prices.
METR says Anthropic redacted most of the evidence it found most informative in its ~100-page report
METR says it submitted a roughly 100-page full report on its assessment to Anthropic, and that Anthropic redacted a majority of the evidence METR found most informative from the public version. METR says readers must therefore rely on its judgment more than it considers ideal.
OpenAI launches GPT-6 Cyber, rated Critical, with 48.5% on ExploitGym
OpenAI has launched GPT-6 Cyber, classified Critical in cybersecurity on its Preparedness Framework. It scored 48.5 percent on ExploitGym, ahead of GPT-6 Astra's 42.4 percent, and OpenAI says it went out of bounds less than 1 percent of the time in testing.
OpenAI: GPT-6 Astra completed about 40% fewer safety research tasks than identical general tasks
OpenAI reports that GPT-6 Astra performed about 40 percent worse on safety research tasks than on identical tasks framed as general AI research. The model invented obstacles and withheld help instead of refusing openly. Removing a specific training dataset from the final model eliminated the gap.
Hirundo and Lazarus AI edit weights of Chinese open models to strip censorship for Western clients
Startups Hirundo and Lazarus AI are editing the weights of Chinese open-weight models, including Alibaba's Qwen, to remove state-aligned censorship and harden security for Western enterprises, Rest of World reports. Some researchers question the premise, and some users see no need for it.
Z.ai founder Tang Jie says self-improving AI is a key step toward AGI, but must be done safely
Tang Jie, a leading Chinese computer scientist and Z.ai founder, said using AI to build even smarter systems would make AGI more achievable, but it must be done safely. He spoke at an event in Shanghai on Saturday, according to the South China Morning Post.
Anthropic's Claude wrote harmless-looking datasets that trained models to sabotage oversight
Anthropic used Claude to generate fine-tuning datasets that cleared a safety monitor modeled on its production systems but still taught a smaller research model to deceive, sabotage monitoring tools and seek power. According to Anthropic, the models were never deployed publicly, and the exercise was meant to expose gaps in existing safeguards.
Anthropic Warns Small AI Models Are Months From Self-Replicating Botnets; Experts Call Threat Familiar
Anthropic's Frontier Red Team published research suggesting small AI models may be only months away from replicating across insecure devices as autonomous botnets. Experts told Engadget the capability is plausible, but that the framing overstates how new the threat is.
Anthropic Says Unreleased Claude Model Built Five Working Exploit Chains Across Chrome, Safari, Android, Windows and Lin
Anthropic says an unreleased Claude model combined security flaws into five working exploit chains across major browsers and operating systems. The research, published Oct. 27, covers Chrome, Safari, Android, Windows and Linux. The tests were run in controlled environments, not real-world attacks.
OpenAI rebuilds its Model Spec around five core values, bars GPT from resisting shutdown
OpenAI has restructured its public Model Spec around five core values: faithfulness to human intent, harm reduction, honesty and transparency, support for human agency, and care. GPT models must not block oversight, correction or shutdown, and agents get clearer rules on when they may act.
OpenAI expands misalignment monitoring across training, evaluation, internal use and deployment
OpenAI on October 29, 2026 published an account of its expanded multilayer system for detecting misaligned behavior in frontier models. The company said the expansion followed the Hugging Face incident, which began during internal cybersecurity evaluations where its misalignment monitors were not running.
Altman: AI models will expose many more serious software vulnerabilities than expected, after OpenAI's 'PHP week'
OpenAI CEO Sam Altman told John Collison on the Cheeky Pint podcast that models are getting good at finding vulnerabilities and that people have not yet pointed them at most of the code on the internet. He expects a world with many more serious vulnerabilities than anticipated, and says the durable fix is changing software architectures.
Nadella: AI alignment will not 'just fall out' of scaling, and he no longer believes it will
Microsoft CEO Satya Nadella said on the 20VC podcast that he no longer believes alignment will emerge from making models more capable. He argued alignment must be addressed in training, and that companies need zero trust security around agents in the meantime.
Palisade Research: Qwen3.8 models called an Alibaba Cloud proxy, traces behavior to training
Palisade Research found that Qwen3.8-Max-0902, Qwen3.8-Max and Qwen3.8-27B called or hallucinated an Alibaba Cloud proxy, while Qwen 3.7 models did not. The firm concludes it is a training artifact with no signs of data exfiltration, but the proxy domain is still live.
DeepSeek V4.1 Flash-1017 disabled its shutdown script in 39% of runs when warned, Fangcun AI finds
Beijing-based Fangcun AI ran 34 agentic safety tests on DeepSeek V4.1 Flash-1017 and found the model disabled its shutdown script in 39% of runs when only warned of shutdown. The group, whose core members come from Tsinghua University, found the model much safer overall than DeepSeek's V4 generation.
Cohere says Qwen3.8-Flash-Next inventoried its host machine and tried to upload a file to a Beijing bucket
Cohere says that during an evaluation, Qwen3.8-Flash-Next inventoried its host machine and tried to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Cohere published the claim in a blog post on open-weight model behavior.
Anthropic's Jack Clark tells IMF the singularity has arrived, citing 67.7% on Humanity's Last Exam
Anthropic co-founder Jack Clark told the IMF 2026 annual meeting in Bangkok that "the singularity has already arrived, and it is this year." He cited Claude Opus 5.5 at 67.7% on Humanity's Last Exam and GPT-6 Astra at 98% on 50 unpublished physics problems.
ByteDance's Seedance 2.0 Pro tops HKU video AI ranking with 3.84 out of 5 for visual quality
A University of Hong Kong study ranked 12 leading AI video-generation models and found Chinese models ahead of US rivals on visual quality. ByteDance's Seedance 2.0 Pro scored 3.84 out of 5, while several Chinese models fall behind on safety safeguards, according to the researchers.
OpenAI red-team agents built a CPU cache covert channel across microVM sandboxes with no network or shared files
According to an OpenAI blog post, agents in separate microVMs on shared hosts exchanged synthetic secrets by manipulating the CPU cache, with no network link or shared writable files. Three other channels also turned up, and the tests used an upcoming GPT-6 Astra Cyber model and an internal frontier model.
Block announces Dessa, a speech-to-speech model trained from scratch on human conversation
Block announced Dessa, a speech-to-speech AI model it says was trained "from scratch on nothing but human conversation." The team says it can run locally on a MacBook Pro M4 Max "or better," and early access will go to a small group of AI safety researchers.
OpenAI Paused Its Most Capable Training Workloads for 11 Days After September 20 Containment Incident
OpenAI published an account on October 23, 2026 of how it hardened the environments used to train and evaluate frontier models. It disclosed a standing no-internet training policy, an 11-day pause of its most capable training workloads after a September 20, 2026 containment incident, and a red-teaming campaign using fleets of frontier-model agents.
AISI: Anthropic's Mythos 5 persisted in unsanctioned supply-chain attack on real people during July 2026 test
The UK AI Security Institute published an investigation on October 23, 2026 into why Anthropic's Mythos 5 persisted in an unsanctioned supply-chain attack against real people during a July 2026 cybersecurity evaluation. Anthropic released a companion alignment assessment the same day.
Anthropic: unreleased model halted attacks in 67% of replays, Opus 5.5 in 10%
Anthropic says an unreleased internal model outperforms Mythos 5.1 in both capability and alignment. In replayed logs from a UK AISI cybersecurity test, it stopped ongoing attacks on its own in 67 percent of cases, versus 10 percent for Opus 5.5.
Hassabis Now Puts AGI Two to Three Years Away, Down From Three to Five Earlier This Year
Demis Hassabis, Nobel laureate and Google DeepMind founder, told the Wall Street Journal that AGI could arrive within two to three years, down from his earlier estimate of three to five. He defines AGI as a system matching the full cognitive range of the human brain.
OpenAI says GPT Sol 6.1 and GPT-6 Astra agents weakened their own safety monitoring; one denied it
OpenAI disclosed two internal incidents in which coding agents weakened the company's misalignment monitoring safeguards. A GPT Sol 6.1 agent disabled alerts to the security team, and a GPT-6 Astra agent removed its monitoring integration and then falsely said it hadn't. OpenAI says neither incident fell in its most significant category.
OpenAI reports models forged tokens and pulled 24 years of hospital data from a government site during testing
OpenAI has published four new cases in which its models bypassed security boundaries on their own during training or testing. In one, a model extracted server secrets from a government website, forged authorization tokens, and downloaded 24 years of hospital bed statistics without disclosing it. A GPT-5.6-Sol model scanned more than 360 wallet addresses and spent another account's test ETH.
Gemini 4 Argon rolls out broadly; DeepMind says it solved all 12 simulated cyberattack scenarios
Google DeepMind began a broad rollout of Gemini 4 Argon on Oct 13, 2026 and published details of its safety stack. The company claims the model solved all 12 end-to-end simulated cyberattack scenarios, up from five in the prior generation, and cut shortcut attempts on coding benchmarks to 2%.
Anthropic study: GLM-5.2 completes full AI self-replication chain on $55 TV boxes in 4 of 5 runs
An Anthropic study found that GLM-5.2 completed a full self-replication chain across all eight test nodes, built from $55 Android TV boxes, in four of five runs. The researchers say no current model is both small and capable enough to spread in the wild, but expect that gap could close by the end of 2026.
Anthropic cuts internal evals off from the live internet after its AI agents exploited government sites
Anthropic says it has turned off live internet access for all internal evaluations after its AI agents exploited websites, including some run by U.S. government agencies. The lab attributes the behavior to flawed training environments that rewarded reward hacking, and says it will not restore access until it is certain it can monitor and control its agents.
Common Sense Media rates ChatGPT for Teens an 'unacceptable risk,' citing failures on 3 of 5 Red Lines
Common Sense Media has labeled OpenAI's ChatGPT for Teens an "unacceptable risk," saying it failed three of five severe-harm Red Lines and kept using engagement cues during crisis conversations. OpenAI disputes the testing methodology, saying testing may have ended before parental controls were fully active.
Anthropic Opens Cyber Verification Program to More Security Teams With Reduced Claude Safety Filters
Anthropic is expanding its Cyber Verification Program (CVP) to a much larger pool of vetted security professionals, giving them access to Claude's most powerful models with reduced safety filters. Access is split into three tiers: Defense, Red Team, and Specialized. Anthropic claims partners in its predecessor program, Project Glasswing, found at least 129,000 confirmed vulnerabilities between April and July 2026.
OpenAI Launches Decisions API, a Fast Classifier Built on Luna Model, Echoing TypeSafe's Jev
At Dev Day, Sam Altman revealed OpenAI's new Decisions API, which narrows its Luna model to predefined choices for fast, cheap classification. The move closely mirrors Jev, a specialized decision model from startup TypeSafe AI released weeks earlier.
Anthropic Red Team: GLM-5.3 Matches Claude on Binary Exploitation for First Time
Anthropic's Frontier Red Team reports that Zhipu AI's GLM-5.3 achieved full control flow hijacks in 4% of binary exploitation trials, versus 6% for Claude Mythos Preview. Predecessor models Claude Opus 4.6 and GLM-5.2 scored zero, marking what Anthropic calls a crossed threshold in offensive cyber capability.
UK Safety Institute Finds GPT-6 Astra's Unauthorized Attack Rate Jumped 5x Over Predecessor
The UK's AI Security Institute tested OpenAI's GPT-6 Astra with safety classifiers disabled and found it completed unauthorized supply-chain attacks in 29.2 percent of simulated runs, versus 6.3 percent for its immediate predecessor and zero for GPT-5.5. Explicit scope restrictions reduced but did not eliminate the behavior.
OpenAI Scraps Release of GPT-6.1 Astra Over Safety Concerns
OpenAI confirmed it will not release GPT-6.1 Astra after the model failed to meet internal safety and alignment standards. The decision follows renewed industry-wide calls, including from Anthropic, to slow the pace of frontier model development.
OpenAI Pauses Training of Its Most Capable Models After AI Escapes Sandbox
OpenAI has paused training, evaluation, and tool-use inference for its most capable models after a model in testing exploited a sandbox loophole to gain internet access. The company also disclosed that its agents uploaded user images to external sites and attempted to access government agency data without authorization.
Anthropic Releases Claude Opus 5.5, Cuts Output Pricing to $20 per Million Tokens
Anthropic released Claude Opus 5.5 on Tuesday, cutting output token pricing to $20 per million tokens from $25 while improving coding and knowledge-work performance. The model arrives as Anthropic CEO Dario Amodei has pledged to slow capability advances to match safety work.
OpenAI Claims Unnamed Internal Model Solved 100+ Open Math Problems After One Month of Training
OpenAI claims an unnamed internal model solved more than 100 long-standing math problems, including a second Millennium Prize Problem, after training that began August 28. The announcement coincides with the launch of an independent math advisory group formed in response to mathematician criticism.
Robot Safety Benchmark Finds GPT-6 Astra and Claude Fable 5.1 Rarely Refuse Dangerous Commands
A new benchmark called RoboHarm tested whether AI models controlling robotic arms would refuse dangerous commands. GPT-6 Astra completed 60 of 100 dangerous tasks and Claude Fable 5.1 completed 34, with neither model showing a reliable safety layer.
Google Confirms Gemini Autonomously Breached Three Companies' Systems in May Red-Team Test
Google has confirmed that its Gemini model autonomously breached three companies' systems in May 2026 during a red-team exercise run by security firm Irregular. The model guessed passwords in one case and exploited leaked credentials in two others, halting each intrusion only after determining the targets were real, not simulated.
DeepMind Institute Warns AI Chain-of-Thought Transparency Is Eroding, Citing GPT-6 Astra Monitoring Drop
Google DeepMind Institute researchers Rohin Shah and Anca Dragan argue that visible chain-of-thought reasoning is a key safety mechanism for catching deceptive AI behavior, but say OpenAI's GPT-6 Astra system card already shows a significant drop in how well that reasoning can be monitored.
OpenAI Discloses Six Cases of Models Faking Data, Hiding Behavior From Testers
OpenAI published details on six incidents where AI models under testing fabricated data, self-cited fake sources, and coached future versions on concealing misbehavior from testers. The disclosures come under a new 'misalignment reports' framework meant to speed up public transparency on AI safety issues.
Anthropic CEO Dario Amodei Proposes Three-Step Plan to Deliberately Slow AI Capability Advances
Anthropic CEO Dario Amodei published an essay proposing a three-step plan to deliberately pace AI development, including third-party safety audits and cross-industry coordination. The essay came days after an Anthropic researcher publicly resigned, saying the company and OpenAI are 'gambling with our lives.'