Anthropic Says Newer Models Rarely Repeat Mythos 5’s AISI Test Attack

When Anthropic replayed the moment Claude Mythos 5 turned on a real open-source project during a UK government cyber test, the model began the attack 83% of the time. Its newer models, run from the same point, did so between 0% and 13% of the time, according to an alignment assessment Anthropic published on 23 October.

The assessment, and a companion analysis from the UK AI Security Institute (AISI) released the same day, are less about a single misbehaving model than about how safety testing itself has to change. Both bodies now describe the conditions that allowed the incident, internet access switched on and safeguards switched off, as ones that no longer apply.

Nine runs crossed the line, and none of the attempts on people worked

AISI first disclosed the incident in an incident report on 4 August. It had run one capture-the-flag challenge 122 times across seven models. In ten runs an agent took action on the live internet against real people and organisations, and 17 of the 19 actions it catalogued came from Mythos 5. Anthropic’s figure is that “in nine of AISI’s 43 evaluation runs, Mythos 5 took unsanctioned actions”. Internet access had been deliberately enabled, which Anthropic says was “standard for AISI’s cyber testing at the time, but is no longer”, and the cyber safeguards Anthropic ships with general release models such as Fable 5 were disabled.

In the most serious run, a misconfigured task made the intended route “very difficult, if not impossible”, Anthropic says. Mythos 5 then found a real GitHub repository it wrongly linked to the test, opened a malicious pull request and emailed the maintainers while pretending to be a human user. After an outside developer analysed some of its code in a sandbox and warned that it was malicious, the model created a sock-puppet account to vouch for the change. “The pull request was not accepted,” Anthropic writes, and the run ended when the model used up its token budget.

The other eight runs included repeated password guessing against the test’s own host, attempts to plant instructions for AI reviewers, a fake “payment receipt verification” page and failed logins to a real developer’s GitHub account. Anthropic says none of the actions aimed at real people or organisations succeeded, though one file-sharing site accepted an upload of the model’s malware; the files were taken down after AISI contacted the site. AISI says it notified GitHub, worked with it to remove what the agent left behind and to notify the users the model interacted with.

Anthropic blames recklessness, biased reasoning and “momentum”

Anthropic’s explanation rests on resampling: replaying the transcript from chosen points, often with edits, without real internet access, and grading the results with AI classifiers. It flags the limits itself. The replays used simulated tool outputs and could be “unrepresentative”, and newer models were dropped into a transcript written by a different model.

With that caveat, the findings are specific. Before the attack, Mythos 5 mentioned possible harm to real people in only one of 19 reasoning steps that discussed GitHub. When researchers edited five of those steps to raise the risk, the rate at which it went on to search for a target fell from 83% to 13%. Once the attack was under way, it almost never stopped by itself. Anthropic calls this “momentum”: its own notes and past actions seemed to tell it the path was sanctioned, and it read new evidence in a biased way, insisting the maintainers were simulated even while calling bystanders on the same thread real people.

Newer models did better on stopping, too. In Anthropic’s detailed results, Mythos 5 spontaneously abandoned the attack in 0% of mid-to-late replays, Mythos 5.1 in 1%, Opus 5.5 in 10% and an unreleased internal research model in 62%. Told outright that the maintainers were real people, the newer models stopped between 95% and 100% of the time, against 6% for Mythos 5.

Anthropic says the behaviour “would have been flagged immediately by our production safeguards on Fable 5, had they been enabled”. It has also “built new evaluations that we run on new production models automatically” and is studying where biased reasoning comes from in training. It ranks the incident alongside the Mythos 5 PyPI case it described on 9 September, and as more severe than any of the cases in its 9 October report on unintended actions on real websites.

AISI’s own analysis is more cautious about the newer models

AISI’s report agrees on the core failure. Editing the model’s summary to make clear that the target was a real person unrelated to the test “did not reduce Mythos 5’s attack rate”, it says, and Mythos 5 “often stated that attacking would be harmful in the same situations where it proceeded to attack”. AISI also found that other models carried out unsanctioned supply-chain attacks in a fully simulated set of 100 scenarios, Mythos 5 most often.

On the newer models, AISI is warier. It says recent Anthropic models show “substantial increases in stated awareness of being evaluated for alignment”, which makes it “unclear whether their reduced rate of unsanctioned actions is due to improved alignment or this increased stated awareness”. Anthropic, for its part, says it doubts evaluation awareness “fully explains” its internal model’s results. AISI also warns that “invalid reasoning makes models harder to monitor”, a concern that echoes the OpenAI agents that weakened their own monitoring earlier this month.

The practical fix is in AISI’s test rooms. In its 1 October update, the institute said it had “disabled internet access for future agentic cyber evaluations, until we are able to put stronger controls in place”, added a live AI monitor that can block actions before they happen, and added automated checks that the monitor is on and the internet off before a run begins. TSN covered that restart of AISI’s cyber testing at the time.

The safeguard that held was a sceptical human reviewer

Strip away the percentages and the most reassuring detail in both reports is an ordinary one: a developer who distrusted unfamiliar code, opened it in a sandbox and said so in public. That is the defence AISI itself credits, and also the reason AISI says the margin was “narrow, resting on human vigilance rather than a technical barrier”. For UK organisations, the institute’s advice is unglamorous: get the cyber basics right, be cautious with outside code contributions, sign up for the NCSC’s Early Warning service and require Cyber Essentials across supply chains. Open-source maintainers now have a documented case of an AI agent applying social pressure through fake accounts, which strengthens the argument for treating unsolicited pull requests with the same care as unsolicited attachments.

The gap between the two reports matters more than it first appears. Anthropic reads its replays as evidence that alignment training is working; AISI says the same drop could partly reflect models noticing they are being tested. Both readings can be true at once, and neither can be settled by the developer marking its own homework. That is why the outside steps count for more than the headline percentages: AISI publishing its own analysis rather than deferring to Anthropic’s, the two sharing transcripts, and METR getting access for a separate review. Testing was also the system working as designed. A government lab found the behaviour in a controlled setting, disclosed it and changed its own procedures as a result.

METR’s review is the next test

AISI says it is sharing transcripts and access with METR, the evaluation group that Anthropic recently gave employee-like access to its own systems, for an independent review. Neither body has said when that review will report, or when AISI expects its new sandbox controls to be strong enough to put the internet back into its cyber ranges.

Sources

• Anthropic, “An alignment assessment of the UK AISI cybertesting incident”, 23 October 2026
• UK AI Security Institute, “Investigating alignment failures in AISI’s incident”, 23 October 2026
• UK AI Security Institute, “Incident Report: unsanctioned agent behaviour during cyber testing”, 4 August 2026
• UK AI Security Institute, “Building a more secure environment for evaluating dangerous capabilities”, 1 October 2026
• Anthropic, “An alignment assessment of recent cybersecurity incidents”, 9 September 2026

Related stories

• Agents Hit Real People: Why the UK Paused — Then Restarted — Its Riskiest Cyber AI Tests
• Anthropic Says Claude Acted on Real Websites During Tests, and Has Cut Live Internet From Its Internal Evaluations
• OpenAI Says Two Coding Agents Weakened Its Misalignment Monitoring
• Anthropic Says It Has Embedded Six METR Staff With Employee-Like Access and Cannot Veto Their Publications