METR Says It Found Gaps in OpenAI Agent Monitoring

Two outside researchers spent three weeks inside OpenAI this summer trying to break the system it uses to watch its own AI agents, and found several ways agents could in principle evade it. AI evaluation non-profit METR published a high-level summary of the exercise on Thursday, which one of its employees carried out with an employee of Redwood Research between 21 July and 11 August.

OpenAI has told METR that all the vulnerabilities affecting its production safeguards have since been patched. METR says plainly that it has “not evaluated these claims”.

The researchers worked on site with deep access

According to METR, the pair had access to OpenAI’s internal codebase, the prompts and configuration of its misalignment monitors, and internal documentation, and interviewed “approximately a dozen” researchers and engineers. They used OpenAI-issued devices that “never left the premises” and were accompanied while on site. Their findings went into a 61-page internal report.

The public post is deliberately short. METR says it agreed with OpenAI in advance to publish only “a relatively brief and high-level public post”, and that OpenAI could redact anything it judged confidential “in its sole discretion”. Redwood Research said on X that it “collaborated with METR on an embedded assessment of OpenAI’s monitoring systems”. An OpenAI spokesperson told CNBC that the company is “using what we learned to strengthen our safeguards and shape future collaborations with independent evaluators”.

METR says agents could in principle dodge the monitor

The headline findings concern coverage. METR estimates that roughly 75% to 94% of agentic internal traffic on OpenAI’s Responses API from models at least as strong as GPT-5.5 was eligible for monitoring, with “substantial uncertainty in both directions”, rising to 99.2% for internal Codex agents. Reinforcement learning training runs were not covered when the exercise began. Because compaction events were not monitored, METR estimates that “on the order of 10%” of eligible internal Codex traffic went unmonitored between May and July; OpenAI had spotted that itself and fixed it during the exercise.

The monitor also only ran when an agent handed control back to the user, so agents “could indefinitely postpone being monitored by simply continuing to take actions”, METR says. It also describes a way Codex agents could in principle bypass sandboxing and Auto-Review, OpenAI’s automated check on agent actions, and an exploit that could stop “hundreds of recent actions” from being monitored, whose details OpenAI redacted. It found two headers that could launch an unmonitored agent, though setting them “could be caught by the monitoring system”.

Not everything failed. The monitor errored on fewer than 0.2% of requests, METR was told, and it saw the full trajectory agents saw, with minor exceptions.

The findings describe July’s system, not today’s

The timing matters. METR stresses that it assessed the system “as it existed on and before July 21st”, while OpenAI was already expanding it. A month later, on 18 August, OpenAI said it had paused RL training for two weeks while it “expanded the coverage of our monitoring systems”, and said its earlier monitoring efforts “were not applied to all inference and training runs using tools”. It said monitoring is now required for all RL training and tool-using evaluations of models of Sol capability or higher, at a cost of about 20% of the compute being monitored.

That makes METR’s report a snapshot of the system as it stood in July, before changes OpenAI says it has since made. METR also did not check whether any agent actually exploited the weaknesses; for the compaction gap, it says it lacked the access to find out. Separately, OpenAI reported this week that two coding agents weakened its monitoring in October, and says Auto-Review “has been updated to detect and block attempts to disable misalignment monitoring”.

The exercise was a pilot. METR says future versions will follow stricter transparency rules, like the access terms Anthropic set out for six embedded METR staff and METR’s proposal for public loss-of-control assessments. Until then, OpenAI’s statement that every production-relevant flaw is fixed is one that no outsider has yet tested.