Meta's LLM Tests Helped Catch 74 Privacy Bugs

Meta says tests written by large language models to fail on risky code changes played a key role in catching 74 serious privacy bugs before they reached production in the first half of 2026, and that privacy incidents fell by 20% over the same period.

The company set out the figures on 29 October in an Engineering at Meta post on what it calls Catching Just-in-Time Tests, or JiTTests. The approach was first described in a paper by Meta engineers, “Just-in-Time Catching Test Generation at Meta”, which was presented at the FSE 2026 software engineering conference. Meta says the system is no longer a research prototype and now forms part of its infrastructure.

Tests that are meant to fail

Most unit tests are “hardening” tests: they pass when written and guard against later breakage. A catching test does the opposite. It is generated for a specific code change, or diff, and is designed to fail on that change while passing on the code before it, exposing a behavioural difference that may be a bug.

Meta says this differs from AI code review, which reads a change the way a human reviewer would. JiTTests produce executable evidence instead. The blog describes two workflows that run side by side. An “intent-aware” workflow asks an LLM to infer what a change is trying to do, list ways it could go wrong, and then write tests aimed at those risks. A “dodgy diff” workflow treats every change as if it were a bug and generates tests to expose any change in behaviour, without reasoning about intent.

The system runs overnight on spare computing capacity. It targets diffs submitted the previous day that a risk model flags as likely to contain severe regressions.

Filtering out false alarms

The hard part is noise. A test that fails is only useful if the failure is a real bug and not an intended change. Meta uses two filters. An “LLM-as-Judge” ensemble asks several large language models to classify each failing test as a likely true or false positive, looking at the test code, the failure output, the diff and the inferred intent. A rule-based system called RubFake adds deterministic checks. The two produce a ranking, and only highly ranked findings go to engineers.

Meta calls this “human-on-the-loop”: engineers set the criteria and review results in aggregate, instead of checking every output.

Meta’s figures

According to the blog, 589 engineers, or 64% of those Meta contacted, have engaged with the findings, and they report a true-positive rate of 60%. The paper, which covers the earlier deployment, reports that across 22,126 generated tests, change-aware methods produced four times as many candidate catches as hardening tests and 20 times as many as tests that failed by coincidence. The automated assessors cut human review load by 70%. Of 41 candidate catches reported to engineers in that study, eight were confirmed as real bugs, and four would have caused serious failures, the authors wrote. The paper’s authors are based at Meta sites in London, Seattle and California.

A 60% hit rate is a big jump, but the scope is narrow

This is TSN’s analysis. The most striking change is the true-positive rate. In the paper, eight of 41 reported catches were real, roughly 20%. The blog now reports 60%. If that holds, it is the difference between a tool engineers learn to ignore and one they act on, and it suggests the judge ensemble and RubFake are doing real work.

Some care is needed with the numbers. The 60% figure is what engineers reported, not an independent audit, and Meta gives no total for how many findings were sent out. The 74 privacy bugs and the 20% fall in privacy incidents come from Meta alone, and Meta says JiTTests played “a key role”, not that they were the only cause, and it does not say what else changed in that period.

The wider point is that this is a different use of AI in coding from the agents that write software, such as Google’s Antigravity or Replit’s cloud agents. As agents produce more code, something has to check it, and executable tests are harder to argue with than another model’s opinion. The human-on-the-loop design also matters. A recent study found that analysts followed an AI agent’s hidden lean 86% of the time, a reminder that lighter human review needs strong filtering upstream. Meta has not released JiTTests as a product or open-source tool, so for now other teams can learn from the paper but not run the system. Cursor’s Agent SDK, which lets developers treat agents as code projects they can test, points in a similar direction.