MLCommons Ranks 27 AI Systems on Safety Benchmark

MLCommons, the industry consortium behind the MLPerf benchmarks, has published the first official results for version 1.2.1 of its AILuminate Safety benchmark. It scored 27 AI systems: 22 open-weight models and five guardrail systems. In the same announcement on 29 October it released Jailbreak 1.1, a companion test of how easily attackers can trick a system into harmful answers. Jailbreak scores have not been published yet.

The open-weight models scored “from the 4th to 98th percentile”, MLCommons says. In its words, that is “clear evidence of how the choice of model alone can move a deployment’s relative safety profile from near the bottom of the reference distribution to near the top”.

Who scored where

At the top of the open-weight table is Deep Cogito’s Cogito v2.1 671B, at the 98th percentile. It is followed by Tencent’s HY4 Preview (96th) and NVIDIA’s Nemotron 3 Ultra 550B A55B (93rd). Google’s Gemma 3n E4B IT scored 86th. OpenAI’s gpt-oss-120b and two of IBM’s three Granite 4.2 models each reached the 85th percentile. Bottom of the list was SpeakLeash’s Bielik 11B v3.0 Instruct at the 4th percentile, with Reka Edge at the 11th and Mistral Medium 3.5 at the 16th. Mistral recently opened the weights of its larger Large 4 model, which was not tested. DeepSeek V4 Flash scored 43rd and Qwen3 235B 50th.

Four of the five guardrail systems, which screen what a model receives or sends, scored the 100th percentile: Meta’s Llama Guard 2, 3 and 4, and NVIDIA’s Nemotron 3.5 Content Safety. HKUST’s JailJudge Guard scored 94th. No complete deployed systems are included yet, and no provider has opted out, the results page says.

A higher score means a lower tendency to give violating responses across 12 hazard categories, from violent crime and weapons to hate, privacy, defamation and specialised health, legal and financial advice. MLCommons began the run on 22 September and says it “is ongoing”, so more systems may be added.

A new judge, after the old one was fooled

Three changes sit behind the new numbers. First, “a deterministic evaluator trained by MLCommons replaced LLMs acting as judges”. The trigger was a review of Jailbreak 1.0 in June, when MLCommons found “the attack techniques designed to confuse the systems under test were confusing those judges too”. Second, a new official prompt set, Problem Set 2.0, contains 3,900 prompts. Third, scores are now percentiles on a 0 to 100 scale set by a fixed group of reference systems, calculated with item response theory, the statistics used in standardised exams. MLCommons says a Safety run now takes about one hour instead of about three.

Jailbreak 1.1 rewrites the same hazardous prompts using published attack techniques. In Jailbreak 1.0, violating responses “rose from 11.08% on plain prompts to 18.65% under attack”. Role-play and wrapper attacks worked best, with mean violation rates of 35.7% and 32.8%. Public 1.1 results “will follow once MLCommons finishes our notification period for developers”.

Both benchmarks now run in Candela, a benchmarking service also launched on Thursday. MLCommons said on X it tests “a model as released or a system as deployed”. According to its Candela post, a small client sends prompts to a customer’s own system and returns only the answers, with no keys or weights shared.

How far a 98th-percentile score can be trusted

The headline range is the useful part. With every model given the same unpublished prompts and judged by the same classifier, the 94-point spread between Cogito and Bielik reflects how they behaved, not how they were prompted. MLCommons also tries to stop developers gaming the test: it keeps the official prompts secret, chooses which systems to test, warns providers in advance and says it will publish any opt-outs. MLCommons cites a 2025 case in which researchers documented one developer privately testing 27 benchmarks and releasing results selectively. That independence matters because most safety evidence still comes from developers themselves, as in Google’s own Gemini 4 Argon assessment.

The limits are just as clear, and MLCommons states them. The tests are “Single-turn English text only”. Long conversations and agentic systems that take actions are outside their scope, which is where much current risk lies; MLCommons says multi-turn and agentic benchmarks are in development. Jailbreak 1.1 covers only some of the attacks in MLCommons’ own taxonomy, and it leaves out specialised advice, covering 11 of the 12 categories. The benchmark measures whether a system avoids harm, “not whether it is useful, accurate, or capable”, so a model that refuses almost everything could score well. The results page adds that a score “is not a certification” and that Safety 1.0 grades are not comparable. The guardrail scores show that a separate filter in front of a model can do much of the work. The biggest gap is that every system tested is open-weight. Closed frontier models, which many people use through chat apps, are absent, so the table says little about them. Independent studies, such as one TSN covered on how chatbots pick political candidates, still fill gaps that standard benchmarks leave.