HomeAIAI AgentsSame AI Model Scores 33% or 62% Depending on Its Coding Tools

Same AI Model Scores 33% or 62% Depending on Its Coding Tools

The same AI model can score very differently depending on which coding tools it runs inside, according to a new release from Hugging Face’s FineEnvs team. In its own tests, one small model with exactly the same weights solved 62% of a set of data-analysis tasks in one coding tool and 33% in another [1][2].

The team’s response is an open framework that trains models inside unmodified coding tools [1]. Hugging Face chief executive Clément Delangue announced it on X on 5 October [2]. All results below are Hugging Face’s own.

Harness, RL, pass@1 and logprobs: the terms used here

  • Harness. The program around a model that runs the loop: which tools it gets, what context it sees, how its replies are parsed and when to stop [1]. Claude Code, Codex, OpenCode and Mini-SWE-Agent are harnesses.
  • Reinforcement learning (RL). The model attempts a task, a grader scores the result, and the model is nudged towards what scored well. In “agentic RL” the attempt is multi-step: write code, run it, read the output, try again [1].
  • Pass@1. The share of tasks solved on the first graded attempt [3].
  • Logprobs. The probability the model gave each word fragment (token) it produced, stored as a logarithm. RL needs the exact tokens and these numbers, not just the final text, to know precisely what to reinforce [1].

Why coding tools change the score

Each harness gives the model different tools, prompts and rules. A model trained in one can call a tool by a name another spells differently, and the call is rejected before it runs [1]. The guide also cites a measurement of GLM-5.2 at 23% in one harness and 52% in another on SWE-bench Pro [1]. A separate preprint TSN covered found that a minimal coding agent can match far more elaborate harnesses, another sign that scaffolding choices move results.

The usual fix is to rebuild a harness as a training environment. Delangue says this means “most models get trained in a scaffold nobody actually ships” [2].

How FineEnvs trains inside unmodified coding tools

The trick, Delangue says, is “a proxy, not a rewrite” [2]:

  • The capture proxy (in OpenEnv). The harness thinks it is talking to a normal model API. In fact it talks to a proxy that speaks the four formats coding agents use (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages and Gemini), forwards requests to a vLLM server running the model in training, and records the exact token IDs and logprobs [1][2].
  • The trainer (TRL). Hugging Face’s TRL library trains on those recordings with Async GRPO, which compares several attempts at the same task and reinforces the better ones. The guide cites the merged TRL pull request #6947; an earlier one, #6420, added the first version of the integration [1][4][5]. TSN noted the library’s progress in its round-up of TRL 1.15 and other AI developer tools.
  • Tasks and sandboxes (Harbor). Harbor supplies the tasks and runs each attempt in an isolated sandbox (E2B in these runs) [1].

The guide says OpenEnv has validated ten harnesses end to end; Delangue says “10 harnesses run through it today, none modified” [1][2].

LFM2.5-2.6B rose from 42.2% to 54.2% across four coding tools

The test model was Liquid AI’s 2.6-billion-parameter LFM2.5-2.6B, scored on 250 held-out data-analysis tasks from FineEnvs’ SmolDataEnvs suite under four harnesses [1][3]. One of the guide’s authors is from Liquid AI [1].

  • Before training: 62% under Mini-SWE-Agent, 33% under Claude Code. The average across four harnesses was 42.2% [1].
  • Trained across four harnesses (OpenCode, Claude Code, Codex, Mini-SWE-Agent): the average rose to 54.2% [1][3]. At release, the model card lists OpenCode 49.6%, Claude Code 48.8%, Codex 53.6% and Mini-SWE-Agent 64.8% [3].
  • Trained in OpenCode only: the OpenCode score rose from 34% to 58%, and the overall average reached 52.3% [1]. The guide says the 1.9-point overall gap between the two runs “is within the noise”. The difference lies in where the gains landed. The multi-harness model did better under Claude Code and Codex, while the OpenCode-only model did better in OpenCode [1].
  • Fewer tool calls: a small reward bonus for solving tasks in fewer steps helped the multi-harness model make 31% fewer tool calls than the base model, on the tasks both solved [1][3]. The guide notes there was no run without the bonus, so the bonus’s effect cannot be isolated [1].
  • Imitation did less well: fine-tuning on a 27B model’s successful attempts reached 47.5% with OpenCode data and 43.1% with all 3,189 attempts from four harnesses, within the noise of the base model. Both are below the RL runs [1]. (Delangue’s post attaches 47.5% to the 3,189-attempt run; the guide does not [1][2].)

Hugging Face has released the proxy, trainer integration, tasks, fine-tuning data, training code and seven models [1][2][3]. The weights carry Liquid AI’s LFM Open License, and the model card says it “is not an official Liquid AI release” [3].

Benchmark tables need a harness column

Hugging Face’s point is that a score describes a model in a particular harness, not the model in general [1]. If one model with identical weights can swing from 33% to 62% on the same tasks because of the tool wrapped around it [1], then a leaderboard that lists only the model name is leaving out a variable as large as many generational upgrades. That cuts both ways for buyers. A vendor can pick the harness that flatters its model, and a team can wrongly reject a model that would have done well in the tool it actually uses. TSN has seen the pattern in vendor claims such as Factory’s tests of Droid against Codex and Claude Code, where the harness is part of the product being compared.

The multi-harness result is the more interesting finding for developers. Training in one tool lifted that tool most, while training across four spread the gains, with a similar overall average [1]. That suggests models trained only in a lab scaffold may underperform once dropped into the coding tools people actually use, and that cheap proxies which record real tool traffic could become a standard part of training. The fewer-tool-calls result points the same way: it rewards finishing tasks efficiently inside real software, not just reaching an answer [1][3].

The limits are clear in Hugging Face’s own write-up. Every run used a small model of 2 to 2.6 billion parameters, trained once, and larger runs are still being set up [1]. The trained model was scored only in the four harnesses it trained in, and most of the OpenCode-only model’s gain stayed in OpenCode [1]. The tasks are data analysis, and Liquid AI’s model card does not recommend LFM2.5-2.6B for agentic coding [6]. Liquid AI says it trained LFM2.5 inside Hermes Agent, OpenClaw “and other harnesses”, not the four used here [1][7]. And these are single-run, company-reported results with the best checkpoint picked on the test set [1][3], so independent replication is the next thing to look for.

Sources

  1. Adithya S Kolavi, Joel Niklaus, Sergio Paniego Blanco, Leonie Monigatti, Amine Dirhoussi, Ben Burtenshaw, Lewis Tunstall and Leandro von Werra, “The ultimate guide to multi-harness RL” (Hugging Face / Liquid AI), published 24 September 2026, updated 1 October 2026 (PDF). https://fineenvs-multi-harness-rl.hf.space/the-ultimate-guide-to-multi-harness-rl.pdf. Interactive version: https://huggingface.co/spaces/FineEnvs/multi-harness-rl
  2. Clément Delangue (@ClementDelangue), X post, 5 October 2026, 15:48 BST (read via the X API). https://x.com/ClementDelangue/status/2107120717980471638
  3. FineEnvs, “LFM2.5-2.6B-multiharness-RL” (Hugging Face model card). https://huggingface.co/FineEnvs/LFM2.5-2.6B-multiharness-RL
  4. Hugging Face TRL, pull request #6420, “Async grpo OpenEnv harness rollout” (merged 24 July 2026). https://github.com/huggingface/trl/pull/6420
  5. Hugging Face TRL, pull request #6947, “AsyncGRPO: train OpenEnv harnesses from validated token captures” (merged 1 October 2026). https://github.com/huggingface/trl/pull/6947
  6. Liquid AI, “LFM2.5-2.6B” (Hugging Face model card). https://huggingface.co/LiquidAI/LFM2.5-2.6B
  7. Liquid AI, “LFM2.5-2.6B: Deploy Agents Everywhere” (blog), 4 August 2026. https://www.liquid.ai/blog/lfm2-5-2-6b

Follow @tsnmediahub on X for breaking AI news as it happens.

Follow on X

Share this story

Latest stories

  1. WashU Makes Edible Fungal Protein From Waste CO2
    AI Science & Health

More in this category

Latest stories

  1. WashU Makes Edible Fungal Protein From Waste CO2
    AI Science & Health

Explore the TSN Atlas: Every topic we cover, mapped. Click a bubble to dive in.