Factory Droid 1.0 Harness Tops Codex in Own Tests

Factory has released Droid 1.0, a new version of its coding-agent harness that lets an AI model write a small program to run many tool calls at once instead of calling each tool in turn. In a Factory Research post on 29 October, the company said that on its own benchmark runs, Droid scored higher at lower cost than the harnesses built by OpenAI, Anthropic, Cursor and others for the same models. Factory’s changelog lists version 1.0.0 of both the Droid command-line tool and the desktop app on the same day.

Factory defines a harness as “everything around the model: the prompts, the tools, how tool results come back, and the loop that runs it all”. It says Droid is “model-independent by design”: users can pick a model or let an automatic router, Factory Router, choose one.

What Factory says Droid scored

All of the figures below come from Factory’s own runs of Terminal-Bench 4, a test of coding agents working in a terminal, carried out between 23 and 28 October. TSN found no independent reproduction of them.

With OpenAI’s GPT-6.1 Sol, Factory says Droid passed 57.6% of tasks at $0.79 per passed task. It reports 54.0% for OpenAI’s Codex, 51.0% for OpenCode and 45.0% for Devin. Factory summarises that as “3.5 points higher than Codex at 12% lower cost”.

With Anthropic’s Claude Opus 5.5, Factory reports 65.2% for Droid at $4.73 per passed task, against 62.1% for Cursor, 62.1% for OpenCode, 59.6% for Devin and 57.6% for Anthropic’s own Claude Code. It says that is “3.0 to 7.6 points higher” at “up to 24% lower cost”. Factory also claims Droid took 26% less time per passed task than Claude Code on Opus 5.5, but 10% longer than Codex on GPT-6.1 Sol.

Factory’s methodology section sets out several adjustments. Each harness ran three trials per task, at high reasoning effort. Trials that failed for reasons Factory judged to be outside the harness were rerun. Where Anthropic’s content moderation stopped Opus 5.5, those trials were rerun on the older Claude Opus 5. Devin cannot switch off its tool for asking the user questions, so a task where it stopped to ask was counted as passed if Codex had passed it, on average 4.7 tasks per trial. Cost and time were measured only on tasks that both harnesses in a pair passed in all three trials.

Script mode

The biggest change, Factory says, is “programmatic tool calling”, now on by default. Droid gives the model a Script tool: the model writes JavaScript that runs in a sandbox “with no file, network, or system access of its own”, and that program calls Droid’s tools. Raw files and search results stay inside the program, and only what it prints goes back to the model.

Factory says one Script can make hundreds of tool calls in one model request. In its own worked example, fixing a failing unit test, the same 15 tool calls took five model calls instead of ten, and the task finished in 41.6 seconds instead of 56.8. It says Script mode cuts Droid’s cost by 15% on Claude Opus 5.5 compared with direct tool calls.

Factory does not claim to have invented the idea. Its post credits the 2024 CodeAct research paper, says Cloudflare and Anthropic “brought the idea to production in late 2025”, and says Codex brought it to coding agents this year.

Factory also says an internal system it calls the Agent Factory uses agents to study failed runs and propose harness changes, which are tested on each affected model before “our engineers review the evidence and decide what ships”. Droid 1.0 is available as a desktop app for macOS and Windows, as a terminal app and through a TypeScript software development kit, @factory/droid-sdk.

Harness versus model: the same Opus 5.5, scores from 57.6% to 66.4%

Factory’s central claim is that the harness, not just the model, decides how well a coding agent performs. Its own table supports that only within its own test setup. Run through five harnesses on the same benchmark, Claude Opus 5.5 scored between 57.6% and 65.2% in Factory’s runs, a spread of 7.6 points from changing the software around the model.

The gap gets wider across vendors. Moonshot’s launch material for Kimi K3.1 Pro put Claude Opus 5.5 at 66.4% on Terminal-Bench 4.0, and many of Kimi’s rival scores came from other companies’ reports or public leaderboards. That figure is above Factory’s best Opus score, in Droid. The two numbers are not like-for-like, because harness, settings and task counts differ. But it shows why a single Terminal-Bench figure means little without the harness and settings next to it.

Every comparison here is also a vendor testing itself. Factory chose the benchmark, the settings, the rerun rules and the Devin scoring adjustment, and it sells the winning product. The Particle-1 launch this week rested on the same kind of company-run efficiency claim. Others have also been releasing harness work, from Tencent Cloud’s TeamAI to Harness’s purchase of Augment Code’s coding agents.

The useful test will be a run by someone with no product in it. Factory has published the harness versions it tested and its methodology, which makes that possible. Until then, the 3.5-point lead over Codex is Factory’s number.