Microsoft Open-Sources Scope for AI Coding Agents

Microsoft has released Scope, an open-source platform for testing how well AI coding agents use a company’s software, documentation and tools. In a post on its Open Source blog on 19 October 2026, Todd Anglin, partner developer relations lead, described it as “an open-source platform built on Kubernetes for turning those interactions into evidence product teams can act on.”

Scope is available on GitHub under the MIT licence, where it is labelled a “Research preview”. It is software teams deploy themselves, not a hosted Microsoft service. The release reflects a shift in how developer tools are judged: increasingly, the question is not just whether a person can use an SDK, but whether an AI agent can find it, set it up and use it correctly.

In brief

Microsoft has open-sourced Scope, a Kubernetes-based platform that drives coding agents such as GitHub Copilot and Claude Code and judges their output against custom criteria.
It is MIT-licensed, self-managed and labelled a research preview on GitHub; Microsoft says it began building it more than a year ago.
Microsoft says its own Azure Cosmos DB team used Scope to fix 20 Agent Kit and 27 documentation issues, results it stresses are not a general ranking of agents.

How Scope works

According to Microsoft, “Scope drives existing coding agents, including GitHub Copilot, Claude Code, and Visual Studio Code with Copilot.” Teams define representative tasks, provide a starting codebase where needed, and choose the agent set-up. Configurable personas vary the simulated user’s perspective and communication style.

Reusable, versioned profiles record the configuration behind each run, including the model, agent version, skills and MCP servers. That lets teams keep the task fixed while changing one thing at a time. Evaluation criteria are expressed as “a directed acyclic graph (DAG)”, so a runtime check, for example, can depend on a successful build.

“The Judge service uses large language model (LLM) based evaluation to assess the output against those criteria,” Microsoft says. Scope also keeps the agent’s interactions and workspace snapshots, so teams can follow tool calls, inspect generated files and trace a failed requirement back to the steps before it. It offers a web portal, a command-line interface and a REST API for running benchmarks from scripts and pipelines.

The Cosmos DB case study

Microsoft’s main example comes from its own Azure Cosmos DB team, which compared bare models with its Agent Kit. “Using SCOPE, we compared bare models with our Agent Kit across eight end-to-end Azure Cosmos DB scenarios in five languages,” said Jay Gordon, senior programme manager. “The evaluations led to 20 Agent Kit issues and 27 documentation issues resolved.”

Measure after one Agent Kit update	Before	After
Hierarchical partition-key adoption	47%	100%
OFFSET/LIMIT usage	47%	0%

Source: Microsoft (Azure Cosmos DB team).

Microsoft is careful about what these numbers show. “The gains describe the Cosmos DB scenarios evaluated, rather than a general ranking of agents,” it says. In other words, they show how one team improved the way agents use one product, not which coding agent is best.

Limits Microsoft acknowledges

Microsoft is also clear that automated evaluation has weaknesses. “Clearly, repeatable configurations don’t guarantee identical agent behavior, and automated judges can make mistakes,” the post says. Teams still need representative tasks, repeated runs and human review of the evidence before making decisions.

Running Scope also needs some set-up. “Scope is software you deploy, not a hosted service included with the release,” Microsoft says. Teams choose the connected services, manage access to data and credentials, and need accounts with the relevant agent and model providers. A local stack can be started with Docker Compose.

Microsoft says it is open-sourcing Scope so that evaluation methods can be inspected and improved, and to avoid every team rebuilding the same machinery. It is inviting contributions of new agent integrations, reusable criteria and analysis improvements.

Our read

Scope tackles a real and growing problem for anyone shipping SDKs, command-line tools or MCP servers: agents now read your docs, and their mistakes are your support tickets. Opening the code is the right move for an evaluation tool, because results only matter if others can check them. But it is a research preview judged partly by LLMs, so its value will depend on teams pairing it with human review, as Microsoft itself advises.