Benchmarks - Page 5
Latest
Arena Raises $200M at $3.1bn Valuation to Rank AI Models
Arena, the public AI model leaderboard, raised $200m at a $3.1bn valuation and says it has passed $100m annualised revenue, up from $30m in January.
Scale AI’s DistressBench: Chatbots Spot Crises but Often Skip Help
Scale AI built and graded a benchmark of 718 clinician-written crisis conversations. Pooled across 25 models, in 35.3% of conversations where a model fully recognised the crisis, it still failed to route the user towards help. What the company research shows, how it was scored, and its limits.
Claude Agents Help Build a UV Sky Map as Studies Question Agent Scores
Anthropic says a team of Claude Science agents helped build the first complete UV map of the sky, about a third of it predicted rather than observed. A preprint finds agent benchmark scores can move because of the grader, not the agent, and Exa previews a search benchmark it designed and runs itself. What each one shows, and what it doesn't.
AI Agent Money, 8 October: Hone, Vesta and Catalyst Raise Funding; Crossmint Launches Agent Checkout
Three AI-agent start-ups announced funding on 8 October: Hone ($60M seed), Vesta ($30M, reported) and Catalyst ($30M seed, reported). Crossmint launched a toolkit that lets AI agents pay by card. What each company says, who reported what, and which figures are company claims.
AI Research Tools: TasteVal Measures “Research Taste”, Karotte Targets Reward Hacking and Zephon Fixes Data Order
A new benchmark, TasteVal, says the best AI model now reaches expert-level results on eight AI research tasks with 2.3 times less compute, though its authors list reasons that may overstate progress. Preference Model has open-sourced Karotte, a framework for training environments that are harder to game, and DatologyAI has released Zephon, a data loader that keeps experiments repeatable when the GPU count changes. What each claims, and who measured it.
AI Trails People on Visual Reasoning Test; Base Models Reason on Cue
Scale AI and Elorian's Humanity's Sixth Sense benchmark finds people score 93.1% on everyday visual questions while the best model, GPT-6 Astra, scores 53.6%, mostly because models misread the picture. A separate preprint, not yet peer reviewed, says base models can reason almost as well as RL-trained ones when their answer starts with the right word.
Grok 4.7 on VulcanBench Frontier v4: what secondary trade press says
Secondary CryptoBriefing coverage says xAI’s Grok 4.7 holds 1st–3rd on VulcanBench Frontier v4. Primary VulcanBench/xAI confirmation was not checked here.
Mistral Large 4 Preview: Artificial Analysis scores it 38
Third-party firm Artificial Analysis scored Mistral Large 4 Preview 38 on Intelligence Index v4.3.2. Company training and open-weight claims are labelled separately.
GitHub’s ReviewBench: A Shared Scorecard for AI Code Reviewers
GitHub has released ReviewBench, an open benchmark that scores AI code-review agents on 219 real public pull requests. Here is what it measures, and why the fine print matters.
Latest on TSN
Explore the TSN Atlas: Every topic we cover, mapped. Click a bubble to dive in.
