How much of the evaluation could an outsider re-run?

For every evaluation reported in the archive, we asked two questions: is the data public (the prompts, tasks or environment), and is the judge public (exact-match or unit-test scoring, or an LLM grader that anyone can call)? An eval counts as replicable only if the answer to both is yes. Every label links to the page that supports it.

Key findings

About 1 in 10 safety eval reports can be re-run from outsideOf 2,124 safety-eval reports, 10% use public data with a public judge. Only 6% qualify if API judges are excluded. 79% rest on internal datasets, red-teaming, audits or third-party outcomes. Capability benchmarks are the reverse: 69% are replicable.

The bottleneck is data, not judgesOnly 11% of safety reports use public data. Where the data is public, the judge is almost always public too. The judge is still poorly documented, though: 28% of safety reports use an LLM grader that is never named, and 19% don't say how grading works at all.

The highest-stakes areas are the least replicable2% or fewer reports are replicable in harmful content, jailbreaks, prompt injection, CBRN, child safety, reward hacking and elections. Bias (BBQ, political even-handedness), cyber (ExploitBench, CyberGym, CVE-Bench) and honesty (MASK, AA-Omniscience, SimpleQA) do better.

Western labs are equally closed, at 7–8% eachAnthropic, OpenAI and Google all sit at 7–8% replicable for safety. The higher figures for GLM, Kimi and Qwen come from a few dozen reports, mostly Qwen3Guard classifier benchmarks and public cyber suites. They show that little safety work is reported, not that it is open.

It got worse over the yearBetween the two halves of the year, the replicable share fell from 13% to 8% for safety and from 78% to 63% for capability. For safety, closed-data reports rose from 75% to 81%. For capability, the drop came from three things: newer benchmarks graded by unnamed or lab-swapped judges (5% → 10% of reports), held-out or modified subsets (5% → 9%), and benchmarks whose release status isn't documented (1% → 6%).

“Public benchmark” often means a private variantSHADE-Arena, VCT (access-gated), ProtocolQA Open-Ended, Cybench and CTF subsets, LinuxArena and OpenAI's StrongREJECT variant all start from public sources, but they are run on unreleased subsets or modified versions. They make up 9% of safety reports.

These figures count each eval once per document and area, include all document types, and reflect labels as of October 2026. The interactive views below recompute them under your filters.

Explore

By lab

Each bar is the share of evaluation reports in that lab's documents, by tier. The figure on the right is the replicable share.

By area

First half vs second half

Replicable share of evaluation reports, Oct 2025–Mar 2026 compared with Apr–Oct 2026.

Who does the judging

How results are scored, across all reports in scope. The public judge models are listed where a document, or the benchmark's official grader, names one.

Most-reported evaluations

Replicable

Not replicable

By document

Replicable share for each document, with the number of evaluation reports. Sort by clicking a column.

Every evaluation

Method and caveats

We started from the evaluations already extracted for each document: the named safety evals in each of the 21 safety areas and the headline capability benchmarks. Claude agents then labelled every distinct eval against a fixed schema. They searched the source markdown for how the eval was built and graded, and used general knowledge for well-known public benchmarks. Safety evals were labelled separately for each lab, because labs often grade the same benchmark differently. Capability benchmarks were labelled once across all labs.

Replicable: the data is public and used as published, and scoring is programmatic or done by a named public LLM (open-weight or available by API).
Open data, closed judge: the data is public, but the grader is an internal classifier, an unnamed model, or human experts.
Partial data: the source is public, but the lab filtered, modified or extended it without releasing the result.
Closed data: internal datasets, red-teaming, behavioural audits, production-traffic analyses and third-party assessments reported only as outcomes.
Items that are not evaluations, such as “inherited from proxy model” or safeguard descriptions, are left out.

A “report” is one eval named in one document. An eval reused across ten cards counts ten times, which reflects what readers actually see. Switch to distinct evals to count each eval once. Some things are not checked: whether the harness, prompts, sampling settings and scaffolds are disclosed. API judges can also be deprecated. The strict figure counts only programmatic and open-weight judges. Read these numbers as an upper bound on how replicable the evals really are.