Standing invitation · open to everyone
The E3 reproduction program
Every one of the 149 records in this index is evidence level E2: an editor inspected the public artifact and it says what the record claims. None has been independently reproduced — which means every number in the index, from the 7-second flight search to the three-cent Pokémon run, is still only the author's own measurement. This page is the standing invitation to change that, and it is aimed at you whether you are the project's author, a skeptical reader, or someone the demo did not work for.
Why reproductions are the missing layer
Artifact verification answers one question: does the thing exist and does it match its description? It deliberately does not answer whether the metric inside the README survives contact with a second machine, a second operator, and a different week. The October 2026 ecosystem report named this as the corpus's biggest hole: no win rates over many seeds, no false-positive rates on labelled data, no third-party cost measurements. Those numbers do not appear because nobody has run the experiments — so this program exists to make running them count for something.
What counts as a reproduction
A reproduction is not a vibe check. To upgrade a record to E3, an editor-reproduced result needs four things:
1 · The run. You executed the project's public artifact yourself — clone, build, run — following its published setup on your own hardware and account.
2 · The conditions. Date, machine, OS, model version if the project exposes one, and every place you had to deviate from the documented setup. Gaps in the instructions are findings, not embarrassments.
3 · The evidence. Logs, screenshots, timing output, or a session recording — whatever the project produces that a stranger could compare against the record's claim.
4 · The verdict. Confirmed, partially confirmed (the mechanism works but the number did not), or contradicted. All three outcomes publish.
What you get — including for contradictions
Accepted reproductions carry a named credit with a link to whatever you choose to publish (a repository, a gist, a post), and the record itself upgrades to E3 with your conditions attached. A contradiction earns the same credit: the record gets corrected, downgraded, or removed, and the correction notes your evidence. An index that only ever hears good news is a marketing channel; the correction path is the part that makes this one worth trusting. Nothing changes hands in either direction — reproductions are unpaid, unranked, and unsellable.
The protocol, in six steps
1 · Pick a record tagged reproducible-repo from the index — the candidates below are deliberately high-traffic. 2 · Read the record first, especially its limitations; they are the claims to test hardest. 3 · Run the public setup exactly as documented, noting every gap. 4 · Capture evidence as you go, not from memory. 5 · Send it through the submission form and note that it is a reproduction — include your conditions and evidence links. 6 · An editor re-runs anything ambiguous before the record moves, so expect a round of questions, not an instant upgrade.
Six records we want reproduced first
Chosen for attention and for reproducibility: each is a public repository or cookbook with a concrete, checkable claim.
- Complete a StarCraft shareware missionclaim to check: Verified development-run outcome = 1 mission completed
- Pick Pokémon turns from live ROM stateclaim to check: Elite Four run model cost = 0.03 USD per run
- Batch a regulatory briefing into one callclaim to check: Batched-call cost ratio = 12.2 × cheaper
- Judge every new arXiv paper each morningclaim to check: Cost to judge 50 papers = 0.002 USD
- Screen Spain's official gazette every morningclaim to check: Full gazette analysis cost = 0.01 USD per daily gazette
- Grep for what code does, not what it's calledclaim to check: the documented mechanism runs as described
Reproduction ledger
No accepted reproductions yet. The ledger starts here: first accepted reproduction gets the E3 flag, a named credit on this page, and the record it belongs to becomes the strongest evidence in the index.