Runner protocol eval campaigns
Every recorded campaign, its cost, and its exact source. Public chat replays are linked below; full provider evidence remains in access-controlled workflow artifacts.
Like-for-like trends
Only identical cells, model configurations, and eval-suite SHAs are compared. A changed suite starts a separate series. Report refreshes never count as new measurements. Regressions distinguish model behavior from infrastructure failures.
358 cells · 11 configurations · 4 recorded runs · evals 34e1846c
Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.
3 cells · 3 configurations · 2 recorded runs · evals 6acc269e
Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.
1 cells · 1 configurations · 3 recorded runs · evals 6acc269e
Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.
All runs · 9
Estimated cost and provider list cost are alternatives, not additive. ≥ means some entries lack usage. Historical final-only cost excludes retry spending. Commit links expose the full SHA on hover.
| Run | Pass / fail | Change vs matching suite | Cost (USD) | Exact source |
|---|---|---|---|---|
gha-34074939112-1Sep 7, 2026, 2:18 AM UTCFull maintained suiteOpen Evalbook → | failed355/358 · 99.2%2 behavior · 1 infrastructure11 model configurations
| 3 regressions · 2 recoveriesVersus gha-34062394019-1Previously passing → failing
| ≥ $7.067692Estimated · 357/360 entriesProvider list: ≥ $14.002806All attempts, including retries | Paperclip: 856813baEvals: 34e1846cGitHub Actions ↗refs/heads/master |
gha-34062394019-1Sep 6, 2026, 10:09 PM UTC358 recorded cellsOpen Evalbook → | failed356/358 · 99.4%2 behavior · 0 infrastructure11 model configurations
| 2 regressions · 17 recoveriesVersus gha-34060573948-1Previously passing → failing
| $6.825017Estimated · 358/358 entriesProvider list: $13.504117Historical final attempts only | Paperclip: ea8ad2d7Evals: 34e1846cGitHub Actions ↗refs/heads/codex/live-eval-acpx-usage-sept6 |
gha-34060573948-1Sep 6, 2026, 9:31 PM UTC358 recorded cellsOpen Evalbook → | failed341/358 · 95.3%17 behavior · 0 infrastructure11 model configurations
| 11 regressions · 102 recoveriesVersus gha-34059009921-1Previously passing → failing
| $8.891645Estimated · 358/358 entriesProvider list: $15.782736Historical final attempts only | Paperclip: ec7b35b4Evals: 34e1846cGitHub Actions ↗refs/heads/codex/live-eval-acpx-usage-sept6 |
gha-34059009921-1Sep 6, 2026, 9:00 PM UTC358 recorded cellsOpen Evalbook → | failed250/358 · 69.8%8 behavior · 100 infrastructure11 model configurations
| No matching baseline | ≥ $5.409070Estimated · 258/358 entriesProvider list: ≥ $11.011875Historical final attempts only | Paperclip: b55ce03eEvals: 34e1846cGitHub Actions ↗refs/heads/master |
gha-34048154081-1Sep 6, 2026, 5:24 PM UTC1 recorded cellsOpen Evalbook → | failed0/1 · 0.0%1 behavior · 0 infrastructure1 model configurations
| 0 regressions · 0 recoveriesVersus gha-34047619729-1 | $0.000000Estimated · 1/1 entriesProvider list: $0.000000Historical final attempts only | Paperclip: c3adbe75Evals: 6acc269eGitHub Actions ↗refs/heads/codex/live-eval-bootstrap-smoke |
gha-34047619729-1Sep 6, 2026, 5:14 PM UTC1 recorded cellsOpen Evalbook → | failed0/1 · 0.0%1 behavior · 0 infrastructure1 model configurations
| 0 regressions · 0 recoveriesVersus gha-34046937609-1 | $0.000000Estimated · 1/1 entriesProvider list: $0.000000Historical final attempts only | Paperclip: 1e806740Evals: 6acc269eGitHub Actions ↗refs/heads/codex/live-eval-bootstrap-smoke |
gha-34046937609-1Sep 6, 2026, 5:01 PM UTC1 recorded cellsOpen Evalbook → | failed0/1 · 0.0%1 behavior · 0 infrastructure1 model configurations
| No matching baseline | $0.000000Estimated · 1/1 entriesProvider list: $0.000000Historical final attempts only | Paperclip: 4425f73cEvals: 6acc269eGitHub Actions ↗refs/heads/codex/live-eval-bootstrap-smoke |
gha-34046065010-1Sep 6, 2026, 4:44 PM UTC3 recorded cellsOpen Evalbook → | failed0/3 · 0.0%0 behavior · 3 infrastructure3 model configurations
| 0 regressions · 0 recoveriesVersus gha-34043987287-1 | UnknownEstimated · 0/3 entriesProvider list: UnknownHistorical final attempts only | Paperclip: 9421d653Evals: 6acc269eGitHub Actions ↗refs/heads/codex/live-eval-bootstrap-smoke |
gha-34043987287-1Sep 6, 2026, 4:06 PM UTC3 recorded cellsOpen Evalbook → | failed0/3 · 0.0%0 behavior · 3 infrastructure3 model configurations
| No matching baseline | UnknownEstimated · 0/3 entriesProvider list: UnknownHistorical final attempts only | Paperclip: 75289b06Evals: 6acc269eGitHub Actions ↗refs/heads/codex/live-eval-bootstrap-smoke |
Report refreshes · 2 (no new measurements)
| Run | Pass / fail | Change vs matching suite | Cost (USD) | Exact source |
|---|---|---|---|---|
gha-34074939112-1-report-history-v1Sep 7, 2026, 2:18 AM UTCReport refresh · no new model calls · source gha-34074939112-1Open Evalbook → | failed355/358 · 99.2%2 behavior · 1 infrastructure11 model configurations
| Presentation only | No additional model cost | Paperclip: 856813baEvals: 34e1846cGitHub Actions ↗refs/heads/master |
gha-34062394019-1-report-site-theme-v3Sep 6, 2026, 10:09 PM UTCReport refresh · no new model calls · source gha-34062394019-1Open Evalbook → | failed356/358 · 99.4%2 behavior · 0 infrastructure11 model configurations
| Presentation only | No additional model cost | Paperclip: ea8ad2d7Evals: 34e1846cGitHub Actions ↗refs/heads/codex/live-eval-acpx-usage-sept6 |