paperclip-runner evalsRun history

Runner protocol eval campaigns

Every recorded campaign, its cost, and its exact source. Public chat replays are linked below; full provider evidence remains in access-controlled workflow artifacts.

Like-for-like trends

Only identical cells, model configurations, and eval-suite SHAs are compared. A changed suite starts a separate series. Report refreshes never count as new measurements. Regressions distinguish model behavior from infrastructure failures.

398 cells · 11 configurations · 1 recorded runs · evals caf7de6c
Pass rate over time (%)
Pass rate over time (%). Each point links to its recorded run.100%0Sep 20, 2026, 11:44 AM UTC · gha-35508139279-1 · 9.8% (39/398)09-20 11:44 UTC09-20 11:44 UTC
Estimated cost over time (USD)
Estimated cost over time (USD). Each point links to its recorded run.$1.610Sep 20, 2026, 11:44 AM UTC · gha-35508139279-1 · ≥ $1.606181 (all_attempts)09-20 11:44 UTC09-20 11:44 UTC

Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.

358 cells · 11 configurations · 4 recorded runs · evals 34e1846c
Pass rate over time (%)
Pass rate over time (%). Each point links to its recorded run.100%0Sep 6, 2026, 9:00 PM UTC · gha-34059009921-1 · 69.8% (250/358)Sep 6, 2026, 9:31 PM UTC · gha-34060573948-1 · 95.3% (341/358)Sep 6, 2026, 10:09 PM UTC · gha-34062394019-1 · 99.4% (356/358)Sep 7, 2026, 2:18 AM UTC · gha-34074939112-1 · 99.2% (355/358)09-06 21:00 UTC09-07 02:18 UTC
Estimated cost over time (USD)
Estimated cost over time (USD). Each point links to its recorded run.$8.890Sep 6, 2026, 9:00 PM UTC · gha-34059009921-1 · ≥ $5.409070 (final_attempts)Sep 6, 2026, 9:31 PM UTC · gha-34060573948-1 · $8.891645 (final_attempts)Sep 6, 2026, 10:09 PM UTC · gha-34062394019-1 · $6.825017 (final_attempts)Sep 7, 2026, 2:18 AM UTC · gha-34074939112-1 · ≥ $7.067692 (all_attempts)09-06 21:00 UTC09-07 02:18 UTC

Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.

39 cells · 1 configurations · 1 recorded runs · evals 92bb4b8c
Pass rate over time (%)
Pass rate over time (%). Each point links to its recorded run.100%0Sep 24, 2026, 8:03 PM UTC · gha-36046839612-1 · 100.0% (39/39)09-24 20:03 UTC09-24 20:03 UTC
Estimated cost over time (USD)
Estimated cost over time (USD). Each point links to its recorded run.$0.00009-24 20:03 UTC09-24 20:03 UTC

Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.

39 cells · 1 configurations · 2 recorded runs · evals 3213dbec
Pass rate over time (%)
Pass rate over time (%). Each point links to its recorded run.100%0Sep 23, 2026, 10:49 PM UTC · gha-35926577007-1 · 100.0% (39/39)Sep 24, 2026, 4:57 PM UTC · gha-36025232556-1 · 100.0% (39/39)09-23 22:49 UTC09-24 16:57 UTC
Estimated cost over time (USD)
Estimated cost over time (USD). Each point links to its recorded run.$0.00009-23 22:49 UTC09-24 16:57 UTC

Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.

8 cells · 1 configurations · 1 recorded runs · evals 8747d0f9
Pass rate over time (%)
Pass rate over time (%). Each point links to its recorded run.100%0Sep 22, 2026, 12:58 AM UTC · gha-35673604555-1 · 100.0% (8/8)09-22 00:58 UTC09-22 00:58 UTC
Estimated cost over time (USD)
Estimated cost over time (USD). Each point links to its recorded run.$0.010Sep 22, 2026, 12:58 AM UTC · gha-35673604555-1 · $0.009546 (all_attempts)09-22 00:58 UTC09-22 00:58 UTC

Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.

8 cells · 1 configurations · 1 recorded runs · evals 2bedab7a
Pass rate over time (%)
Pass rate over time (%). Each point links to its recorded run.100%0Sep 22, 2026, 12:46 AM UTC · gha-35672841208-1 · 75.0% (6/8)09-22 00:46 UTC09-22 00:46 UTC
Estimated cost over time (USD)
Estimated cost over time (USD). Each point links to its recorded run.$0.010Sep 22, 2026, 12:46 AM UTC · gha-35672841208-1 · $0.010008 (all_attempts)09-22 00:46 UTC09-22 00:46 UTC

Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.

3 cells · 3 configurations · 2 recorded runs · evals 6acc269e
Pass rate over time (%)
Pass rate over time (%). Each point links to its recorded run.100%0Sep 6, 2026, 4:06 PM UTC · gha-34043987287-1 · 0.0% (0/3)Sep 6, 2026, 4:44 PM UTC · gha-34046065010-1 · 0.0% (0/3)09-06 16:06 UTC09-06 16:44 UTC
Estimated cost over time (USD)
Estimated cost over time (USD). Each point links to its recorded run.$0.00009-06 16:06 UTC09-06 16:44 UTC

Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.

1 cells · 1 configurations · 2 recorded runs · evals 7157eca7
Pass rate over time (%)
Pass rate over time (%). Each point links to its recorded run.100%0Nov 4, 2026, 3:49 PM UTC · gha-41860234840-1 · 0.0% (0/1)Nov 4, 2026, 4:31 PM UTC · gha-41863410602-1 · 0.0% (0/1)11-04 15:49 UTC11-04 16:31 UTC
Estimated cost over time (USD)
Estimated cost over time (USD). Each point links to its recorded run.$0.000Nov 4, 2026, 3:49 PM UTC · gha-41860234840-1 · $0.000000 (all_attempts)Nov 4, 2026, 4:31 PM UTC · gha-41863410602-1 · $0.000000 (all_attempts)11-04 15:49 UTC11-04 16:31 UTC

Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.

1 cells · 1 configurations · 3 recorded runs · evals 6acc269e
Pass rate over time (%)
Pass rate over time (%). Each point links to its recorded run.100%0Sep 6, 2026, 5:01 PM UTC · gha-34046937609-1 · 0.0% (0/1)Sep 6, 2026, 5:14 PM UTC · gha-34047619729-1 · 0.0% (0/1)Sep 6, 2026, 5:24 PM UTC · gha-34048154081-1 · 0.0% (0/1)09-06 17:01 UTC09-06 17:24 UTC
Estimated cost over time (USD)
Estimated cost over time (USD). Each point links to its recorded run.$0.000Sep 6, 2026, 5:01 PM UTC · gha-34046937609-1 · $0.000000 (final_attempts)Sep 6, 2026, 5:14 PM UTC · gha-34047619729-1 · $0.000000 (final_attempts)Sep 6, 2026, 5:24 PM UTC · gha-34048154081-1 · $0.000000 (final_attempts)09-06 17:01 UTC09-06 17:24 UTC

Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.

All runs · 17

Estimated cost and provider list cost are alternatives, not additive. ≥ means some entries lack usage. Historical final-only cost excludes retry spending. Commit links expose the full SHA on hover.

RunPass / failChange vs matching suiteCost (USD)Exact source
gha-41863410602-1Nov 4, 2026, 4:31 PM UTCSelected subsetOpen Evalbook →failed0/1 · 0.0%1 behavior · 0 infrastructure
1 model configurations
  • claude-sonnet-5 · 0/1acpx_runtime · protocol-live-cost-qualification-claude
0 regressions · 0 recoveriesVersus gha-41860234840-1
$0.000000Estimated · 1/1 entriesProvider list: $0.000000All attempts, including retriesPaperclip: b850637fEvals: 7157eca7GitHub Actions ↗refs/heads/codex/testops-runner-diagnostic-b850637f-20261104
gha-41860234840-1Nov 4, 2026, 3:49 PM UTCSelected subsetOpen Evalbook →failed0/1 · 0.0%1 behavior · 0 infrastructure
1 model configurations
  • claude-sonnet-5 · 0/1acpx_runtime · protocol-live-cost-qualification-claude
No matching baseline$0.000000Estimated · 1/1 entriesProvider list: $0.000000All attempts, including retriesPaperclip: 86814c65Evals: 7157eca7GitHub Actions ↗refs/heads/master
gha-36046839612-1Sep 24, 2026, 8:03 PM UTCSelected subsetOpen Evalbook →passed39/39 · 100.0%0 behavior · 0 infrastructure
1 model configurations
  • grok-4.7 · 39/39acpx_runtime · protocol-live-acpx-grok
No matching baselineUnknownEstimated · 0/39 entriesProvider list: UnknownAll attempts, including retriesPaperclip: bc6833f7Evals: 92bb4b8cGitHub Actions ↗refs/heads/codex/grok-native-runner
gha-36025232556-1Sep 24, 2026, 4:57 PM UTCSelected subsetOpen Evalbook →passed39/39 · 100.0%0 behavior · 0 infrastructure
1 model configurations
  • grok-4.7 · 39/39acpx_runtime · protocol-live-acpx-grok
0 regressions · 0 recoveriesVersus gha-35926577007-1
UnknownEstimated · 0/39 entriesProvider list: UnknownAll attempts, including retriesPaperclip: a8f10908Evals: 3213dbecGitHub Actions ↗refs/heads/codex/grok-native-runner
gha-35926577007-1Sep 23, 2026, 10:49 PM UTCSelected subsetOpen Evalbook →passed39/39 · 100.0%0 behavior · 0 infrastructure
1 model configurations
  • grok-4.7 · 39/39acpx_runtime · protocol-live-acpx-grok
No matching baselineUnknownEstimated · 0/39 entriesProvider list: UnknownAll attempts, including retriesPaperclip: 4a1061c8Evals: 3213dbecGitHub Actions ↗refs/heads/codex/grok-native-runner
gha-35673604555-1Sep 22, 2026, 12:58 AM UTCSelected subsetOpen Evalbook →passed8/8 · 100.0%0 behavior · 0 infrastructure
1 model configurations
  • gpt-5.4-mini · 8/8codex_app_server · lifecycle-narrative-baseline
No matching baseline$0.009546Estimated · 8/8 entriesProvider list: $0.000000All attempts, including retriesPaperclip: 91b62ce1Evals: 8747d0f9GitHub Actions ↗refs/heads/codex/lifecycle-behavior-baseline-20260921
gha-35672841208-1Sep 22, 2026, 12:46 AM UTCSelected subsetOpen Evalbook →failed6/8 · 75.0%2 behavior · 0 infrastructure
1 model configurations
  • gpt-5.4-mini · 6/8codex_app_server · lifecycle-narrative-baseline
No matching baseline$0.010008Estimated · 8/8 entriesProvider list: $0.000000All attempts, including retriesPaperclip: 91b62ce1Evals: 2bedab7aGitHub Actions ↗refs/heads/codex/lifecycle-behavior-baseline-20260921
gha-35508139279-1Sep 20, 2026, 11:44 AM UTCFull maintained suiteOpen Evalbook →failed39/398 · 9.8%39 behavior · 320 infrastructure
11 model configurations
  • claude-sonnet-5 · 0/39acpx_runtime · protocol-live-acpx-claude
  • gpt-5.6-sol · 0/8acpx_runtime · protocol-live-acpx-codex-control
  • global.anthropic.claude-sonnet-4-6 · 0/39aws_agentcore_harness_api · protocol-live-aws-agentcore
  • claude-sonnet-5 · 39/39claude_managed_agents_api · protocol-live-claude-managed
  • gpt-5.4-mini · 0/39codex_app_server · protocol-live-mini
  • openrouter/anthropic/claude-sonnet-5 · 0/39opencode_server · protocol-live-opencode-openrouter-claude-sonnet-5
  • openrouter/google/gemini-3.8-flash · 0/39opencode_server · protocol-live-opencode-openrouter-gemini-3-8-flash
  • openrouter/z-ai/glm-5.3 · 0/39opencode_server · protocol-live-opencode-openrouter-glm-5-3
  • openrouter/openai/gpt-6-astra · 0/39opencode_server · protocol-live-opencode-openrouter-gpt-6-astra
  • openrouter/qwen/qwen3.8-max-0902 · 0/39opencode_server · protocol-live-opencode-openrouter-qwen3-8-max-0902
  • openrouter/deepseek/deepseek-v4-flash-0731 · 0/39opencode_server · protocol-live-opencode-openrouter
No matching baseline≥ $1.606181Estimated · 125/437 entriesProvider list: ≥ $3.278414All attempts, including retriesPaperclip: aedb8c6dEvals: caf7de6cGitHub Actions ↗refs/heads/codex/runner-stress-audit-20260916
gha-34074939112-1Sep 7, 2026, 2:18 AM UTCFull maintained suiteOpen Evalbook →failed355/358 · 99.2%2 behavior · 1 infrastructure
11 model configurations
  • claude-sonnet-5 · 35/35acpx_runtime · protocol-live-acpx-claude
  • gpt-5.6-sol · 8/8acpx_runtime · protocol-live-acpx-codex-control
  • global.anthropic.claude-sonnet-4-6 · 35/35aws_agentcore_harness_api · protocol-live-aws-agentcore
  • claude-sonnet-5 · 35/35claude_managed_agents_api · protocol-live-claude-managed
  • gpt-5.4-mini · 34/35codex_app_server · protocol-live-mini
  • openrouter/anthropic/claude-sonnet-5 · 35/35opencode_server · protocol-live-opencode-openrouter-claude-sonnet-5
  • openrouter/google/gemini-3.8-flash · 35/35opencode_server · protocol-live-opencode-openrouter-gemini-3-8-flash
  • openrouter/z-ai/glm-5.3 · 33/35opencode_server · protocol-live-opencode-openrouter-glm-5-3
  • openrouter/openai/gpt-6-astra · 35/35opencode_server · protocol-live-opencode-openrouter-gpt-6-astra
  • openrouter/qwen/qwen3.8-max-0902 · 35/35opencode_server · protocol-live-opencode-openrouter-qwen3-8-max-0902
  • openrouter/deepseek/deepseek-v4-flash-0731 · 35/35opencode_server · protocol-live-opencode-openrouter
3 regressions · 2 recoveriesVersus gha-34062394019-1Previously passing → failingPreviously failing → passing
≥ $7.067692Estimated · 357/360 entriesProvider list: ≥ $14.002806All attempts, including retriesPaperclip: 856813baEvals: 34e1846cGitHub Actions ↗refs/heads/master
gha-34062394019-1Sep 6, 2026, 10:09 PM UTC358 recorded cellsOpen Evalbook →failed356/358 · 99.4%2 behavior · 0 infrastructure
11 model configurations
  • claude-sonnet-5 · 35/35acpx_runtime · protocol-live-acpx-claude
  • gpt-5.6-sol · 7/8acpx_runtime · protocol-live-acpx-codex-control
  • global.anthropic.claude-sonnet-4-6 · 35/35aws_agentcore_harness_api · protocol-live-aws-agentcore
  • claude-sonnet-5 · 34/35claude_managed_agents_api · protocol-live-claude-managed
  • gpt-5.4-mini · 35/35codex_app_server · protocol-live-mini
  • openrouter/anthropic/claude-sonnet-5 · 35/35opencode_server · protocol-live-opencode-openrouter-claude-sonnet-5
  • openrouter/google/gemini-3.8-flash · 35/35opencode_server · protocol-live-opencode-openrouter-gemini-3-8-flash
  • openrouter/z-ai/glm-5.3 · 35/35opencode_server · protocol-live-opencode-openrouter-glm-5-3
  • openrouter/openai/gpt-6-astra · 35/35opencode_server · protocol-live-opencode-openrouter-gpt-6-astra
  • openrouter/qwen/qwen3.8-max-0902 · 35/35opencode_server · protocol-live-opencode-openrouter-qwen3-8-max-0902
  • openrouter/deepseek/deepseek-v4-flash-0731 · 35/35opencode_server · protocol-live-opencode-openrouter
2 regressions · 17 recoveriesVersus gha-34060573948-1Previously passing → failingPreviously failing → passing
$6.825017Estimated · 358/358 entriesProvider list: $13.504117Historical final attempts onlyPaperclip: ea8ad2d7Evals: 34e1846cGitHub Actions ↗refs/heads/codex/live-eval-acpx-usage-sept6
gha-34060573948-1Sep 6, 2026, 9:31 PM UTC358 recorded cellsOpen Evalbook →failed341/358 · 95.3%17 behavior · 0 infrastructure
11 model configurations
  • claude-sonnet-5 · 34/35acpx_runtime · protocol-live-acpx-claude
  • gpt-5.6-sol · 7/8acpx_runtime · protocol-live-acpx-codex-control
  • global.anthropic.claude-sonnet-4-6 · 35/35aws_agentcore_harness_api · protocol-live-aws-agentcore
  • claude-sonnet-5 · 33/35claude_managed_agents_api · protocol-live-claude-managed
  • gpt-5.4-mini · 34/35codex_app_server · protocol-live-mini
  • openrouter/anthropic/claude-sonnet-5 · 32/35opencode_server · protocol-live-opencode-openrouter-claude-sonnet-5
  • openrouter/google/gemini-3.8-flash · 35/35opencode_server · protocol-live-opencode-openrouter-gemini-3-8-flash
  • openrouter/z-ai/glm-5.3 · 31/35opencode_server · protocol-live-opencode-openrouter-glm-5-3
  • openrouter/openai/gpt-6-astra · 34/35opencode_server · protocol-live-opencode-openrouter-gpt-6-astra
  • openrouter/qwen/qwen3.8-max-0902 · 33/35opencode_server · protocol-live-opencode-openrouter-qwen3-8-max-0902
  • openrouter/deepseek/deepseek-v4-flash-0731 · 33/35opencode_server · protocol-live-opencode-openrouter
11 regressions · 102 recoveriesVersus gha-34059009921-1Previously passing → failingPreviously failing → passing
$8.891645Estimated · 358/358 entriesProvider list: $15.782736Historical final attempts onlyPaperclip: ec7b35b4Evals: 34e1846cGitHub Actions ↗refs/heads/codex/live-eval-acpx-usage-sept6
gha-34059009921-1Sep 6, 2026, 9:00 PM UTC358 recorded cellsOpen Evalbook →failed250/358 · 69.8%8 behavior · 100 infrastructure
11 model configurations
  • claude-sonnet-5 · 0/35acpx_runtime · protocol-live-acpx-claude
  • gpt-5.6-sol · 0/8acpx_runtime · protocol-live-acpx-codex-control
  • global.anthropic.claude-sonnet-4-6 · 13/35aws_agentcore_harness_api · protocol-live-aws-agentcore
  • claude-sonnet-5 · 34/35claude_managed_agents_api · protocol-live-claude-managed
  • gpt-5.4-mini · 0/35codex_app_server · protocol-live-mini
  • openrouter/anthropic/claude-sonnet-5 · 34/35opencode_server · protocol-live-opencode-openrouter-claude-sonnet-5
  • openrouter/google/gemini-3.8-flash · 35/35opencode_server · protocol-live-opencode-openrouter-gemini-3-8-flash
  • openrouter/z-ai/glm-5.3 · 32/35opencode_server · protocol-live-opencode-openrouter-glm-5-3
  • openrouter/openai/gpt-6-astra · 34/35opencode_server · protocol-live-opencode-openrouter-gpt-6-astra
  • openrouter/qwen/qwen3.8-max-0902 · 34/35opencode_server · protocol-live-opencode-openrouter-qwen3-8-max-0902
  • openrouter/deepseek/deepseek-v4-flash-0731 · 34/35opencode_server · protocol-live-opencode-openrouter
No matching baseline≥ $5.409070Estimated · 258/358 entriesProvider list: ≥ $11.011875Historical final attempts onlyPaperclip: b55ce03eEvals: 34e1846cGitHub Actions ↗refs/heads/master
gha-34048154081-1Sep 6, 2026, 5:24 PM UTC1 recorded cellsOpen Evalbook →failed0/1 · 0.0%1 behavior · 0 infrastructure
1 model configurations
  • global.anthropic.claude-sonnet-4-6 · 0/1aws_agentcore_harness_api · protocol-live-aws-agentcore
0 regressions · 0 recoveriesVersus gha-34047619729-1
$0.000000Estimated · 1/1 entriesProvider list: $0.000000Historical final attempts onlyPaperclip: c3adbe75Evals: 6acc269eGitHub Actions ↗refs/heads/codex/live-eval-bootstrap-smoke
gha-34047619729-1Sep 6, 2026, 5:14 PM UTC1 recorded cellsOpen Evalbook →failed0/1 · 0.0%1 behavior · 0 infrastructure
1 model configurations
  • global.anthropic.claude-sonnet-4-6 · 0/1aws_agentcore_harness_api · protocol-live-aws-agentcore
0 regressions · 0 recoveriesVersus gha-34046937609-1
$0.000000Estimated · 1/1 entriesProvider list: $0.000000Historical final attempts onlyPaperclip: 1e806740Evals: 6acc269eGitHub Actions ↗refs/heads/codex/live-eval-bootstrap-smoke
gha-34046937609-1Sep 6, 2026, 5:01 PM UTC1 recorded cellsOpen Evalbook →failed0/1 · 0.0%1 behavior · 0 infrastructure
1 model configurations
  • global.anthropic.claude-sonnet-4-6 · 0/1aws_agentcore_harness_api · protocol-live-aws-agentcore
No matching baseline$0.000000Estimated · 1/1 entriesProvider list: $0.000000Historical final attempts onlyPaperclip: 4425f73cEvals: 6acc269eGitHub Actions ↗refs/heads/codex/live-eval-bootstrap-smoke
gha-34046065010-1Sep 6, 2026, 4:44 PM UTC3 recorded cellsOpen Evalbook →failed0/3 · 0.0%0 behavior · 3 infrastructure
3 model configurations
  • claude-sonnet-5 · 0/1acpx_runtime · protocol-live-acpx-claude
  • gpt-5.6-sol · 0/1acpx_runtime · protocol-live-acpx-codex-control
  • global.anthropic.claude-sonnet-4-6 · 0/1aws_agentcore_harness_api · protocol-live-aws-agentcore
0 regressions · 0 recoveriesVersus gha-34043987287-1
UnknownEstimated · 0/3 entriesProvider list: UnknownHistorical final attempts onlyPaperclip: 9421d653Evals: 6acc269eGitHub Actions ↗refs/heads/codex/live-eval-bootstrap-smoke
gha-34043987287-1Sep 6, 2026, 4:06 PM UTC3 recorded cellsOpen Evalbook →failed0/3 · 0.0%0 behavior · 3 infrastructure
3 model configurations
  • claude-sonnet-5 · 0/1acpx_runtime · protocol-live-acpx-claude
  • gpt-5.6-sol · 0/1acpx_runtime · protocol-live-acpx-codex-control
  • global.anthropic.claude-sonnet-4-6 · 0/1aws_agentcore_harness_api · protocol-live-aws-agentcore
No matching baselineUnknownEstimated · 0/3 entriesProvider list: UnknownHistorical final attempts onlyPaperclip: 75289b06Evals: 6acc269eGitHub Actions ↗refs/heads/codex/live-eval-bootstrap-smoke
Report refreshes · 2 (no new measurements)
RunPass / failChange vs matching suiteCost (USD)Exact source
gha-34074939112-1-report-history-v1Sep 7, 2026, 2:18 AM UTCReport refresh · no new model calls · source gha-34074939112-1Open Evalbook →failed355/358 · 99.2%2 behavior · 1 infrastructure
11 model configurations
  • claude-sonnet-5 · 35/35acpx_runtime · protocol-live-acpx-claude
  • gpt-5.6-sol · 8/8acpx_runtime · protocol-live-acpx-codex-control
  • global.anthropic.claude-sonnet-4-6 · 35/35aws_agentcore_harness_api · protocol-live-aws-agentcore
  • claude-sonnet-5 · 35/35claude_managed_agents_api · protocol-live-claude-managed
  • gpt-5.4-mini · 34/35codex_app_server · protocol-live-mini
  • openrouter/anthropic/claude-sonnet-5 · 35/35opencode_server · protocol-live-opencode-openrouter-claude-sonnet-5
  • openrouter/google/gemini-3.8-flash · 35/35opencode_server · protocol-live-opencode-openrouter-gemini-3-8-flash
  • openrouter/z-ai/glm-5.3 · 33/35opencode_server · protocol-live-opencode-openrouter-glm-5-3
  • openrouter/openai/gpt-6-astra · 35/35opencode_server · protocol-live-opencode-openrouter-gpt-6-astra
  • openrouter/qwen/qwen3.8-max-0902 · 35/35opencode_server · protocol-live-opencode-openrouter-qwen3-8-max-0902
  • openrouter/deepseek/deepseek-v4-flash-0731 · 35/35opencode_server · protocol-live-opencode-openrouter
Presentation onlyNo additional model costPaperclip: 856813baEvals: 34e1846cGitHub Actions ↗refs/heads/master
gha-34062394019-1-report-site-theme-v3Sep 6, 2026, 10:09 PM UTCReport refresh · no new model calls · source gha-34062394019-1Open Evalbook →failed356/358 · 99.4%2 behavior · 0 infrastructure
11 model configurations
  • claude-sonnet-5 · 35/35acpx_runtime · protocol-live-acpx-claude
  • gpt-5.6-sol · 7/8acpx_runtime · protocol-live-acpx-codex-control
  • global.anthropic.claude-sonnet-4-6 · 35/35aws_agentcore_harness_api · protocol-live-aws-agentcore
  • claude-sonnet-5 · 34/35claude_managed_agents_api · protocol-live-claude-managed
  • gpt-5.4-mini · 35/35codex_app_server · protocol-live-mini
  • openrouter/anthropic/claude-sonnet-5 · 35/35opencode_server · protocol-live-opencode-openrouter-claude-sonnet-5
  • openrouter/google/gemini-3.8-flash · 35/35opencode_server · protocol-live-opencode-openrouter-gemini-3-8-flash
  • openrouter/z-ai/glm-5.3 · 35/35opencode_server · protocol-live-opencode-openrouter-glm-5-3
  • openrouter/openai/gpt-6-astra · 35/35opencode_server · protocol-live-opencode-openrouter-gpt-6-astra
  • openrouter/qwen/qwen3.8-max-0902 · 35/35opencode_server · protocol-live-opencode-openrouter-qwen3-8-max-0902
  • openrouter/deepseek/deepseek-v4-flash-0731 · 35/35opencode_server · protocol-live-opencode-openrouter
Presentation onlyNo additional model costPaperclip: ea8ad2d7Evals: 34e1846cGitHub Actions ↗refs/heads/codex/live-eval-acpx-usage-sept6