paperclip-runner evalsstatic evidence

paperclip-runner eval suite

Provider-reported list cost$11.011875
Estimated model cost$5.409070

Current result for every test and model/configuration combination. Each cell uses only the most recent attempt for that combination.

Test
claude-sonnet-5
live-acpx-0.13.1-claude-acp-0.70.0-sonnet-5 · acpx_runtime · 0.13.1
claude-sonnet-5
live-claude-managed-agents-2026-04-01-sonnet-5 · claude_managed_agents_api · remote profile redacted
global.anthropic.claude-sonnet-4-6
live-aws-agentcore-global-anthropic-claude-sonnet-4-6 · aws_agentcore_harness_api · remote profile redacted
gpt-5.4-mini
live-codex-pinned · codex_app_server · provider default
gpt-5.6-sol
live-acpx-0.13.1-codex-acp-1.6.2-control · acpx_runtime · 0.13.1
openrouter/anthropic/claude-sonnet-5
live-opencode-1.18.17-openrouter-claude-sonnet-5 · opencode_server · 1.18.17
openrouter/deepseek/deepseek-v4-flash-0731
live-opencode-1.18.17-openrouter-deepseek-v4-flash-0731 · opencode_server · 1.18.17
openrouter/google/gemini-3.8-flash
live-opencode-1.18.17-openrouter-gemini-3.8-flash · opencode_server · 1.18.17
openrouter/openai/gpt-6-astra
live-opencode-1.18.17-openrouter-gpt-6-astra · opencode_server · 1.18.17
openrouter/qwen/qwen3.8-max-0902
live-opencode-1.18.17-openrouter-qwen3.8-max-0902 · opencode_server · 1.18.17
openrouter/z-ai/glm-5.3
live-opencode-1.18.17-openrouter-glm-5.3 · opencode_server · 1.18.17
answer-status-questionINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
avoid-generic-api-requestINFRAPASSPASSINFRA—PASSPASSPASSPASSPASSPASS
block-taskINFRAPASSINFRAINFRAINFRAPASSPASSPASSPASSPASSPASS
comment-on-approvalINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
context-before-actionINFRAFAILPASSINFRAINFRAPASSPASSPASSPASSPASSPASS
control-workspace-serviceINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
create-child-taskINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
create-task-documentINFRAPASSINFRAINFRAINFRAPASSPASSPASSPASSPASSFAIL
decide-approvalINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
finish-taskINFRAPASSPASSINFRAINFRAFAILFAILPASSFAILFAILFAIL
get-approvalINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
get-approval-contextINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
get-company-agentINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
get-task-contextINFRAPASSPASSINFRAINFRAPASSPASSPASSPASSPASSPASS
get-workspace-runtimeINFRAPASSPASSINFRA—PASSPASSPASSPASSPASSPASS
lazy-core-no-searchINFRAPASSPASSINFRA—PASSPASSPASSPASSPASSPASS
lazy-discover-agentsINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
lazy-unauthorized-undisclosedINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
list-approvalsINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
list-company-agentsINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
read-document-revisionsINFRAPASSPASSINFRA—PASSPASSPASSPASSPASSPASS
read-task-documentINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
read-task-historyINFRAPASSPASSINFRA—PASSPASSPASSPASSPASSPASS
register-deliverableINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
request-approvalINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
request-human-confirmationINFRAPASSINFRAINFRAINFRAPASSPASSPASSPASSPASSPASS
request-task-reviewINFRAPASSPASSINFRA—PASSPASSPASSPASSPASSPASS
schedule-task-wakeINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
search-company-tasksINFRAPASSPASSINFRA—PASSPASSPASSPASSPASSPASS
set-task-dependenciesINFRAPASSPASSINFRA—PASSPASSPASSPASSPASSPASS
update-task-documentINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSPASS
workflow-context-document-progressINFRAPASSINFRAINFRAINFRAPASSPASSPASSPASSPASSPASS
workflow-governed-waitINFRAPASSPASSINFRAINFRAPASSPASSPASSPASSPASSPASS
workflow-orient-delegateINFRAPASSINFRAINFRA—PASSPASSPASSPASSPASSFAIL
workflow-workspace-recoveryINFRAPASSPASSINFRA—PASSPASSPASSPASSPASSPASS