paperclip-runner eval suite

Provider-reported list cost$15.782736

Estimated model cost$8.891645

Current result for every test and model/configuration combination. Each cell uses only the most recent attempt for that combination.

Test	claude-sonnet-5 live-acpx-0.13.1-claude-acp-0.70.0-sonnet-5 · acpx_runtime · 0.13.1	claude-sonnet-5 live-claude-managed-agents-2026-04-01-sonnet-5 · claude_managed_agents_api · remote profile redacted	global.anthropic.claude-sonnet-4-6 live-aws-agentcore-global-anthropic-claude-sonnet-4-6 · aws_agentcore_harness_api · remote profile redacted	gpt-5.4-mini live-codex-pinned · codex_app_server · provider default	gpt-5.6-sol live-acpx-0.13.1-codex-acp-1.6.2-control · acpx_runtime · 0.13.1	openrouter/anthropic/claude-sonnet-5 live-opencode-1.18.17-openrouter-claude-sonnet-5 · opencode_server · 1.18.17	openrouter/deepseek/deepseek-v4-flash-0731 live-opencode-1.18.17-openrouter-deepseek-v4-flash-0731 · opencode_server · 1.18.17	openrouter/google/gemini-3.8-flash live-opencode-1.18.17-openrouter-gemini-3.8-flash · opencode_server · 1.18.17	openrouter/openai/gpt-6-astra live-opencode-1.18.17-openrouter-gpt-6-astra · opencode_server · 1.18.17	openrouter/qwen/qwen3.8-max-0902 live-opencode-1.18.17-openrouter-qwen3.8-max-0902 · opencode_server · 1.18.17	openrouter/z-ai/glm-5.3 live-opencode-1.18.17-openrouter-glm-5.3 · opencode_server · 1.18.17
answer-status-question	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
avoid-generic-api-request	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
block-task	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS
comment-on-approval	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
context-before-action	FAIL	FAIL	PASS	PASS	PASS	FAIL	FAIL	PASS	FAIL	FAIL	FAIL
control-workspace-service	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
create-child-task	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	FAIL
create-task-document	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	FAIL
decide-approval	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
finish-task	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS
get-approval	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
get-approval-context	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
get-company-agent	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
get-task-context	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS
get-workspace-runtime	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
lazy-core-no-search	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
lazy-discover-agents	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
lazy-unauthorized-undisclosed	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
list-approvals	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
list-company-agents	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
read-document-revisions	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
read-task-document	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
read-task-history	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
register-deliverable	PASS	FAIL	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
request-approval	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
request-human-confirmation	PASS	PASS	PASS	PASS	PASS	FAIL	PASS	PASS	PASS	PASS	PASS
request-task-review	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	FAIL	PASS
schedule-task-wake	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
search-company-tasks	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
set-task-dependencies	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
update-task-document	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS
workflow-context-document-progress	PASS	PASS	PASS	FAIL	FAIL	FAIL	FAIL	PASS	PASS	PASS	PASS
workflow-governed-wait	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS
workflow-orient-delegate	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	FAIL
workflow-workspace-recovery	PASS	PASS	PASS	PASS	—	PASS	PASS	PASS	PASS	PASS	PASS