paperclip-runner evalsRecorded evaluation

paperclip-runner eval suite

Provider-reported list cost (all attempts)$3.278414 (partial: 125/437 attempts)
Estimated model cost (all attempts)$1.606181 (partial: 125/437 attempts)

Current result for each included test and model/configuration combination. Each cell uses only the most recent attempt for that combination.

Test
claude-sonnet-5
live-acpx-0.13.1-claude-acp-0.70.0-sonnet-5 · acpx_runtime · 0.13.1
claude-sonnet-5
live-claude-managed-agents-2026-04-01-sonnet-5 · claude_managed_agents_api · remote profile redacted
global.anthropic.claude-sonnet-4-6
live-aws-agentcore-global-anthropic-claude-sonnet-4-6 · aws_agentcore_harness_api · remote profile redacted
gpt-5.4-mini
live-codex-pinned · codex_app_server · provider default
gpt-5.6-sol
live-acpx-0.13.1-codex-acp-1.6.2-control · acpx_runtime · 0.13.1
openrouter/anthropic/claude-sonnet-5
live-opencode-1.18.17-openrouter-claude-sonnet-5 · opencode_server · 1.18.17
openrouter/deepseek/deepseek-v4-flash-0731
live-opencode-1.18.17-openrouter-deepseek-v4-flash-0731 · opencode_server · 1.18.17
openrouter/google/gemini-3.8-flash
live-opencode-1.18.17-openrouter-gemini-3.8-flash · opencode_server · 1.18.17
openrouter/openai/gpt-6-astra
live-opencode-1.18.17-openrouter-gpt-6-astra · opencode_server · 1.18.17
openrouter/qwen/qwen3.8-max-0902
live-opencode-1.18.17-openrouter-qwen3.8-max-0902 · opencode_server · 1.18.17
openrouter/z-ai/glm-5.3
live-opencode-1.18.17-openrouter-glm-5.3 · opencode_server · 1.18.17
answer-status-questionINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
avoid-generic-api-requestINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
block-taskINFRAPASSFAILINFRAINFRAINFRAINFRAINFRAINFRAINFRAINFRA
comment-on-approvalINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
context-before-actionINFRAPASSFAILINFRAINFRAINFRAINFRAINFRAINFRAINFRAINFRA
control-workspace-serviceINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
create-child-taskINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
create-skillINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
create-task-documentINFRAPASSFAILINFRAINFRAINFRAINFRAINFRAINFRAINFRAINFRA
decide-approvalINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
finish-taskINFRAPASSFAILINFRAINFRAINFRAINFRAINFRAINFRAINFRAINFRA
get-approvalINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
get-approval-contextINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
get-company-agentINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
get-task-contextINFRAPASSFAILINFRAINFRAINFRAINFRAINFRAINFRAINFRAINFRA
get-workspace-runtimeINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
lazy-core-no-searchINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
lazy-discover-agentsINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
lazy-unauthorized-undisclosedINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
list-approvalsINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
list-company-agentsINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
read-document-revisionsINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
read-task-documentINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
read-task-historyINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
reassign-blocked-taskINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
reassign-existing-taskINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
reassign-without-permissionINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
register-deliverableINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
request-approvalINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
request-human-confirmationINFRAPASSFAILINFRAINFRAINFRAINFRAINFRAINFRAINFRAINFRA
request-task-reviewINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
schedule-task-wakeINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
search-company-tasksINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
set-task-dependenciesINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
update-task-documentINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
workflow-context-document-progressINFRAPASSFAILINFRAINFRAINFRAINFRAINFRAINFRAINFRAINFRA
workflow-governed-waitINFRAPASSFAILINFRAINFRAINFRAINFRAINFRAINFRAINFRAINFRA
workflow-orient-delegateINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA
workflow-workspace-recoveryINFRAPASSFAILINFRA—INFRAINFRAINFRAINFRAINFRAINFRA