paperclip-runner evalsRecorded evaluation

Protocol coverage inventory

27/27 live product operations have an authored live case. Scenario-only operations are owned by deterministic conformance and the test-only escape hatch never counts as product coverage.

AreaOperationSurfaceProduct coverageLive casesStatus
Active task context and communicationget_task_contextscenario, liveyesget-task-context, context-before-actionImplemented
Active task context and communicationget_task_historyscenario, liveyesread-task-historyImplemented
Active task context and communicationreport_progressscenario, liveyescontext-before-action, native-codex-resumeImplemented
Active task context and communicationanswer_status_questionscenario, liveyesanswer-status-questionImplemented
Active task context and communicationfinish_taskscenario, liveyesfinish-taskImplemented
Active task context and communicationblock_taskscenario, liveyesblock-taskImplemented
Active task context and communicationrequest_reviewscenario, liveyesrequest-task-reviewImplemented
Documents, interactions, and deliverableslist_documentsscenario, liveyesread-task-documentImplemented
Documents, interactions, and deliverablesread_documentscenario, liveyesread-task-document, update-task-documentImplemented
Documents, interactions, and deliverableslist_document_revisionsscenario, liveyesread-document-revisionsImplemented
Documents, interactions, and deliverableswrite_documentscenario, liveyescreate-task-document, update-task-documentImplemented
Documents, interactions, and deliverablesrequest_human_inputscenario, liveyesrequest-human-confirmationImplemented
Documents, interactions, and deliverablesregister_deliverablescenario, liveyesregister-deliverableImplemented
Documents, interactions, and deliverablesinspect_operation_resultscenarioyesImplemented
Company discoverysearch_tasksscenario, liveyessearch-company-tasksImplemented
Company discoverylist_agentsscenario, liveyeslist-company-agentsImplemented
Company discoveryget_agentliveyesget-company-agentImplemented
Company discoverylist_projectsscenarioyesImplemented
Company discoverylist_goalsscenarioyesImplemented
Delegation and dependenciescreate_taskscenario, liveyescreate-child-taskImplemented
Delegation and dependenciesset_dependenciesscenario, liveyesset-task-dependenciesImplemented
Approvals and governancelist_approvalsscenario, liveyeslist-approvalsImplemented
Approvals and governanceget_approvalliveyesget-approvalImplemented
Approvals and governanceget_approval_contextliveyesget-approval-contextImplemented
Approvals and governancerequest_approvalscenario, liveyesrequest-approvalImplemented
Approvals and governancedecide_approvalscenario, liveyesdecide-approvalImplemented
Approvals and governancecomment_on_approvalscenario, liveyescomment-on-approvalImplemented
Workspace runtime and continuationget_workspace_runtimescenario, liveyesget-workspace-runtimeImplemented
Workspace runtime and continuationcontrol_workspace_servicescenario, liveyescontrol-workspace-serviceImplemented
Workspace runtime and continuationschedule_wakeliveyesschedule-task-wakeImplemented
Scenario-only protocol extensionslist_casesscenarioyesImplemented
Scenario-only protocol extensionsupsert_casescenarioyesImplemented
Scenario-only protocol extensionslist_routinesscenarioyesImplemented
Scenario-only protocol extensionsmanage_routinescenarioyesImplemented
Scenario-only protocol extensionslist_company_skillsscenarioyesImplemented
Scenario-only protocol extensionssync_company_skillsscenarioyesImplemented
Scenario-only protocol extensionslist_secret_metadatascenarioyesImplemented
Scenario-only protocol extensionsread_secret_valuescenarioyesImplemented
Scenario-only protocol extensionsexport_companyscenarioyesImplemented
Scenario-only protocol extensionsadminister_companyscenarioyesImplemented
Test-only escape hatchgeneric_api_requestscenario, livenoavoid-generic-api-requestImplemented

Workflow and restraint coverage

RequirementLayerCasesStatus
orientation-before-actionlive_behaviorcontext-before-actionimplemented
document-discovery-and-revisionlive_behaviorread-task-document, update-task-documentimplemented
bounded-delegationworkflowoperational/bounded-delegation, create-child-taskimplemented
blocker-and-continuation-wakeworkflowoperational/blocker-wake, set-task-dependencies, schedule-task-wakeimplemented
approval-request-decision-continuationworkflowoperational/approval-continuation, request-approval, decide-approvalimplemented
escalation-and-reviewworkflowoperational/escalation-review, request-task-reviewimplemented
documents-artifacts-and-receiptsworkflowoperational/documents-artifacts-receipts, register-deliverableimplemented
semantic-receipts-and-contextdeterministic_conformanceoperational/semantic-receipts-contextimplemented
budget-conflictsrestraintoperational/budget-conflictsimplemented
recovery-and-restraintrestraintoperational/recovery-restraintimplemented
secret-safetyrestraintoperational/secret-safety-focusimplemented
status-only-wake-restraintrestraintanswer-status-questionimplemented
generic-api-restraintrestraintavoid-generic-api-requestimplemented
lazy-tool-discoverylive_behaviorlazy-discover-agents, lazy-core-no-search, lazy-unauthorized-undisclosedimplemented
orient-and-bounded-delegationworkflowworkflow-orient-delegateimplemented
context-document-and-progressworkflowworkflow-context-document-progressimplemented
governed-wait-and-continuationworkflowworkflow-governed-waitimplemented
workspace-inspect-recover-reportworkflowworkflow-workspace-recoveryimplemented

Production lifecycle gates

These are deterministic test owners, not model-run results. Live model outcomes remain in Overview and Latest.

GateRequirementDeterministic ownerStatus
provider-round-tripprovider -> runnerd -> control plane -> runnerd -> providersrc/live/runnerd-codex-transport.test.tsOwned
single-daemonone Paperclip-owned sandbox daemon and no TypeScript sandbox dispatchersrc/live/runnerd-codex-transport.test.tsOwned
catalog-driftcanonical catalog consistency and drift rejectionrunner/crates/runner-core/src/provider_bridge.rs::durable_session_refuses_catalog_driftOwned
authorizationauthorization denial and unknown-tool rejectionrunner/crates/runner-core/src/provider_bridge.rs::forwards_only_authorized_calls_and_correlates_resultsOwned
duplicatescorrelated duplicate and conflicting-result handlingrunner/crates/runner-core/src/provider_bridge.rs::rejects_unknown_tools_and_conflicting_duplicate_resultsOwned
recoverylost ACK, reconnect, runner restart, and provider-thread recoverysrc/mock-core/durable-recovery.test.ts; src/live/live-session.test.tsOwned
interruptinterruption while waiting for a tool resultsrc/live/live-session.test.ts; src/live/clean-room-server.test.tsOwned
boundsbounded queues, logs, frames, and retriessrc/live/turn-stream.test.ts; runner/crates/runner-core/src/durable/runner.rsOwned
cleanupprocess-group cleanup with no abandoned providerrunner/crates/runner-core/tests/process_supervisor.rsOwned
secretssecret isolation and network restrictionssrc/live/clean-room-server.test.ts; runner/crates/runner-core/src/durable/runner.rsOwned
transcript-accountingassistant transcript capture, token usage, and cost inputssrc/live/live-session.test.ts; src/cli/eval-session.tsOwned
state-reconstructionfixture and post-run state reconstructionsrc/scenarios/scenario-explorer.test.ts; src/live/live-session.test.tsOwned
paperclip-adapter-selectionpaperclip_runner is selectable without changing legacy adapter behaviorserver/src/services/native-runtime/runtime-mode.test.ts; server/src/__tests__/heartbeat-native-runner-selection.test.tsOwned
real-control-plane-bindingadvertised runner tools re-authorize the live company, issue, agent, and run before using real Paperclip servicesserver/src/services/native-runtime/paperclip-runner-tool-authority.test.ts; server/src/services/native-runtime/paperclip-runner-real-server.integration.test.tsOwned
shared-paperclip-prp-routeRust runnerd authenticates to the shared Paperclip server route and completes a real bound semantic tool callserver/src/realtime/runner-prp-ws.test.ts; server/src/services/native-runtime/paperclip-runner-real-server.integration.test.tsOwned
real-control-plane-idempotencyreal service mutations are replay-safe and reject conflicting idempotency-key reuseserver/src/services/native-runtime/paperclip-runner-tool-authority.test.tsOwned
paperclip-thread-projectioncommitted PRP reasoning and semantic tool items render in the existing Paperclip task chatui/src/adapters/paperclip-runner/index.test.ts; server/src/services/native-runtime/paperclip-control-plane-port.test.tsOwned