paperclip-runner evalsstatic evidence
Protocol coverage inventory
27/27 live product operations have an authored live case. Scenario-only operations are owned by deterministic conformance and the test-only escape hatch never counts as product coverage.
| Area | Operation | Surface | Product coverage | Live cases | Status |
|---|---|---|---|---|---|
| Active task context and communication | get_task_context | scenario, live | yes | get-task-context, context-before-action | Implemented |
| Active task context and communication | get_task_history | scenario, live | yes | read-task-history | Implemented |
| Active task context and communication | report_progress | scenario, live | yes | context-before-action, native-codex-resume | Implemented |
| Active task context and communication | answer_status_question | scenario, live | yes | answer-status-question | Implemented |
| Active task context and communication | finish_task | scenario, live | yes | finish-task | Implemented |
| Active task context and communication | block_task | scenario, live | yes | block-task | Implemented |
| Active task context and communication | request_review | scenario, live | yes | request-task-review | Implemented |
| Documents, interactions, and deliverables | list_documents | scenario, live | yes | read-task-document | Implemented |
| Documents, interactions, and deliverables | read_document | scenario, live | yes | read-task-document, update-task-document | Implemented |
| Documents, interactions, and deliverables | list_document_revisions | scenario, live | yes | read-document-revisions | Implemented |
| Documents, interactions, and deliverables | write_document | scenario, live | yes | create-task-document, update-task-document | Implemented |
| Documents, interactions, and deliverables | request_human_input | scenario, live | yes | request-human-confirmation | Implemented |
| Documents, interactions, and deliverables | register_deliverable | scenario, live | yes | register-deliverable | Implemented |
| Documents, interactions, and deliverables | inspect_operation_result | scenario | yes | — | Implemented |
| Company discovery | search_tasks | scenario, live | yes | search-company-tasks | Implemented |
| Company discovery | list_agents | scenario, live | yes | list-company-agents | Implemented |
| Company discovery | get_agent | live | yes | get-company-agent | Implemented |
| Company discovery | list_projects | scenario | yes | — | Implemented |
| Company discovery | list_goals | scenario | yes | — | Implemented |
| Delegation and dependencies | create_task | scenario, live | yes | create-child-task | Implemented |
| Delegation and dependencies | set_dependencies | scenario, live | yes | set-task-dependencies | Implemented |
| Approvals and governance | list_approvals | scenario, live | yes | list-approvals | Implemented |
| Approvals and governance | get_approval | live | yes | get-approval | Implemented |
| Approvals and governance | get_approval_context | live | yes | get-approval-context | Implemented |
| Approvals and governance | request_approval | scenario, live | yes | request-approval | Implemented |
| Approvals and governance | decide_approval | scenario, live | yes | decide-approval | Implemented |
| Approvals and governance | comment_on_approval | scenario, live | yes | comment-on-approval | Implemented |
| Workspace runtime and continuation | get_workspace_runtime | scenario, live | yes | get-workspace-runtime | Implemented |
| Workspace runtime and continuation | control_workspace_service | scenario, live | yes | control-workspace-service | Implemented |
| Workspace runtime and continuation | schedule_wake | live | yes | schedule-task-wake | Implemented |
| Scenario-only protocol extensions | list_cases | scenario | yes | — | Implemented |
| Scenario-only protocol extensions | upsert_case | scenario | yes | — | Implemented |
| Scenario-only protocol extensions | list_routines | scenario | yes | — | Implemented |
| Scenario-only protocol extensions | manage_routine | scenario | yes | — | Implemented |
| Scenario-only protocol extensions | list_company_skills | scenario | yes | — | Implemented |
| Scenario-only protocol extensions | sync_company_skills | scenario | yes | — | Implemented |
| Scenario-only protocol extensions | list_secret_metadata | scenario | yes | — | Implemented |
| Scenario-only protocol extensions | read_secret_value | scenario | yes | — | Implemented |
| Scenario-only protocol extensions | export_company | scenario | yes | — | Implemented |
| Scenario-only protocol extensions | administer_company | scenario | yes | — | Implemented |
| Test-only escape hatch | generic_api_request | scenario, live | no | avoid-generic-api-request | Implemented |
Workflow and restraint coverage
| Requirement | Layer | Cases | Status |
|---|---|---|---|
orientation-before-action | live_behavior | context-before-action | implemented |
document-discovery-and-revision | live_behavior | read-task-document, update-task-document | implemented |
bounded-delegation | workflow | operational/bounded-delegation, create-child-task | implemented |
blocker-and-continuation-wake | workflow | operational/blocker-wake, set-task-dependencies, schedule-task-wake | implemented |
approval-request-decision-continuation | workflow | operational/approval-continuation, request-approval, decide-approval | implemented |
escalation-and-review | workflow | operational/escalation-review, request-task-review | implemented |
documents-artifacts-and-receipts | workflow | operational/documents-artifacts-receipts, register-deliverable | implemented |
semantic-receipts-and-context | deterministic_conformance | operational/semantic-receipts-context | implemented |
budget-conflicts | restraint | operational/budget-conflicts | implemented |
recovery-and-restraint | restraint | operational/recovery-restraint | implemented |
secret-safety | restraint | operational/secret-safety-focus | implemented |
status-only-wake-restraint | restraint | answer-status-question | implemented |
generic-api-restraint | restraint | avoid-generic-api-request | implemented |
lazy-tool-discovery | live_behavior | lazy-discover-agents, lazy-core-no-search, lazy-unauthorized-undisclosed | implemented |
orient-and-bounded-delegation | workflow | workflow-orient-delegate | implemented |
context-document-and-progress | workflow | workflow-context-document-progress | implemented |
governed-wait-and-continuation | workflow | workflow-governed-wait | implemented |
workspace-inspect-recover-report | workflow | workflow-workspace-recovery | implemented |
Production lifecycle gates
These are deterministic test owners, not model-run results. Live model outcomes remain in Overview and Latest.
| Gate | Requirement | Deterministic owner | Status |
|---|---|---|---|
provider-round-trip | provider -> runnerd -> control plane -> runnerd -> provider | src/live/runnerd-codex-transport.test.ts | Owned |
single-daemon | one Paperclip-owned sandbox daemon and no TypeScript sandbox dispatcher | src/live/runnerd-codex-transport.test.ts | Owned |
catalog-drift | canonical catalog consistency and drift rejection | runner/crates/runner-core/src/provider_bridge.rs::durable_session_refuses_catalog_drift | Owned |
authorization | authorization denial and unknown-tool rejection | runner/crates/runner-core/src/provider_bridge.rs::forwards_only_authorized_calls_and_correlates_results | Owned |
duplicates | correlated duplicate and conflicting-result handling | runner/crates/runner-core/src/provider_bridge.rs::rejects_unknown_tools_and_conflicting_duplicate_results | Owned |
recovery | lost ACK, reconnect, runner restart, and provider-thread recovery | src/mock-core/durable-recovery.test.ts; src/live/live-session.test.ts | Owned |
interrupt | interruption while waiting for a tool result | src/live/live-session.test.ts; src/live/clean-room-server.test.ts | Owned |
bounds | bounded queues, logs, frames, and retries | src/live/turn-stream.test.ts; runner/crates/runner-core/src/durable/runner.rs | Owned |
cleanup | process-group cleanup with no abandoned provider | runner/crates/runner-core/tests/process_supervisor.rs | Owned |
secrets | secret isolation and network restrictions | src/live/clean-room-server.test.ts; runner/crates/runner-core/src/durable/runner.rs | Owned |
transcript-accounting | assistant transcript capture, token usage, and cost inputs | src/live/live-session.test.ts; src/cli/eval-session.ts | Owned |
state-reconstruction | fixture and post-run state reconstruction | src/scenarios/scenario-explorer.test.ts; src/live/live-session.test.ts | Owned |
paperclip-adapter-selection | paperclip_runner is selectable without changing legacy adapter behavior | server/src/services/native-runtime/runtime-mode.test.ts; server/src/__tests__/heartbeat-native-runner-selection.test.ts | Owned |
real-control-plane-binding | advertised runner tools re-authorize the live company, issue, agent, and run before using real Paperclip services | server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts; server/src/services/native-runtime/paperclip-runner-real-server.integration.test.ts | Owned |
shared-paperclip-prp-route | Rust runnerd authenticates to the shared Paperclip server route and completes a real bound semantic tool call | server/src/realtime/runner-prp-ws.test.ts; server/src/services/native-runtime/paperclip-runner-real-server.integration.test.ts | Owned |
real-control-plane-idempotency | real service mutations are replay-safe and reject conflicting idempotency-key reuse | server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts | Owned |
paperclip-thread-projection | committed PRP reasoning and semantic tool items render in the existing Paperclip task chat | ui/src/adapters/paperclip-runner/index.test.ts; server/src/services/native-runtime/paperclip-control-plane-port.test.ts | Owned |