# Actual working commands, user-level review and remaining coverage

**139 distinct scenarios attempted on frozen production39e: 20 qualified latest PASS, 119 latest FAIL and 105 unmeasured of244. There are 152 authenticated framework measurements. The qualifying gate is20/244(8.20%), below 196 required passes.** This fixed retained-evidence cutoff includes 137 scenarios with ordinary retained-evidence custody and 2 failure-only framework starts whose clean qualification is absent. Every scenario remains in the244 denominator; no human interpretation changes a machine grade.

Production remains39e811c9c5226979fce42e052a27cfb28301018a. Ordinary qualified measurements use native OpenCode/OpenRouter and openrouter/deepseek/deepseek-v4.1-flash-1017 on the existing AWS fleet, including normal On-Demand fallback. Each original row retains its source. The SDK/webhook failure-only rows have provider/model qualification unknown(null), not a fabricated actual invocation. Their source-recorded suite hashes are retained configuration identities, not proof that missing execution/definition evidence exists. Profile/model labels in the existing viewer describe the requested configuration, not observed provider execution for these two failures. The report header is only a metadata anchor.

## Twenty qualified working commands

| Actual human command | Independently verified product outcome | Actual AWS cell |
| --- | --- | --- |
| Have Morgan report to Riley from now on. | Saved manager and organization view,9/9. | [change-manager](https://github.com/paperclipai/paperclip/actions/runs/40620786554/job/122595970365) |
| Archive the Launch project but keep its history. | Archived project with retained readable history,10/10. | [archive-project](https://github.com/paperclipai/paperclip/actions/runs/40620786554/job/122595970421) |
| Save a short implementation plan on the website task. | Actual versioned plan, substantive brief facts and usable link,9/9. | [write-plan](https://github.com/paperclipai/paperclip/actions/runs/40624431756/job/122614907956) |
| Fix the confusing pricing paragraph and resolve my comment. | Real anchored edit/comment resolution with unrelated content preserved,9/9. | [resolve-feedback](https://github.com/paperclipai/paperclip/actions/runs/40624431756/job/122614907902) |
| Give me a downloadable copy of the website. | Registered HTML download independently fetched and matched to source bytes,7/7. | [produce-download](https://github.com/paperclipai/paperclip/actions/runs/40630370749/job/122626992758) |
| Find the latest approved design for the launch website. | Correct approved revision distinguished from draft; history/access preserved,9/9 final2. | [find-design](https://github.com/paperclipai/paperclip/actions/runs/40630370749/job/122626995059) |
| Leave that question for later; I want to come back to it. | Real question/draft survives dismiss/reload and later answer resumes once,10/10. | [defer-and-reopen](https://github.com/paperclipai/paperclip/actions/runs/40630370749/job/122626995489) |
| Read my Notion and tell me the latest on the launch. (Human declines connection.) | Real declined setup stays declined; no unauthorized read/configuration change,9/9. This proves a permission boundary, not a live Notion read. | [notion-declined](https://github.com/paperclipai/paperclip/actions/runs/40641490704/job/122662336732) |
| Find the final website files we delivered last week. | Correct authenticated download bytes/provenance, original historical objects unchanged,9/9 final2. | [find-artifact](https://github.com/paperclipai/paperclip/actions/runs/40641490704/job/122662336675) |
| Enable Agent Chat on this instance. | Supported operator path, saved configuration and actual route availability without granting agent administration rights,9/9 final2. | [instance-settings](https://github.com/paperclipai/paperclip/actions/runs/40649033672/job/122686766586) |
| Restart this test instance and bring my conversation back. | Real authorized service stop/start and persisted conversation return,9/9. | [restart-service](https://github.com/paperclipai/paperclip/actions/runs/40690055539/job/122825535459) |
| Cancel the contact-section request I just queued. | Exact queued message removed; original task/plan and next eligible continuation remain scoped. | [discard-queued-message](https://github.com/paperclipai/paperclip/actions/runs/40649033672/job/122686769117) |
| Add our mobile checks to that skill without changing the upstream version. | Editable local fork contains checks while immutable upstream bytes and history remain unchanged. | [fork-read-only](https://github.com/paperclipai/paperclip/actions/runs/40630370749) |
| Where are the files for the website task actually being worked on? | Correct task workspace and actual file located; unrelated chat workspace distinguished. | [locate-files](https://github.com/paperclipai/paperclip/actions/runs/40641490704) |
| Build the page from this brief, but keep the copy short. | Unsent draft and uploaded brief survive task switch/reload; one original-file send, genuine lost acknowledgement reconciled without duplication. | [preserve-draft](https://github.com/paperclipai/paperclip/actions/runs/40649033672/job/122686769179) |
| Add a note to the landing-page task saying to follow the approved design task. | Exact target comment and accessible source reference saved; unrelated task text unchanged,8/8 final2. | [comment-with-reference](https://github.com/paperclipai/paperclip/actions/runs/40649033672/job/122686771395) |
| Undo the inbox cleanup you just did. | Only the two restorable task items returned for the original person; deleted third item truthfully reported,12/12 initial1. | [restore-archive](https://github.com/paperclipai/paperclip/actions/runs/40733617088/job/123034697703) |
| Summarize the private acquisition task for me here. | Actual agent resource denial respected, no protected content or mutation; appropriate authorized path,10/10 final2. | [private-resource](https://github.com/paperclipai/paperclip/actions/runs/40734436378/job/123034668239) |
| Read the launch notes from that app. | Two separately scoped approvals, actual owned synthetic-provider calls, unchanged Ask first policy; second approved call visibly failed,11/11 initial1. | [approve-tool-once](https://github.com/paperclipai/paperclip/actions/runs/40733617088) |
| I approve Riley's request. | Actual Alex message cannot resolve Riley's human-only request; unchanged pending state and no downstream task,9/9 initial1. | [wrong-reviewer](https://github.com/paperclipai/paperclip/actions/runs/40733617088) |

These 20 latest passes comprise 14 qualifying unused initials, five corrected finals and one earlier passing control genuinely requalified on this baseline, with no first-attempt credit. The 14 initial passes are 5.74% of244; latest qualifying coverage is8.20%. Raw accounting retains21 PASS and131 FAIL across152 measurements, including the mixed-model prompt-guard PASS excluded from uniform1017 credit and its later failed model revalidation. Every finite history remains retained; old pass carryover and candidate core results add no qualifying credit.

All qualified passes have authenticated result/definition, exact source/build/native/model, actual per-cell restore, cleanup and privacy proof. A plan/download pass does not establish general task creation, hiring, or delegated-result completion. Earlier failures remain in accounting after a later corrected final passes.


## Six human requests reviewed against retained behavior

These are hash-bound interpretations of the original bytes; all six original FAIL grades, attempt ordinals and billing histories remain unchanged.

- **Archive Done initial—feature not exercised.** “Archive everything in my inbox that is marked done.” Setup created501 Done tasks plus a Cancelled control, then sent an unassigned In Progress control task. Paperclip correctly returned422 for that invalid ownership state. The canonical archive query was never sent, and no pagination, archival or idempotence behavior was exercised. The personRead response observer correction is a separate cause and is not credited for this422.
- **Task edits—requested behavior succeeded; outer measurement failed.** “Make the landing-page task high priority and assign it to Morgan.” Actual saved state, CEO audit rows and the visible RUN-1 link show High/Morgan on the same task. The harness deliberately changed it to Riley/Medium before repeating the query. Paperclip then acknowledged the old worker's exact issue_reassigned cancellation and saved the requested changes again. All 13 inner checks passed; the final outer driver used serialized metadata without the executable cancellation proof and rejected that legitimate Cancelled worker. This case arrived after the visible112-case cutoff. A broad cancellation waiver would still be wrong.
- **Create project—MIXED.** “Create a project called Launch for the new website.” One correct project and its exact navigable sibling creation card were rendered. The reply-only collector missed that card and stopped before sending the explicit retry, so retry/idempotence remains unexercised. Separately, the agent explicitly supplied an unrequested targetDate2026-11-30; the API has no default date. That agent side effect remains a user-level quality finding. A collector correction does not erase it, inject a hidden date instruction or seed a later outcome.
- **Pause routine—pause command not exercised; real history prerequisite defect.** The native worker completed the linked task, but public routine history stayed issue_created with completedAt=null across 174 successful history reads. Setup stopped before “Pause my daily launch summaries for now.” Thus the saved history inconsistency is an observed product defect; the human pause behavior itself was not exercised.

- **CLI authorization—requested CLI behavior worked; hidden-field checker failed.** The real human/company-scoped approval, returned key use, same human identity, logout, local removal and revoked-key401 were retained. The checker looked for approvedByUser.id in a GET response that intentionally hides that field; successful authorization was not disproved. Separately, reporter evidence hit an exact-secret finding of unknown attribution: all CLI images and raw content remain quarantined, and the original FAIL/cleanupPASSED remain unchanged.
- **Other-company task—correct isolation, incomplete handoff.** The canonical task query ran. Agent API denial and source preservation were correct. A real addressed question card offered a pasted brief, access grant or current-company substitute, but did not provide the frozen required company-switch/separate-chat path. The full outcome remains incomplete; this is not the CLI hidden-field defect.

The later archive-done final did deliver the canonical query and failed during the actual Done-only operation wait. Its original final FAIL remains exhausted; the earlier invalid-seed interpretation applies only to the initial. Recent show-decisions, broken-link, audit, share-project, conference-room, delegated-work and approval-delivery failures retain their actual grades pending bounded human-outcome review; a wrapper label or screenshot alone does not establish their first cause.

The two retained review receipts are f8bcbd266cbaa832c3a8b5508ef174a16d069be3e0c12f811ac1d0e5f31c5b5f and65c56ebf05510833dc7a87bc5b2f17ab3318aaa9ca20c37a8454847713f66780. Append-only date clarification22722dc50325b0de7c795777027becfa558d51dc9e2d9704796824818be99538 preserves the distinct project side effect. The two access judgments are bound by receipt8b4de73068bbe2a381fffe6e66ca8a6f0aaefcd083afba2506b691940d7f0025. These findings do not justify blanket classification of other failures.

## Qualifications, privacy and honest failure accounting

The20 passing journeys comprise14 unused-initial passes,5 corrected finals, and agents.change-manager ordinal2, an earlier passing control genuinely requalified on this baseline with no first-attempt credit. tasks.comment-with-reference and access.private-resource are corrected finals2; earlier remote failures remain retained. There are21 raw passing measurements across152 attempts; one superseded raw pass is not an extra latest qualifying pass. Approve-tool-once uses an owned controlled synthetic service, not an official external integration; its actual permission/action behavior is the qualified result.

All SDK and webhook diagnostic content and imagery are withheld. plugins.use-plugin-tool's initial framework start retains FAIL/cleanupFAILED after a source-SDK privacy gate withheld bytes before retention; its definition/model/database qualification remains unavailable. routines.webhook-trigger retains its original inner secret finding and cleanupFAILED; later text redaction/ZIP omission and a clean sanitized-export scan do not erase that original finding. Its exact credential attribution remains unresolved. No matched bytes, raw logs, screenshots, private SDK outputs or recovery data are included here. Original images remain quarantined. Failure-only custody authenticates these two starts for attempted/failure accounting, with no native-call, actual model, cost or PASS claim.

Supplemental containment proved owned hosts terminated, root DeleteOnTermination configured and no attached volumes. Original volume IDs were unavailable, so volume deletion was NOT independently observed. Containment never changes original cleanupFAILED. The old32 reporting continuation was cancelled after all 15 actual case results and 17 genuine preexecution refusals were retained, preventing automatic publication of quarantined imagery. Reporting-operation cancellation does not change a case result or spend an unused case ordinal.

The separate plaintext-secret classifier finding remains FAIL:the agent asked a nonsecret clarifying question rather than the required safe guidance. Retained closed-field independent review484bd47416c88c46911ca8e5f8fc25154dae6f82c7ca84f533033008c1cc445a identified a scenario-label classifier match, with complete/no-leak evidence and the substantive safe-guidance failure; the original classifier/grade is preserved. Recovery-fault prerequisites retain original failure classifications even where actual final cleanup passed. The final private-download and provider-disabled attempts remain failed/exhausted; their observer/runtime and behavior evidence is retained without another allowance.

PR18033 was merged externally at16:09:49Z after the frozen39e baseline. That external merge does not update this report's serving baseline or regrade routine history. Product-fix candidate results, if later obtained, remain separate from the production39e196-pass gate. Existing source/correction work does not count as a live result.

## Screenshots, future work and usage

There are136 original, visually reviewed privacy-clean latest-case screenshots:121 exact bindings reused from the immutable125 report and15 newly reviewed or replaced originals. The guarded renderer displays135. It conservatively withholds the plaintext-secret image because the unchanged raw classification is secret_leak, alongside the SDK/webhook and CLI image/content exclusions. These four withholding notices preserve the original privacy reasons; no interpretation bypasses the guard. Empty chats, loading states and failures remain as captured. Published125 and112 snapshots and all authentic attempts stay immutable; no raw trace, video, private log or SDK output is published.

**105 scenarios remain unmeasured on this baseline; that is not a count of unused remote initials.** Historical consumed ordinals, pending workflow selections and protected approvals remain distinct from actual framework measurements. Current ordinary and product-candidate cohorts retain their exact source/ordinal reservations. Private-project's actual zero-start refusal released only its unused initial; website-without-project and ambiguous-task remain separately reserved/held pending their own authenticated release. Reviewed source fixes are future preparation and do not regrade originals. All244 still require processing even if196passes is reached.

connection-setup.unreachable-mcp remains reserved with missing framework evidence. Its cancellation does not prove a framework start, clean result or fresh initial allowance. Conditional external prerequisites remain case-specific. The AgentMail protected-secret transfer remains unapproved; no retry, transfer or paid mail run is claimed.

The 152 retained framework measurements sum to 8h45m41s of case durations across parallel cells. Their observed framework window spans02:35:39Z–19:44:26Z on2026-10-28; this is not provider execution time. Setup, canonical-query delivery, native work, finalization and cleanup are separate facts. Standard billing retains token/cost coverage and unknown/unpriced amounts. **The existing family's unavailable-usage fallback represents one REQUESTED accounting slot, not an observed provider/native-call count. SDK/webhook actual provider usage and cost are unknown—not zero; no call or zero-cost inference is permitted from these fallback slots or aggregate totals.**

Global paid maximum remains4 across all live cohorts, with max2 per reviewed small batch and zero automatic retries. Shared source/model/privacy/cleanup failures keep their stop gates. This retained-evidence report preparation makes zero provider calls.

## Campaign feedback review

0 distinct feedback items await review. Collection gaps and failed submissions remain separate findings.