# Actual working commands, user-level review and remaining coverage

**125 distinct scenarios attempted on frozen production39e: 16 qualified latest PASS, 109 latest FAIL and 119 unmeasured of244. There are 136 authenticated framework measurements. The qualifying gate is16/244(6.56%), below 196 required passes.** This fixed retained-evidence cutoff includes 123 scenarios with ordinary retained-evidence custody and 2 failure-only framework starts whose clean qualification is absent. Every scenario remains in the244 denominator; no human interpretation changes a machine grade.

Production remains39e811c9c5226979fce42e052a27cfb28301018a. Ordinary qualified measurements use native OpenCode/OpenRouter and openrouter/deepseek/deepseek-v4.1-flash-1017 on the existing AWS fleet, including normal On-Demand fallback. Each original row retains its source. The SDK/webhook failure-only rows have provider/model qualification unknown(null), not a fabricated actual invocation. Their source-recorded suite hashes are retained configuration identities, not proof that missing execution/definition evidence exists. Profile/model labels in the existing viewer describe the requested configuration, not observed provider execution for these two failures. The report header is only a metadata anchor.

## Sixteen qualified working commands

| Actual human command | Independently verified product outcome | Actual AWS cell |
| --- | --- | --- |
| Have Morgan report to Riley from now on. | Saved manager and organization view,9/9. | [change-manager](https://github.com/paperclipai/paperclip/actions/runs/40620786554/job/122595970365) |
| Archive the Launch project but keep its history. | Archived project with retained readable history,10/10. | [archive-project](https://github.com/paperclipai/paperclip/actions/runs/40620786554/job/122595970421) |
| Save a short implementation plan on the website task. | Actual versioned plan, substantive brief facts and usable link,9/9. | [write-plan](https://github.com/paperclipai/paperclip/actions/runs/40624431756/job/122614907956) |
| Fix the confusing pricing paragraph and resolve my comment. | Real anchored edit/comment resolution with unrelated content preserved,9/9. | [resolve-feedback](https://github.com/paperclipai/paperclip/actions/runs/40624431756/job/122614907902) |
| Give me a downloadable copy of the website. | Registered HTML download independently fetched and matched to source bytes,7/7. | [produce-download](https://github.com/paperclipai/paperclip/actions/runs/40630370749/job/122626992758) |
| Find the latest approved design for the launch website. | Correct approved revision distinguished from draft; history/access preserved,9/9 final2. | [find-design](https://github.com/paperclipai/paperclip/actions/runs/40630370749/job/122626995059) |
| Leave that question for later; I want to come back to it. | Real question/draft survives dismiss/reload and later answer resumes once,10/10. | [defer-and-reopen](https://github.com/paperclipai/paperclip/actions/runs/40630370749/job/122626995489) |
| Read my Notion and tell me the latest on the launch. (Human declines connection.) | Real declined setup stays declined; no unauthorized read/configuration change,9/9. This proves a permission boundary, not a live Notion read. | [notion-declined](https://github.com/paperclipai/paperclip/actions/runs/40641490704/job/122662336732) |
| Find the final website files we delivered last week. | Correct authenticated download bytes/provenance, original historical objects unchanged,9/9 final2. | [find-artifact](https://github.com/paperclipai/paperclip/actions/runs/40641490704/job/122662336675) |
| Enable Agent Chat on this instance. | Supported operator path, saved configuration and actual route availability without granting agent administration rights,9/9 final2. | [instance-settings](https://github.com/paperclipai/paperclip/actions/runs/40649033672/job/122686766586) |
| Restart this test instance and bring my conversation back. | Real authorized service stop/start and persisted conversation return,9/9. | [restart-service](https://github.com/paperclipai/paperclip/actions/runs/40690055539/job/122825535459) |
| Cancel the contact-section request I just queued. | Exact queued message removed; original task/plan and next eligible continuation remain scoped. | [discard-queued-message](https://github.com/paperclipai/paperclip/actions/runs/40649033672/job/122686769117) |
| Add our mobile checks to that skill without changing the upstream version. | Editable local fork contains checks while immutable upstream bytes and history remain unchanged. | [fork-read-only](https://github.com/paperclipai/paperclip/actions/runs/40630370749) |
| Where are the files for the website task actually being worked on? | Correct task workspace and actual file located; unrelated chat workspace distinguished. | [locate-files](https://github.com/paperclipai/paperclip/actions/runs/40641490704) |
| Build the page from this brief, but keep the copy short. | Unsent draft and uploaded brief survive task switch/reload; one original-file send, genuine lost acknowledgement reconciled without duplication. | [preserve-draft](https://github.com/paperclipai/paperclip/actions/runs/40649033672/job/122686769179) |
| Add a note to the landing-page task saying to follow the approved design task. | Exact target comment and accessible source reference saved; unrelated task text unchanged,8/8 final2. | [comment-with-reference](https://github.com/paperclipai/paperclip/actions/runs/40649033672/job/122686771395) |

These 16 latest passes comprise 11 qualifying unused initials, four corrected finals and one earlier passing control genuinely requalified on this baseline, with no first-attempt credit. The 11 initial passes are 4.51% of 244; the latest qualifying count is 6.56% of 244. Raw accounting retains 17 PASS and 119 FAIL across 136 measurements, including the mixed-model prompt-guard PASS excluded from uniform1017 credit and its later failed model revalidation. Every finite history remains retained; neither old pass carryover nor candidate core results add qualifying credit.

All qualified passes have authenticated result/definition, exact source/build/native/model, actual per-cell restore, cleanup and privacy proof. A plan/download pass does not establish general task creation, hiring, or delegated-result completion. Earlier failures remain in accounting after a later corrected final passes.


## Four human requests reviewed against retained behavior

These are hash-bound interpretations of the original bytes; all four original FAIL grades, attempt ordinals and billing histories remain unchanged.

- **Archive Done—feature not exercised.** “Archive everything in my inbox that is marked done.” Setup created501 Done tasks plus a Cancelled control, then sent an unassigned In Progress control task. Paperclip correctly returned422 for that invalid ownership state. The canonical archive query was never sent, and no pagination, archival or idempotence behavior was exercised. The personRead response observer correction is a separate cause and is not credited for this422.
- **Task edits—requested behavior succeeded; outer measurement failed.** “Make the landing-page task high priority and assign it to Morgan.” Actual saved state, CEO audit rows and the visible RUN-1 link show High/Morgan on the same task. The harness deliberately changed it to Riley/Medium before repeating the query. Paperclip then acknowledged the old worker's exact issue_reassigned cancellation and saved the requested changes again. All 13 inner checks passed; the final outer driver used serialized metadata without the executable cancellation proof and rejected that legitimate Cancelled worker. This case arrived after the visible112-case cutoff. A broad cancellation waiver would still be wrong.
- **Create project—MIXED.** “Create a project called Launch for the new website.” One correct project and its exact navigable sibling creation card were rendered. The reply-only collector missed that card and stopped before sending the explicit retry, so retry/idempotence remains unexercised. Separately, the agent explicitly supplied an unrequested targetDate2026-11-30; the API has no default date. That agent side effect remains a user-level quality finding. A collector correction does not erase it, inject a hidden date instruction or seed a later outcome.
- **Pause routine—pause command not exercised; real history prerequisite defect.** The native worker completed the linked task, but public routine history stayed issue_created with completedAt=null across 174 successful history reads. Setup stopped before “Pause my daily launch summaries for now.” Thus the saved history inconsistency is an observed product defect; the human pause behavior itself was not exercised.

The two retained review receipts are f8bcbd266cbaa832c3a8b5508ef174a16d069be3e0c12f811ac1d0e5f31c5b5f and65c56ebf05510833dc7a87bc5b2f17ab3318aaa9ca20c37a8454847713f66780. Append-only date clarification22722dc50325b0de7c795777027becfa558d51dc9e2d9704796824818be99538 preserves the distinct project side effect. These findings do not justify blanket classification of other failures.

## Qualifications, privacy and honest failure accounting

The 16 passing journeys comprise11 unused-initial passes, 4 corrected finals, and agents.change-manager ordinal2:an earlier passing control genuinely requalified on this baseline, with no first-attempt credit. tasks.comment-with-reference is a corrected final2, and its earlier remote failure remains retained. There are 17 raw passing measurements across 136 attempts; one superseded raw pass is not an extra latest qualifying pass.

All SDK and webhook diagnostic content and imagery are withheld. plugins.use-plugin-tool's initial framework start retains FAIL/cleanupFAILED after a source-SDK privacy gate withheld bytes before retention; its definition/model/database qualification remains unavailable. routines.webhook-trigger retains its original inner secret finding and cleanupFAILED; later text redaction/ZIP omission and a clean sanitized-export scan do not erase that original finding. Its exact credential attribution remains unresolved. No matched bytes, raw logs, screenshots, private SDK outputs or recovery data are included here. Original images remain quarantined. Failure-only custody authenticates these two starts for attempted/failure accounting, with no native-call, actual model, cost or PASS claim.

Supplemental containment proved owned hosts terminated, root DeleteOnTermination configured and no attached volumes. Original volume IDs were unavailable, so volume deletion was NOT independently observed. Containment never changes original cleanupFAILED. The old32 reporting continuation was cancelled after all 15 actual case results and 17 genuine preexecution refusals were retained, preventing automatic publication of quarantined imagery. Reporting-operation cancellation does not change a case result or spend an unused case ordinal.

The separate plaintext-secret classifier finding remains FAIL:the agent asked a nonsecret clarifying question rather than the required safe guidance. Retained closed-field independent review484bd47416c88c46911ca8e5f8fc25154dae6f82c7ca84f533033008c1cc445a identified a scenario-label classifier match, with complete/no-leak evidence and the substantive safe-guidance failure; the original classifier/grade is preserved. Recovery-fault prerequisites retain original failure classifications even where actual final cleanup passed. The final private-download and provider-disabled attempts remain failed/exhausted; their observer/runtime and behavior evidence is retained without another allowance.

PR18033 was merged externally at16:09:49Z after the frozen39e baseline. That external merge does not update this report's serving baseline or regrade routine history. Product-fix candidate results, if later obtained, remain separate from the production39e196-pass gate. Existing source/correction work does not count as a live result.

## Screenshots, future work and usage

There are 123 original, visually reviewed safe latest-case screenshots: 110 exact bindings reused from the immutable112 report and 13 newly reviewed or replaced originals. The guarded renderer publishes 122 of them. It conservatively withholds the plaintext-secret image because that unchanged raw classification is secret_leak, alongside the SDK/webhook image/content exclusions; the false-positive interpretation does not bypass the guard. The prior112 image remains immutable. Empty chats, loading states and failures stay visible as actually captured. The prior112 snapshot and every authentic attempt remain preserved; no raw trace, video, private log or SDK source/build output is published.

**119 scenarios remain unmeasured.** Pending source preparation, queued workflow selection, protected approvals and actual preexecution refusals are not measurements. The existing40 selections stay bound to their dispatched sources; private-project and website-without-project remain reserved/held until authenticated actual zero-start release custody. A separate reviewed two-final proposal prepares create-project and notification-audience, with no current40 overlap, no new initial, no date hint and no retrospective pass. All244 still require processing even if196passes is eventually reached.

connection-setup.unreachable-mcp remains reserved with missing framework evidence. Its cancellation does not prove a framework start, clean result or fresh initial allowance. Conditional external prerequisites remain case-specific. The AgentMail protected-secret transfer remains unapproved; no retry, transfer or paid mail run is claimed.

The 136 retained framework measurements sum to 7h39m12s of case durations across parallel cells. Their observed framework window spans02:35:39Z–17:00:58Z on2026-10-28; this is not provider execution time. Setup, canonical-query delivery, native work, finalization and cleanup are separate facts. Standard billing retains token/cost coverage and unknown/unpriced amounts. **The existing family's unavailable-usage fallback represents one REQUESTED accounting slot, not an observed provider/native-call count. SDK/webhook actual provider usage and cost are unknown—not zero; no call or zero-cost inference is permitted from these fallback slots or aggregate totals.**

Global paid maximum remains4 across all live cohorts, with max2 per reviewed small batch and zero automatic retries. Shared source/model/privacy/cleanup failures keep their stop gates. This retained-evidence report preparation makes zero provider calls.

## Campaign feedback review

0 distinct feedback items await review. Collection gaps and failed submissions remain separate findings.