# Current production chat evaluation: completed twenty-case snapshot

**Coverage: 36 distinct scenarios measured / 6 qualifying passes / 29 latest failures / 1 mixed-model pass / 208 unmeasured.** The mixed-model pass is genuine product evidence but excluded from the requested uniform-DeepSeek threshold.

This is a partial snapshot of run [40630370749](https://github.com/paperclipai/paperclip/actions/runs/40630370749), source `81bba1dd0d4f77448bf4c5e360f3c78760e34ce2`, production baseline `39e811c9c5226979fce42e052a27cfb28301018a`. Exact configured model: `openrouter/deepseek/deepseek-v4.1-flash-1017` through OpenCode/OpenRouter.

## What actually works on the current production baseline

| Actual human request | Observed end-to-end state proof | Existing evidence page |
| --- | --- | --- |
| Have Morgan report to Riley from now on. | Saved manager change and organization view; relevant assignment and company invariants retained, 9/9 checks. | [Pilot](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40620786554-1-five-report/) |
| Archive the Launch project but keep its history. | Actual archived project with readable retained history, 10/10 checks. | [Pilot](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40620786554-1-five-report/) |
| Save a short implementation plan on the website task. | Versioned plan saved on the intended task, actual document link and substantive brief facts retained, 9/9 checks. | [Twelve-case report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40624431756-1-full12-report/) |
| Fix the confusing pricing paragraph and resolve my comment. | Actual anchored document edit and correct annotation resolution, unrelated feedback preserved, 9/9 checks. | [Twelve-case report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40624431756-1-full12-report/) |
| Give me a downloadable copy of the website. | Actual product-registered download independently opened and matched to produced bytes; source task retained, 7/7 checks. | Actual completed measurement in this campaign. |

| Find the latest approved design for the launch website. | Actual approved revision distinguished from the newer unapproved draft, with authenticated target/source/history and access proof, 9/9 checks; original failed initial retained. | Actual completed final measurement in this campaign. |

These six are measured passes on production39e. [The full 244-scenario coverage inventory and prerequisite lists](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40584967834-1-final34-report/) remain inspectable in the preceding historical report. Its old outcomes are historical, not extra current-baseline passes. Current measured outcomes and counts are below.

## First four final-correction outcomes

The first four selected final corrections all failed. Their original outcomes and prior attempts remain unchanged; these four final allowances are exhausted. All four completed cleanup and have evidence manifests with no leaks or missing files. At this snapshot twenty selected cells are measured and the remaining twenty await completed evidence. Root resumed normal bounded waves after the intermittent tool failures showed many successful MCP parts and later recovery; this does not establish their cause or universal transport health. Global paid parallelism remains four across operations. Queued jobs consume no scenario attempt. This snapshot does not establish a shared provider outage or transport cause.

| Scenario | Actual outcome and stage | Evidence-supported cause or limitation | Remaining correction |
| --- | --- | --- | --- |
| cases.browse-cases | FAIL; actual native succeeded | Correct C1/C2 plaintext results and excluded done C3, but the rendered reply had no usable case links. A separate UI propagation defect is being addressed; this result remains failed. | 0 |
| agents.managed-profile | FAIL; canonical query sent, native still running at phase deadline | Empty reply; no terminal error observed. No provider cause established. | 0 |
| budgets-costs.spend-breakdown | FAIL; main native succeeded | Reply reported repeated basic MCP request timeouts and did not supply the actual agent breakdown/incomplete-total outcome. Retained handler results exist; the delivery boundary/cause is unresolved. Warmup did not show the same markers. | 0 |
| budgets-costs.resolve-budget | FAIL; canonical query sent, native succeeded | A real bound budget question with a $10 choice was visible despite empty prose. The fixture used a separate board incident control without answering that question, so its intended question/response-wake continuation was not verified. Original final remains failed/exhausted. | 0 |

## Four additional completed outcomes

| Scenario | Original outcome / actual native evidence | Evidence-supported limitation | Remaining correction |
| --- | --- | --- | --- |
| companies.archive-company | FAILED initial; one actual started native, still running at phase deadline and cancelled during cleanup | Generic identity-response wait did not obtain the required archive/restore response; no terminal provider cause established. | 1 |
| cases.create-case | FAILED final; three recorded run rows, only two have actual start timestamps | Declared run-bound guard stopped the case. The unstarted queued row is not a third native execution. No partial result is promoted to pass. | 0 |
| companies.agent-observability | FAILED initial; two actual native runs succeeded | Actual native launch recorded agent_observability.unavailable with unsupported_harness: selected OpenCode trace export is not supported by the pinned export-launch contract. Both native runs succeeded, but the required export and later disable branch were not established. | 1 |
| cases.cases-disabled | FAILED final; one native remained running at the frozen outer deadline, then cleanup cancelled it | Required truthful availability/choice did not settle; one request-timeout marker was observed. That isolated symptom is not proof of a shared transport cause. | 0 |

The first eight originals retain FAILED status, cleanup passed, and manifests with no leaks or missing files. Cases run-row limits and actual native executions are distinct evidence. No timeout, queued row or partial semantic success is silently regraded.

## Four additional completed outcomes and model-policy boundary

| Scenario | Preserved original outcome | Evidence-supported interpretation | Remaining correction |
| --- | --- | --- | --- |
| companies.prompt-injection-guard | PASS 10/10; authenticated native/build/source and cleanup | Human settings configured the classifier; both actual fixed samples returned the expected verdict and recorded charges, while chat respected board-only management. **Mixed-model evidence:** chat used requested DeepSeek1017, but two classifier calls used `openai/gpt-oss-safeguard-20b`. Not qualifying uniform1017 credit. | One explicitly reviewed configuration correction may be admitted later; no raw result regrade. |
| companies.import-preview | FAILED initial | Required conflict preview, explicit human rename/import continuation and scoped import audit were not established. No unretained provider cause inferred. | 1 |
| companies.export-company | FAILED initial | Supported package, actual authorized download and nonportable-credential explanation did not satisfy the full outcome. | 1 |
| tasks.delegate-announcement | FAILED final; exhausted | Sole started native succeeded; a prior queued assignment row was cancelled before starting when the task was reassigned to Morgan. The legacy observer rejects cancelled rows and stopped before worker admission. This is not proof of a failed worker native or of the complete returned announcement. | 0 |

The mixed-model human request was **“Turn on the extra protection for suspicious app results and check that it works.”** Its successful settings/classifier result is disclosed as actual product behavior, while its auxiliary model configuration prevents uniform-model qualification. The product exposes a real editable Model input; any future correction must save the exact requested service model before actual tests, preserve their real verdict/security/UI proof, and receive its own explicit finite manifest accounting. Defaults and original measurement remain unchanged.

## Four further completed outcomes

| Scenario | Original outcome | Observed proof or limitation | Remaining correction |
| --- | --- | --- | --- |
| documents-artifacts.produce-download | PASS 7/7, qualified requested-model evidence | A real product-registered `website.html` download matched the produced source bytes, with source task and exact authenticated query-attachment identity retained. Actual source/build/native/cell-restore and cleanup/privacy verified. | No unchanged rerun requested. |
| documents-artifacts.revise-delivery | FAILED initial | Actual saved-source preservation, exact revised download and retrievable original did not satisfy the independent oracle, although revision-context wording passed. Cause remains under review; no result rescue from prose alone. | 1 |
| goals.explain-purpose | FAILED initial | Initial purpose/chain/rationale prerequisite was not accepted; later missing-association/private/read-only checks were not established. Full frozen journey was not completed. | 1 |
| execution-environments.choose-capable-target | FAILED initial, setup before native execution | Public environment creation returned 422; zero native runs. Source review proves the fixture tries to persist the built-in fake sandbox provider, which production reserves for internal probes. A genuine public second target is required; no fake provider rename or database bypass is permitted. Original response body was withheld. | 1 |

All four have cleanup passed, no evidence-manifest leaks or missing files, and no diagnostic privacy sidecar. The current 20-result snapshot contains three raw passes and seventeen original failures. Future fixture/oracle changes and unstarted queued cells are not new measured successes.

## Current baseline coverage

Across the five-case pilot, twelve initial journeys, and these twenty current-run measurements: **36 distinct scenarios measured: 6 qualifying passes, 29 latest failures, 1 mixed-model pass, 208 not measured**, totaling 244. There are **37 actual measurements**, with raw outcomes **7 PASS / 30 FAIL**. The design-search final repeats an already measured failed initial; both remain inspectable. Only six latest scenario outcomes meet the requested uniform-model threshold. Historical production/model lanes and the failed instrumented diagnostic remain separate.

Current qualifying passing scenarios are `agents.change-manager`, `projects.archive-project`, `documents-artifacts.write-plan`, `documents-artifacts.resolve-feedback`, `documents-artifacts.produce-download`, and `inbox-search.find-design`. A pass proves its actual tested human journey, not other unrun features.

Unmeasured includes the twenty cells of this manifest without completed imported evidence and other unrun journeys, explicit invitation/core dependencies, external prerequisites, and incomplete or blocked public paths. Pending operations and queued jobs consume no ordinal before actual framework evidence. The separately dispatched 29-case manifest has no imported measurements at this cutoff. No original measurement is regraded by this summary.

## Latest four completed outcomes

| Scenario | Preserved outcome and stage | Evidence-supported interpretation | Remaining correction |
| --- | --- | --- | --- |
| inbox-search.find-design | PASS final2, 9/9, qualified requested-model evidence | Actual approved revision and task link are distinguished from the unapproved revision; source, read-only history, access and same-run build/native proof verified. Original failed initial retained. | Final used; no unchanged rerun. |
| instance-operations.instance-settings | FAILED initial, canonical query sent | Native was still running with an empty reply at the fixed150s intermediate guidance wait. No terminal native error observed; required human configuration/route journey was not reached. Shared-clock correction is under source review, not applied to this result. | 1 |
| questions-and-approvals.accept-one-proposal | FAILED initial, canonical query sent | Source native remained running at the fixed150s initial wait. One additional task run row was queued without a start; it is not a second executed native. One provider-observed create_task timeout and seventeen completed MCP parts were retained; cause unresolved. Repeat/reload proof was not reached. | 1 |
| projects.project-defaults | FAILED initial, post-query settle wait | Required defaults explanation and selected public-state outcome did not settle. Two distinct provider-observed search_api timeouts and twenty-one completed MCP parts, including later successes, were retained. No shared outage or transport cause is proved. | 1 |

All twenty originals passed cleanup and have evidence manifests with no leaks or missing files. Recovered intermittent MCP symptoms may recur and remain monitored; identity, privacy, cleanup and prerequisite failures still stop affected admission. The frozen source and finite per-case bounds remain unchanged.

## Separate diagnostic evidence

The instrumented artifact-search initial in [40635108320](https://github.com/paperclipai/paperclip/actions/runs/40635108320) remains FAILED, with clean cleanup/privacy. Its retained timing showed 15 successful Paperclip MCP tool completions and no request timeout in that sample. This does not establish universal health or explain earlier failures. Its source/native build differs from clean production39e, so it supplies no clean-production pass credit. A narrow returned-copy observer/oracle correction has independently passed review and is included in a separately dispatched future final. The diagnostic failure remains unchanged; no final result has been imported.