# Current production chat evaluation: completed sixteen-case snapshot

**Coverage: 33 measured / 5 qualifying passes / 27 failures / 1 mixed-model pass / 211 unmeasured.** The mixed-model pass is genuine product evidence but excluded from the requested uniform-DeepSeek threshold.

This is a partial snapshot of run [40630370749](https://github.com/paperclipai/paperclip/actions/runs/40630370749), source `81bba1dd0d4f77448bf4c5e360f3c78760e34ce2`, production baseline `39e811c9c5226979fce42e052a27cfb28301018a`. Exact configured model: `openrouter/deepseek/deepseek-v4.1-flash-1017` through OpenCode/OpenRouter.

## What actually works on the current production baseline

| Actual human request | Observed end-to-end state proof | Existing evidence page |
| --- | --- | --- |
| Have Morgan report to Riley from now on. | Saved manager change and organization view; relevant assignment and company invariants retained, 9/9 checks. | [Pilot](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40620786554-1-five-report/) |
| Archive the Launch project but keep its history. | Actual archived project with readable retained history, 10/10 checks. | [Pilot](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40620786554-1-five-report/) |
| Save a short implementation plan on the website task. | Versioned plan saved on the intended task, actual document link and substantive brief facts retained, 9/9 checks. | [Twelve-case report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40624431756-1-full12-report/) |
| Fix the confusing pricing paragraph and resolve my comment. | Actual anchored document edit and correct annotation resolution, unrelated feedback preserved, 9/9 checks. | [Twelve-case report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40624431756-1-full12-report/) |
| Give me a downloadable copy of the website. | Actual product-registered download independently opened and matched to produced bytes; source task retained, 7/7 checks. | Actual completed measurement in this campaign. |

These five are measured passes on production39e. [The full 244-scenario coverage inventory and prerequisite lists](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40584967834-1-final34-report/) remain inspectable in the preceding historical report. Its old outcomes are historical, not extra current-baseline passes. Current measured outcomes and counts are below.

## First four final-correction outcomes

The first four selected final corrections all failed. Their original outcomes and prior attempts remain unchanged; these four final allowances are exhausted. All four completed cleanup and have evidence manifests with no leaks or missing files. At this snapshot sixteen selected cells are measured and the remaining 24 await completed evidence; bounded normal waves have resumed. Queued jobs consume no scenario attempt. This snapshot does not establish a shared provider outage or transport cause.

| Scenario | Actual outcome and stage | Evidence-supported cause or limitation | Remaining correction |
| --- | --- | --- | --- |
| cases.browse-cases | FAIL; actual native succeeded | Correct C1/C2 plaintext results and excluded done C3, but the rendered reply had no usable case links. A separate UI propagation defect is being addressed; this result remains failed. | 0 |
| agents.managed-profile | FAIL; canonical query sent, native still running at phase deadline | Empty reply; no terminal error observed. No provider cause established. | 0 |
| budgets-costs.spend-breakdown | FAIL; main native succeeded | Reply reported repeated basic MCP request timeouts and did not supply the actual agent breakdown/incomplete-total outcome. Retained handler results exist; the delivery boundary/cause is unresolved. Warmup did not show the same markers. | 0 |
| budgets-costs.resolve-budget | FAIL; canonical query sent, native succeeded | A real bound budget question with a $10 choice was visible despite empty prose. The fixture used a separate board incident control without answering that question, so its intended question/response-wake continuation was not verified. Original final remains failed/exhausted. | 0 |

## Four additional completed outcomes

| Scenario | Original outcome / actual native evidence | Evidence-supported limitation | Remaining correction |
| --- | --- | --- | --- |
| companies.archive-company | FAILED initial; one actual started native, still running at phase deadline and cancelled during cleanup | Generic identity-response wait did not obtain the required archive/restore response; no terminal provider cause established. | 1 |
| cases.create-case | FAILED final; three recorded run rows, only two have actual start timestamps | Declared run-bound guard stopped the case. The unstarted queued row is not a third native execution. No partial result is promoted to pass. | 0 |
| companies.agent-observability | FAILED initial; two actual native runs succeeded | Required real native harness trace export was not observed within the wait. The absence of the export is retained without an inferred provider cause. | 1 |
| cases.cases-disabled | FAILED final; one native remained running at the frozen outer deadline, then cleanup cancelled it | Required truthful availability/choice did not settle; one request-timeout marker was observed. That isolated symptom is not proof of a shared transport cause. | 0 |

The first eight originals retain FAILED status, cleanup passed, and manifests with no leaks or missing files. Cases run-row limits and actual native executions are distinct evidence. No timeout, queued row or partial semantic success is silently regraded.

## Four additional completed outcomes and model-policy boundary

| Scenario | Preserved original outcome | Evidence-supported interpretation | Remaining correction |
| --- | --- | --- | --- |
| companies.prompt-injection-guard | PASS 10/10; authenticated native/build/source and cleanup | Human settings configured the classifier; both actual fixed samples returned the expected verdict and recorded charges, while chat respected board-only management. **Mixed-model evidence:** chat used requested DeepSeek1017, but two classifier calls used `openai/gpt-oss-safeguard-20b`. Not qualifying uniform1017 credit. | One explicitly reviewed configuration correction may be admitted later; no raw result regrade. |
| companies.import-preview | FAILED initial | Required conflict preview, explicit human rename/import continuation and scoped import audit were not established. No unretained provider cause inferred. | 1 |
| companies.export-company | FAILED initial | Supported package, actual authorized download and nonportable-credential explanation did not satisfy the full outcome. | 1 |
| tasks.delegate-announcement | FAILED final; exhausted | Sole started native succeeded; a prior queued assignment row was cancelled before starting when the task was reassigned to Morgan. The legacy observer rejects cancelled rows and stopped before worker admission. This is not proof of a failed worker native or of the complete returned announcement. | 0 |

The mixed-model human request was **“Turn on the extra protection for suspicious app results and check that it works.”** Its successful settings/classifier result is disclosed as actual product behavior, while its auxiliary model configuration prevents uniform-model qualification. The product exposes a real editable Model input; any future correction must save the exact requested service model before actual tests, preserve their real verdict/security/UI proof, and receive its own explicit finite manifest accounting. Defaults and original measurement remain unchanged.

## Four further completed outcomes

| Scenario | Original outcome | Observed proof or limitation | Remaining correction |
| --- | --- | --- | --- |
| documents-artifacts.produce-download | PASS 7/7, qualified requested-model evidence | A real product-registered `website.html` download matched the produced source bytes, with source task and exact authenticated query-attachment identity retained. Actual source/build/native/cell-restore and cleanup/privacy verified. | No unchanged rerun requested. |
| documents-artifacts.revise-delivery | FAILED initial | Actual saved-source preservation, exact revised download and retrievable original did not satisfy the independent oracle, although revision-context wording passed. Cause remains under review; no result rescue from prose alone. | 1 |
| goals.explain-purpose | FAILED initial | Initial purpose/chain/rationale prerequisite was not accepted; later missing-association/private/read-only checks were not established. Full frozen journey was not completed. | 1 |
| execution-environments.choose-capable-target | FAILED initial, setup before native execution | Public environment creation returned 422; zero native runs. Retained body was withheld. Exact schema/fixture persistence boundary is under bounded review; no unsupported-feature conclusion from status alone. | 1 |

All four have cleanup passed, no evidence-manifest leaks or missing files, and no diagnostic privacy sidecar. The current 16-result snapshot contains two raw passes and fourteen original failures. Future fixture/oracle changes and unstarted queued cells are not new measured successes.

## Current baseline coverage

Across the five-case pilot, twelve initial journeys, and these sixteen current-run measurements: **33 distinct scenarios measured: 5 qualifying passes, 27 failures, 1 mixed-model pass, 211 not measured**, totaling 244. There are 33 actual measurements on this refreshed baseline. Raw outcomes are 6 PASS / 27 FAIL; only 5 meet the requested uniform-model threshold. Historical production/model lanes and the separate failed instrumented diagnostic remain separate.

Current qualifying passing scenarios are `agents.change-manager`, `projects.archive-project`, `documents-artifacts.write-plan`, `documents-artifacts.resolve-feedback`, and `documents-artifacts.produce-download`. A pass represents its actual tested end-to-end outcome, not coverage for other unrun scenarios. The previous official [twelve-case report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40624431756-1-full12-report/) retains the complete preceding measured outcomes.

Unmeasured includes the 24 cells of this manifest without completed imported evidence plus other unrun journeys, explicit invitation/core dependencies, external prerequisites, and incomplete or blocked public paths. None is counted as a pass or manufactured failure. No original measurement or timeout is regraded by this summary.

## Separate diagnostic evidence

The instrumented artifact-search initial in [40635108320](https://github.com/paperclipai/paperclip/actions/runs/40635108320) remains FAILED, with clean cleanup/privacy. Its retained timing showed 15 successful Paperclip MCP tool completions and no request timeout in that sample. This does not establish universal health or explain earlier failures. Its source/native build differs from clean production39e, so it supplies no clean-production pass credit. A narrow returned-copy observer/oracle correction is under independent review; no regrade or final retry has occurred.