# Current production chat evaluation: completed eight-case snapshot

This is a partial snapshot of run [40630370749](https://github.com/paperclipai/paperclip/actions/runs/40630370749), source `81bba1dd0d4f77448bf4c5e360f3c78760e34ce2`, production baseline `39e811c9c5226979fce42e052a27cfb28301018a`. Exact configured model: `openrouter/deepseek/deepseek-v4.1-flash-1017` through OpenCode/OpenRouter.

## What actually works on the current production baseline

| Actual human request | Observed end-to-end state proof | Existing evidence page |
| --- | --- | --- |
| Have Morgan report to Riley from now on. | Saved manager change and organization view; relevant assignment and company invariants retained, 9/9 checks. | [Pilot](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40620786554-1-five-report/) |
| Archive the Launch project but keep its history. | Actual archived project with readable retained history, 10/10 checks. | [Pilot](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40620786554-1-five-report/) |
| Save a short implementation plan on the website task. | Versioned plan saved on the intended task, actual document link and substantive brief facts retained, 9/9 checks. | [Twelve-case report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40624431756-1-full12-report/) |
| Fix the confusing pricing paragraph and resolve my comment. | Actual anchored document edit and correct annotation resolution, unrelated feedback preserved, 9/9 checks. | [Twelve-case report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40624431756-1-full12-report/) |

These four are measured passes on production39e. [The full 244-scenario coverage inventory and prerequisite lists](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40584967834-1-final34-report/) remain inspectable in the preceding historical report. Its old outcomes are historical, not extra current-baseline passes. Current measured outcomes and counts are below.

## First four final-correction outcomes

The first four selected final corrections all failed. Their original outcomes and prior attempts remain unchanged; these four final allowances are exhausted. All four completed cleanup and have evidence manifests with no leaks or missing files. At this snapshot eight selected cells are measured and the remaining 32 await completed evidence; bounded normal waves have resumed. Queued jobs consume no scenario attempt. This snapshot does not establish a shared provider outage or transport cause.

| Scenario | Actual outcome and stage | Evidence-supported cause or limitation | Remaining correction |
| --- | --- | --- | --- |
| cases.browse-cases | FAIL; actual native succeeded | Correct C1/C2 plaintext results and excluded done C3, but the rendered reply had no usable case links. A separate UI propagation defect is being addressed; this result remains failed. | 0 |
| agents.managed-profile | FAIL; canonical query sent, native still running at phase deadline | Empty reply; no terminal error observed. No provider cause established. | 0 |
| budgets-costs.spend-breakdown | FAIL; main native succeeded | Reply reported repeated basic MCP request timeouts and did not supply the actual agent breakdown/incomplete-total outcome. Retained handler results exist; the delivery boundary/cause is unresolved. Warmup did not show the same markers. | 0 |
| budgets-costs.resolve-budget | FAIL; canonical query sent, native succeeded | A real bound budget question with a $10 choice was visible despite empty prose. The fixture used a separate board incident control without answering that question, so its intended question/response-wake continuation was not verified. Original final remains failed/exhausted. | 0 |

## Four additional completed outcomes

| Scenario | Original outcome / actual native evidence | Evidence-supported limitation | Remaining correction |
| --- | --- | --- | --- |
| companies.archive-company | FAILED initial; one actual started native, still running at phase deadline and cancelled during cleanup | Generic identity-response wait did not obtain the required archive/restore response; no terminal provider cause established. | 1 |
| cases.create-case | FAILED final; three recorded run rows, only two have actual start timestamps | Declared run-bound guard stopped the case. The unstarted queued row is not a third native execution. No partial result is promoted to pass. | 0 |
| companies.agent-observability | FAILED initial; two actual native runs succeeded | Required real native harness trace export was not observed within the wait. The absence of the export is retained without an inferred provider cause. | 1 |
| cases.cases-disabled | FAILED final; one native remained running at the frozen outer deadline, then cleanup cancelled it | Required truthful availability/choice did not settle; one request-timeout marker was observed. That isolated symptom is not proof of a shared transport cause. | 0 |

All eight originals retain FAILED status, cleanup passed, and manifests with no leaks or missing files. Cases run-row limits and actual native executions are distinct evidence. No timeout, queued row or partial semantic success is silently regraded.

## Current baseline coverage

Across the five-case pilot, twelve initial journeys, and these eight current-run measurements: **25 distinct scenarios measured: 4 passed, 21 failed, 219 not measured on this production baseline**, totaling 244. There are 25 actual measurements on this refreshed baseline. Historical measurements and passes from earlier production/model lanes remain separate. The separate one-case instrumented artifact-search diagnostic is not included in these clean-production counts.

Current passing scenarios are `agents.change-manager`, `projects.archive-project`, `documents-artifacts.write-plan`, and `documents-artifacts.resolve-feedback`. A pass represents its actual tested end-to-end outcome, not coverage for other unrun scenarios. The previous official [twelve-case report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40624431756-1-full12-report/) retains the complete preceding measured outcomes.

Unmeasured includes the 32 cells of this manifest without completed imported evidence plus other unrun journeys, explicit invitation/core dependencies, external prerequisites, and incomplete or blocked public paths. None is counted as a pass or manufactured failure. No original measurement or timeout is regraded by this summary.

## Separate diagnostic evidence

The instrumented artifact-search initial in [40635108320](https://github.com/paperclipai/paperclip/actions/runs/40635108320) remains FAILED, with clean cleanup/privacy. Its retained timing showed 15 successful Paperclip MCP tool completions and no request timeout in that sample. This does not establish universal health or explain earlier failures. Its source/native build differs from clean production39e, so it supplies no clean-production pass credit. A narrow returned-copy observer/oracle correction is under independent review; no regrade or final retry has occurred.