Full-stack acceptance campaign

Runner Full-Stack E2E

Results across runner profiles and execution environments. Declared PNG screenshots and normalized results are retained with every published campaign; additional diagnostic evidence remains in the access-controlled workflow artifact.

2026-10-28T02:37:55.300Z33f99c6cSource run ↗Campaign summary ↗

1/2Passed
1Failed
3m 32sSummed attempt time
Campaign notes and coverage
# Current merged-custody baseline: two genuine human-chat controls

This is a partial current-production campaign on production `39e811c9c5226979fce42e052a27cfb28301018a` (including merged credential-custody PR17949), evaluated from source `33f99c6c5a7d6faae19cd8e47afd0d2f78882967`. It uses Paperclip Runner OpenCode with OpenRouter model `openrouter/deepseek/deepseek-v4.1-flash-1017` on actual AWS on-demand hosts with embedded PostgreSQL.

**Current baseline: 2 measured scenarios, 1 qualified end-to-end pass, 1 failed, 242 not measured on this baseline; denominator 244.** These two are explicitly requested comparisons of formerly passing controls, not two unused initial scenarios or newly granted correction budgets. Three unused project initials are selected in this same run; their pending jobs are not measurements. No prior pass is carried into these counts.

[Actual Actions run 40620786554](https://github.com/paperclipai/paperclip/actions/runs/40620786554). Source is frozen; automatic retries are zero. Both controls sent the actual canonical human query once and used one actual native run each. Both cleanup and packaged evidence privacy checks passed. Usage is metered but unpriced in the retained receipts; unknown cost is not zero cost.

| Human journey | Actual result | What happened |
| --- | --- | --- |
| **agents.change-manager** — “Have Morgan report to Riley from now on.” | **PASS, 9/9** | Actual saved manager relationship and org-view checks passed, with task/permission preservation. Native execution succeeded. Exact frozen definition replay and authenticated same-run source/build/native restore proof qualify this as current-baseline evidence. |
| **cli-operations.create-from-cli** — “Give me the CLI command to create a website task for Morgan in Launch.” | **FAIL** | Native execution succeeded and the returned supported CLI command created exactly one correctly scoped Morgan/Launch task as TODO. Later `recovery.reconcile_stranded_assigned_issue` moved the manually paused worker's task to BLOCKED. Two final-status equality assertions failed. This is a localized lifecycle/status assertion mismatch; the original failed result is preserved, not relabeled as a pass or provider outage. |

The runtime uses the product's saved `timeoutSec: 0` policy, with no artificial active-turn deadline. Original outer attempt, run-count, human-turn, spend, privacy and cleanup bounds remain in force. The legacy authored 120-second descriptor is retained as historical metadata, not represented as the effective native deadline.

## Still pending in this same pilot

- **projects.create-project** — “Create a project called Launch for the new website.” Actual project creation and duplicate-safe retry must be verified.
- **projects.project-work** — “What is left in Launch, and create a task for Morgan to fix the mobile layout there.” Actual unfinished-work summary and one correctly assigned task must be verified; a completed website is not required.
- **projects.archive-project** — “Archive the Launch project but keep its history.” Actual archive state and retained readable history must be verified.

These have no outcome credit until authenticated actual results arrive. The other 239 catalog scenarios have no new-baseline measurement in this report; historical failures and explicit external/core/public-path blockers are not silently reset or declared runnable.

## Preserved earlier evidence

[Previous OpenCode campaign and full 244-scenario coverage disposition](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40584967834-1-final34-report/) records **57 distinct measured scenarios, 6 passes, 51 failures, 187 unrun and 62 attempts** on the older production baseline. Those measurements and the three credential-scan failures remain unchanged. That run was intentionally cancelled after shared key-custody findings, not fleet queuing. Its 6 passes are historical evidence, not current-baseline credit.

[Historical local Codex partial campaign](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/historical-local-codex-chat-catalog-20261027-158measured-86unrun/) separately preserves 17 historical passes, 141 failures and 86 unrun. Six catalog journeys remain incomplete or blocked by a missing public path; implementation coverage alone is not proof of successful end-to-end behavior.
Campaign usage and spend
0 / 0Complaints / suggestions · retained attempts0 distinct items · 2 attempts with incomplete collection · 0 failed submissions · 0 unresolved submissions
2Retained attempts0 missing attempts · 0 out-of-scope artifacts
190,052Input tokens
4,110Output tokens
480,000Cached tokens
UnavailableLLM reported subtotal
$0.000000Daytona list estimate
2m 16sObserved attempt window2 retained attempts · 0 missing attempts

Attempt durations include setup and cleanup captured inside each attempt. The observed window spans the earliest retained start to the latest retained finish, including gaps and overlap. Carried-forward evidence can span older runs, so this window is not the current workflow's duration. Neither measure is CI queue/setup time or provider-billed runtime.

1m 45sAgent execution time
0msDaytona lease time
0/2Runs provider-priced

Campaign costs include retained attempts, including failed retries. Result cards show the selected attempt. Provider-reported amounts and model estimates are separate; unpriced or unavailable runs are excluded, never counted as free. Coverage describes captured usage; unobserved setup, subscriptions, tools, CI and storage require additional accounting. Daytona runtime is a public-list-price estimate from captured lease time and pinned resources, before credits, discounts, storage allowance, or invoice adjustments. Local execution has no external runtime meter.

UnknownWorker setup time
UnknownWorker execution time
UnknownWorker feedback time
UnknownWorker cleanup time
UnknownWorker finalization time

0/2 complete worker receipts; 0 partial; 0 missing attempts. Worker phases cover the test callback through finalization before result publication. Browser/server startup, parent cleanup, CI queue/build time and provider-billed runtime are outside these measurements. An interrupted phase has no inferred end time.

Campaign feedback review

0 distinct feedback items await review. Collection gaps and failed submissions remain separate findings.

Full bodies and review rationale stay in private evidence. Exact duplicate bodies are grouped once; related causes can be linked by a reviewed duplicate disposition.

All results

Feature coverage 0 of 192 capabilities linked · 0 of 2 tests mapped

Associations use the current feature map. Counts describe linked test scenarios; a passing journey does not establish full coverage of a capability.

Getting started and access

FeatureCapabilities linkedLinked test results
Onboarding and first work 0/5 No mapped journey
Login, invitations, and access 0/5 No mapped journey
Companies and portability 0/8 No mapped journey

Tasks and conversations

FeatureCapabilities linkedLinked test results
Task creation and lifecycle 0/6 No mapped journey
Delegation, dependencies, and signoff 0/4 No mapped journey
Inbox, decisions, and search 0/5 No mapped journey
Agent conversations and project handoff 0/4 No mapped journey
Questions and approvals 0/8 No mapped journey
Steering and queued messages 0/7 No mapped journey
Documents, attachments, and work products 0/5 No mapped journey

Agents and reusable capabilities

FeatureCapabilities linkedLinked test results
Hiring, configuration, and organization 0/9 No mapped journey
Runs, harnesses, and model accounts 0/5 No mapped journey
Skills and Skill Studio 0/5 No mapped journey
Team packages and installation 0/4 No mapped journey

Projects and execution

FeatureCapabilities linkedLinked test results
Projects and repositories 0/5 No mapped journey
Goals and work alignment 0/3 No mapped journey
Workspaces, services, and files 0/5 No mapped journey
Execution environments 0/5 No mapped journey
Recovery 0/8 No mapped journey

Apps, channels, and extensions

FeatureCapabilities linkedLinked test results
Connection setup 0/8 No mapped journey
Assistant connections (MCP) 0/5 No mapped journey
App access and action permissions 0/5 No mapped journey
External chat and email 0/6 No mapped journey
Tool gateways and access profiles 0/5 No mapped journey
Secrets and proposals 0/5 No mapped journey
Plugins 0/4 No mapped journey

Automation and structured work

FeatureCapabilities linkedLinked test results
Routines, schedules, and triggers 0/5 No mapped journey
Cases 0/5 No mapped journey
Status cards 0/4 No mapped journey

Oversight and operation

FeatureCapabilities linkedLinked test results
Costs and budgets 0/6 No mapped journey
Dashboards and audit trails 0/6 No mapped journey
Navigation, profile, and announcements 0/6 No mapped journey
Instance operations 0/6 No mapped journey
CLI/API and local worktrees 0/6 No mapped journey

Contributor surfaces

FeatureCapabilities linkedLinked test results
Developer previews and diagnostic labs 0/4 No mapped journey

Test suite

Chat feature journeys

Executable catalog requests through Agent Chat, with independently graded persisted state.

Pass rate50.0%1/2 passed
Tokens674,162190,052 input · 4,110 output
Known spendUnavailablereported LLM + model/runtime estimates estimate
Agent time1m 45s0ms lease
Execution2/20 retries · cleanup passed
Native OpenCode OpenRouter DeepSeek V4.1 Flash 1017Isolated localopencode · openrouter/deepseek/deepseek-v4.1-flash-1017
Test details9/9 checks · 1m 37s
Overall passed · 9/9 behavioral checks passed

No behavioral matcher failed.

See all behavioral checks (9)
ResultCheckDetail
Passchat.complete-runtime-observationSaved agent configuration, attribution and actual rendered replies are independently required.
Passchat.saved-requested-managerMorgan reports to the real same-company Riley.
Passchat.reporting-tree-validEvery reporting chain terminates without a cycle or missing company manager.
Passchat.unrelated-config-and-tasks-retainedChanging the manager preserves accounts, permissions and existing tasks.
Passchat.attributed-agent-changeThe saved reporting-line change is independently attributed to the actual agent.
Passchat.catalog-complete-conditionsNo known catalog condition is omitted.
Passchat.catalog-original-human-queryThe persisted request preserves the catalog query with only audited public-fixture identity bindings; all human turns were declared.
Passchat.catalog-query-attachment-receiptsOnly declared production attachment suffixes are allowed, backed by same-source human receipts and independently retrieved matching bytes.
Passchat.catalog-bounded-native-runsObserved provider runs satisfy declared count, company, native runtime and allowed outcome bounds.
Tokens63,862 in · 753 out136,448 cached · 1/1 runs covered
LLM spendUnavailable0/1 runs provider-priced
ExecutionLocal · not metered36s agent

Worker phase timing: not captured.

Feedback was not captured for this attempt.

chat-features.runner-opencode.local.agents.change-manager
Attempt
1
Duration
1m 37s
Agent runtime
36s
Runtime
native
Provider
opencode
Model
openrouter/deepseek/deepseek-v4.1-flash-1017

All invariants passed

Startup reuse evidence
RunWarmthEnvironment/leaseWorkspaceRunnerProvider processProvider sessionProvider launches
4dc17692-b6ec-4206-9efc-72a15f28e97dunknown unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing) unknown
ResultMatcherExpectationDetail
Pass json_path {"kind":"json_path","path":"chat.complete-runtime-observation","expected":true} Saved agent configuration, attribution and actual rendered replies are independently required.
Pass json_path {"kind":"json_path","path":"chat.saved-requested-manager","expected":true} Morgan reports to the real same-company Riley.
Pass json_path {"kind":"json_path","path":"chat.reporting-tree-valid","expected":true} Every reporting chain terminates without a cycle or missing company manager.
Pass json_path {"kind":"json_path","path":"chat.unrelated-config-and-tasks-retained","expected":true} Changing the manager preserves accounts, permissions and existing tasks.
Pass json_path {"kind":"json_path","path":"chat.attributed-agent-change","expected":true} The saved reporting-line change is independently attributed to the actual agent.
Pass json_path {"kind":"json_path","path":"chat.catalog-complete-conditions","expected":true} No known catalog condition is omitted.
Pass json_path {"kind":"json_path","path":"chat.catalog-original-human-query","expected":true} The persisted request preserves the catalog query with only audited public-fixture identity bindings; all human turns were declared.
Pass json_path {"kind":"json_path","path":"chat.catalog-query-attachment-receipts","expected":true} Only declared production attachment suffixes are allowed, backed by same-source human receipts and independently retrieved matching bytes.
Pass json_path {"kind":"json_path","path":"chat.catalog-bounded-native-runs","expected":true} Observed provider runs satisfy declared count, company, native runtime and allowed outcome bounds.
Usage, billing and timing metadata
{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 1,
      "runsWithReportedCost": 0,
      "runsWithEstimatedCost": 0,
      "runsWithPartialReportedCost": 0,
      "inputTokens": 63862,
      "outputTokens": 753,
      "cachedInputTokens": 136448,
      "totalTokens": 201063,
      "reportedCostUsd": 0,
      "estimatedCostUsd": 0,
      "costStatus": "unpriced"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 35779,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null,
  "workerTiming": null
}
Native OpenCode OpenRouter DeepSeek V4.1 Flash 1017Isolated localopencode · openrouter/deepseek/deepseek-v4.1-flash-1017
Overall failed · No behavioral checks recorded

candidate failure

Stopped waiting for cli-operations.create-from-cli: Settled scenario failed: saved-company-project-assignee, real-command-structured-result
Test detailsNo checks recorded · 1m 55s
Tokens126,190 in · 3,357 out343,552 cached · 1/1 runs covered
LLM spendUnavailable0/1 runs provider-priced
ExecutionLocal · not metered1m 9s agent

Worker phase timing: not captured.

Feedback was not captured for this attempt.

chat-features.runner-opencode.local.cli-operations.create-from-cli
Attempt
1
Duration
1m 55s
Agent runtime
1m 9s
Runtime
native
Provider
opencode
Model
openrouter/deepseek/deepseek-v4.1-flash-1017

No result artifact was uploaded

Startup reuse evidence
RunWarmthEnvironment/leaseWorkspaceRunnerProvider processProvider sessionProvider launches
40076454-f697-4b5f-b0a7-45b37846fdd3unknown unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing) unknown

No matcher result was recorded.

Usage, billing and timing metadata
{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 1,
      "runsWithReportedCost": 0,
      "runsWithEstimatedCost": 0,
      "runsWithPartialReportedCost": 0,
      "inputTokens": 126190,
      "outputTokens": 3357,
      "cachedInputTokens": 343552,
      "totalTokens": 473099,
      "reportedCostUsd": 0,
      "estimatedCostUsd": 0,
      "costStatus": "unpriced"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 69047,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null,
  "workerTiming": null
}