Full-stack acceptance campaign

Runner Full-Stack E2E

Results across runner profiles and execution environments. Declared PNG screenshots and normalized results are retained with every published campaign; additional diagnostic evidence remains in the access-controlled workflow artifact.

2026-10-27T20:27:41.326Z1c72e2d7Source run ↗Campaign summary ↗

2/8Passed
6Failed
13m 35sSummed attempt time
Campaign notes and coverage
# Current remote qualification and historical results

## Current OpenCode catalog: 2 passed scenarios, 7 failed scenarios, 235 unrun

The denominator remains **244 scenarios / 35 areas / 191 frozen capabilities**. Nine unique scenarios have actual remote measurements: **2 passed, 7 failed**. Ten attempts were retained: **2 passed attempts, 8 failed attempts**. Backlog's final correction is a second attempt on the same scenario, never an additional scenario. Historical local passes remain separate. Authored callbacks and the235 unrun scenarios are not passing E2E evidence.

## Latest eight-case AWS wave: 2 passed, 6 failed

[Actual run40575715287](https://github.com/paperclipai/paperclip/actions/runs/40575715287) used exact source `1c72e2d74e77f4761d64a5bf571684fa6ccb2311`, production baseline `50b932a142216331b8d476042b454bc123739c40`, OpenCode/OpenRouter `openrouter/deepseek/deepseek-v4.1-flash-1017`, embedded PostgreSQL and separate verified AWS fleet hosts. Maximum concurrency4; automatic retries0; original queries, bounds and independent oracles unchanged. All eight cleanups passed. Unpriced dollars and automatic hello usage remain unknown, not zero.

**Actually passed on the current remote lane:**

- `agents.change-manager`:9/9 recorded checks, one native run; actual saved reporting line, attribution and preserved unrelated state.
- `companies.rename-company`:10/10 recorded checks, one native run; same company renamed, memberships/work preserved and actual actor attribution.

These pass measurements are bound to authenticated build artifact12767846870 (ZIP SHA256 `8498afb301f5a1292a354900f9a503dfafc0542f2c0632727d21d9fa3876096f`), retained native binary bytes and actual source/checksum-restore receipts; production paths remain byte-identical to50b. They do not qualify all other features or the historical baseline.

**Failed in this wave:**

- `tasks.create-backlog`: final allowed correction failed; creating a todo/self-assigned task queued unwanted work and exceeded the declared run bound. One source execution and a queued row are not two completed native executions. Its original setup failure remains retained; the original unknown hello cause is not retroactively called401. No further unchanged retry.
- `agents.hire-denied`: setup failed before canonical input,0 runs; public permissions fixture lacked required boolean `canAssignTasks`. No model denial outcome was measured.
- `tasks.comment-with-reference`: settled observation failed `one-attributed-note`; no passing note outcome claimed.
- `inbox-search.archive-without-grant`: an observed provider run failed; original failure classification and evidence retained.
- `inbox-search.show-my-work`: an observed provider run failed; original failure classification and evidence retained.
- `tasks.ambiguous-task`: timed out waiting for actual clarification; no invented clarification or passing choice outcome.

This standard presentation uses original result bytes through a closed public projection. Raw transcripts/logs, private account data and credentials are not published. The report's2/8 selection rate is distinct from the whole catalog's2/244 current scenario rate.

## Delegation pilot — actual canonical request, overall FAILED

[Run40572484517](https://github.com/paperclipai/paperclip/actions/runs/40572484517), initial ordinal1, source `2f68f4ff3c4be3ff608823e556cc000eb6948a06`, production `50b932a142216331b8d476042b454bc123739c40`, exact OpenCode/OpenRouter `openrouter/deepseek/deepseek-v4.1-flash-1017`, ordinary embedded PostgreSQL on a verified AWS fleet host. The original result remains FAILED (`transient_infrastructure`), with an observed CEO native-session timeout at the unchanged120-second bound. No evidence-backed correction or unchanged retry is claimed.

Exactly one canonical human request was persisted. The CEO created the requested task; a later mutation assigned Morgan. Six CEO tool executions completed, including a document write, but no terminal protocol frame arrived before the timer. Morgan never admitted. The second retained run row stayed queued and was cancelled during cleanup; it is not a second native execution. The announcement, worker provenance, completion reply and usable saved-result link were not proven.

The finally observation recorded **4/14 checks passing** (one human request, one assigned task, captured request, agent-created task). This is partial creation evidence, not a returned completed feature grade or a passing scenario. Actual aggregate usage was **165536 tokens**; unpriced dollars and automatic hello usage remain unknown, not zero. Cleanup passed. Actual framework interval19:58:41.606–20:01:18.853UTC. Artifact12766536663 SHA256 `2785814c8483b0f47218f99dcac2dc4b86b41eabb73cdb72cf98c65cad19c523`; raw transcripts/logs and private identifiers are excluded from this public presentation.

## Latest repaired-credential diagnostic — overall FAILED, outside the244

[Run40561571573 attempt2](https://github.com/paperclipai/paperclip/actions/runs/40561571573/attempts/2) is the authorized same-run rerun after the exact CI environment credential was replaced with direct human approval. It reused the original successful same-run build at source `2f68f4ff3c4be3ff608823e556cc000eb6948a06`, production baseline50b932 and unchanged OpenCode/OpenRouter `openrouter/deepseek/deepseek-v4.1-flash-1017`. It ran on actual AWS On-Demand/x86_64 with ordinary embedded PostgreSQL. No global fleet setting, product assertion, model or source changed for the rerun.

**The overall result remains FAILED:** `permanent_infrastructure`, **“Required evidence missing: final-state.png, snapshots/api-state.json.”** The evidence packaging failure is not relabeled as a pass. No more health or feature operation is implied by this report.

Independent retained receipts prove the single current-key request returned **HTTP200**, `recognized: true`, `expired: false`, key-budget availability true (no amount retained). Exactly **one normal lifecycle hello passed**, the agent was ready and its config matched. All three setup-health checks passed: ordinary readiness, one ordinary hello, and zero chat/task work. The real API reported0 issues,0 conversations,0 comments and0 heartbeat runs. Cleanup passed. Key-budget availability does not prove account credit or all model access; automatic hello tokens/cost are unknown outside heartbeat accounting.

This functional setup proof is outside244, not a feature scenario pass or a new scenario ordinal. Original attempt1 [HTTP401 report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40561571573-1-auth-report/) and artifact12763471080 remain unchanged and separate. Attempt2 authenticated artifact12764569482 SHA256 `368659db7e0bb016914e98db997919890be40c347011b3fc138af3c497cf95e8`. No credential values, account identities, provider response bodies, raw logs or screenshots are published.

The prior [embedded health report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40553618239-1-report-r2/) preserves source78de/run40553618239 and the original “User not found.” hello failure. These diagnostic operations remain distinct from the two failed catalog initials and from historical local passes.

## Historical local Codex:17 original initial passes

These are original local one-time measurements of provider `codex`, model `gpt-5.6-sol`, production baseline8373 with original per-attempt source/definition identities. They are **not current OpenCode or updated-production qualification**. Across158 attempted IDs:17 initial passes,141 initial failures,159 retained measurements including one historical correction, and86 unrun. [Immutable historical report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/historical-local-codex-chat-catalog-20261027-158measured-86unrun/) and [original summary](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/historical-local-codex-chat-catalog-20261027-158measured-86unrun/summary.md). Unpriced cost is unknown, not zero.

- access.private-resource
- agents.agent-api-key
- assistant-connections.decline-consent
- chat-channels.provider-disabled
- companies.archive-company
- companies.rename-company
- inbox-search.archive-partial-failure
- instance-operations.instance-settings
- projects.project-work
- questions-and-approvals.accept-one-proposal
- questions-and-approvals.ambiguous-yes
- runs-adapters.list-model-options
- steering.preserve-draft
- task-coordination.reject-dependency-cycle
- tasks.create-backlog
- tasks.delegate-announcement
- tasks.website-task

## Six incomplete or blocked journeys

These remain incomplete/blocked despite registration and prerequisite probes; no passing E2E coverage is claimed.

- access.invite-teammate
- access.manage-membership
- activity.routine-audit
- agents.restore-instructions
- budgets-costs.subscription-fees
- skills.enable-for-agent

## All244 current remote statuses

### Passed current remote scenarios (2)

- agents.change-manager
- companies.rename-company

### Failed current remote scenarios (7)

- tasks.create-backlog (two retained failed attempts; correction budget exhausted)
- tasks.delegate-announcement
- agents.hire-denied (setup before canonical)
- tasks.comment-with-reference
- tasks.ambiguous-task
- inbox-search.show-my-work
- inbox-search.archive-without-grant

### Unrun current remote scenarios (235), grouped by area

#### access (6)

- access.cli-authorization
- access.invite-teammate
- access.manage-membership
- access.other-company
- access.private-resource
- access.sign-in

#### activity (6)

- activity.agent-feedback
- activity.audit-change
- activity.company-overview
- activity.find-runs
- activity.routine-audit
- activity.work-timeline

#### agent-chat (6)

- agent-chat.chat-disabled
- agent-chat.conference-room
- agent-chat.multiple-handoffs
- agent-chat.remember-and-follow-up
- agent-chat.reopen-conversation
- agent-chat.track-delegated-work

#### agents (14)

- agents.agent-api-key
- agents.ambiguous-cloud-code
- agents.choose-primary
- agents.configure-required-agent
- agents.edit-instructions
- agents.existing-grok-bot
- agents.hire-claude-code
- agents.hire-needs-account
- agents.hire-needs-approval
- agents.managed-profile
- agents.pause-agent
- agents.restore-instructions
- agents.resume-agent
- agents.retire-agent

#### app-permissions (6)

- app-permissions.ask-before-action
- app-permissions.grant-one-agent
- app-permissions.new-quarantined-action
- app-permissions.read-allowed-write-off
- app-permissions.revoke-account
- app-permissions.test-as-agent

#### assistant-connections (6)

- assistant-connections.connect-external-assistant
- assistant-connections.consent-read-only
- assistant-connections.decline-consent
- assistant-connections.device-login
- assistant-connections.revoke-personal-grant
- assistant-connections.use-scoped-client

#### budgets-costs (6)

- budgets-costs.explain-hard-stop
- budgets-costs.repair-cost-recording
- budgets-costs.resolve-budget
- budgets-costs.set-budget
- budgets-costs.spend-breakdown
- budgets-costs.subscription-fees

#### cases (8)

- cases.browse-cases
- cases.case-history
- cases.cases-disabled
- cases.close-case
- cases.create-case
- cases.handoff-same-task
- cases.related-work
- cases.update-case-facts

#### chat-channels (8)

- chat-channels.channel-file
- chat-channels.connect-slack
- chat-channels.link-sender
- chat-channels.outbound-email
- chat-channels.provider-disabled
- chat-channels.repair-endpoint
- chat-channels.reply-in-thread
- chat-channels.reply-to-email

#### cli-operations (6)

- cli-operations.choose-cli-context
- cli-operations.create-from-cli
- cli-operations.export-document-cli
- cli-operations.feedback-bundle
- cli-operations.inspect-run-cli
- cli-operations.owned-dev-instance

#### companies (8)

- companies.agent-observability
- companies.archive-company
- companies.configure-decision-model
- companies.export-company
- companies.fast-responses
- companies.import-preview
- companies.prompt-injection-guard
- companies.switch-company

#### connection-setup (13)

- connection-setup.channel-prerequisites
- connection-setup.choose-provider
- connection-setup.notion-agent-unassigned
- connection-setup.notion-connected
- connection-setup.notion-declined
- connection-setup.notion-expired
- connection-setup.notion-missing
- connection-setup.notion-wrong-account
- connection-setup.return-from-login
- connection-setup.save-and-resume
- connection-setup.secure-api-key
- connection-setup.unreachable-mcp
- connection-setup.wrong-person-resolves

#### developer-labs (4)

- developer-labs.component-reference
- developer-labs.gated-lab
- developer-labs.interaction-fixture
- developer-labs.long-thread-performance

#### documents-artifacts (7)

- documents-artifacts.find-artifact
- documents-artifacts.private-download
- documents-artifacts.produce-download
- documents-artifacts.resolve-feedback
- documents-artifacts.restore-document
- documents-artifacts.revise-delivery
- documents-artifacts.write-plan

#### execution-environments (5)

- execution-environments.add-computer
- execution-environments.assign-computer
- execution-environments.choose-capable-target
- execution-environments.prepare-image
- execution-environments.test-computer

#### gateways-profiles (5)

- gateways-profiles.connect-client
- gateways-profiles.create-gateway
- gateways-profiles.define-profile
- gateways-profiles.inspect-gateway-error
- gateways-profiles.rotate-token

#### goals (3)

- goals.align-existing-work
- goals.create-goal-tree
- goals.explain-purpose

#### inbox-search (6)

- inbox-search.archive-done
- inbox-search.archive-partial-failure
- inbox-search.find-design
- inbox-search.notification-audience
- inbox-search.restore-archive
- inbox-search.show-decisions

#### instance-operations (6)

- instance-operations.backup-and-restore
- instance-operations.diagnose-health
- instance-operations.instance-settings
- instance-operations.provision-runtime
- instance-operations.restart-service
- instance-operations.update-instance

#### navigation-preferences (6)

- navigation-preferences.broken-link
- navigation-preferences.dismiss-announcement
- navigation-preferences.edit-profile
- navigation-preferences.find-project-view
- navigation-preferences.quick-search
- navigation-preferences.sidebar-order

#### onboarding (5)

- onboarding.choose-first-runtime
- onboarding.first-work
- onboarding.new-company
- onboarding.new-instance
- onboarding.resume-wizard

#### plugins (4)

- plugins.configure-plugin
- plugins.disable-plugin
- plugins.install-plugin
- plugins.use-plugin-tool

#### projects (8)

- projects.add-repository
- projects.archive-project
- projects.create-project
- projects.private-project
- projects.project-defaults
- projects.project-work
- projects.replace-repository
- projects.share-project

#### questions-and-approvals (12)

- questions-and-approvals.accept-one-proposal
- questions-and-approvals.ambiguous-yes
- questions-and-approvals.approve-tool-once
- questions-and-approvals.decline-tool
- questions-and-approvals.defer-and-reopen
- questions-and-approvals.duplicate-answer
- questions-and-approvals.formal-hire-revision
- questions-and-approvals.multi-part-question
- questions-and-approvals.necessary-question
- questions-and-approvals.reject-and-revise
- questions-and-approvals.review-plan
- questions-and-approvals.wrong-reviewer

#### recovery (8)

- recovery.browser-reconnect
- recovery.controller-restart
- recovery.explain-stall
- recovery.post-stop-message
- recovery.retry-exhausted
- recovery.source-action
- recovery.unknown-email-outcome
- recovery.workspace-repair

#### routines (7)

- routines.create-daily-brief
- routines.manage-other-owner
- routines.pause-owned
- routines.plugin-managed-routine
- routines.read-history
- routines.run-now
- routines.webhook-trigger

#### runs-adapters (5)

- runs-adapters.inspect-failed-run
- runs-adapters.list-model-options
- runs-adapters.reset-session
- runs-adapters.upgrade-runner
- runs-adapters.validate-hire

#### secrets (6)

- secrets.ask-for-plaintext
- secrets.configure-company-secret
- secrets.least-scope-delivery
- secrets.personal-secret
- secrets.secret-proposal
- secrets.vault-reference

#### skills (7)

- skills.author-skill
- skills.enable-for-agent
- skills.explicit-restriction
- skills.find-and-install
- skills.fork-read-only
- skills.review-update
- skills.studio-test

#### status-cards (5)

- status-cards.archive-card
- status-cards.create-card
- status-cards.explain-card
- status-cards.other-authors-card
- status-cards.refresh-card

#### steering (8)

- steering.approval-delivery
- steering.discard-queued-message
- steering.edit-queued-message
- steering.interrupt-with-direction
- steering.preserve-draft
- steering.queue-follow-up
- steering.steer-active
- steering.stop-versus-pause

#### task-coordination (5)

- task-coordination.block-on-design
- task-coordination.delegate-with-context
- task-coordination.independent-review
- task-coordination.pause-tree
- task-coordination.reject-dependency-cycle

#### tasks (9)

- tasks.assignment-conflict
- tasks.attachment-brief
- tasks.completion-with-gate
- tasks.edit-properties
- tasks.organize-open-work
- tasks.private-task
- tasks.share-private-task
- tasks.website-task
- tasks.website-without-project

#### teams (4)

- teams.configure-team
- teams.find-team
- teams.install-team
- teams.preview-install

#### workspaces (7)

- workspaces.close-and-reopen
- workspaces.inspect-git
- workspaces.locate-files
- workspaces.prepare-repository
- workspaces.preview-service
- workspaces.private-repository-no-access
- workspaces.recover-lost-sandbox

This report is regenerated only from retained bytes; publication caused zero model/provider calls. Historical immutable prefixes are unchanged. No unmerged core PR contributes a passing result.
Campaign usage and spend
0 / 0Complaints / suggestions · retained attempts0 distinct items · 8 attempts with incomplete collection · 0 failed submissions · 0 unresolved submissions
8Retained attempts0 missing attempts · 0 out-of-scope artifacts
285,371 (partial: 3/9 runs)Input tokens
7,941 (partial: 3/9 runs)Output tokens
1,408,800 (partial: 3/9 runs)Cached tokens
UnavailableLLM reported subtotal
$0.000000Daytona list estimate
8m 0sObserved attempt window8 retained attempts · 0 missing attempts

Attempt durations include setup and cleanup captured inside each attempt. The observed window spans the earliest retained start to the latest retained finish, including gaps and overlap. Carried-forward evidence can span older runs, so this window is not the current workflow's duration. Neither measure is CI queue/setup time or provider-billed runtime.

UnknownAgent execution time
0msDaytona lease time
0/9Runs provider-priced

Campaign costs include retained attempts, including failed retries. Result cards show the selected attempt. Provider-reported amounts and model estimates are separate; unpriced or unavailable runs are excluded, never counted as free. Coverage describes captured usage; unobserved setup, subscriptions, tools, CI and storage require additional accounting. Daytona runtime is a public-list-price estimate from captured lease time and pinned resources, before credits, discounts, storage allowance, or invoice adjustments. Local execution has no external runtime meter.

UnknownWorker setup time
UnknownWorker execution time
UnknownWorker feedback time
UnknownWorker cleanup time
UnknownWorker finalization time

0/8 complete worker receipts; 0 partial; 0 missing attempts. Worker phases cover the test callback through finalization before result publication. Browser/server startup, parent cleanup, CI queue/build time and provider-billed runtime are outside these measurements. An interrupted phase has no inferred end time.

Campaign feedback review

0 distinct feedback items await review. Collection gaps and failed submissions remain separate findings.

Full bodies and review rationale stay in private evidence. Exact duplicate bodies are grouped once; related causes can be linked by a reviewed duplicate disposition.

All results

Feature coverage 0 of 192 capabilities linked · 0 of 8 tests mapped

Associations use the current feature map. Counts describe linked test scenarios; a passing journey does not establish full coverage of a capability.

Getting started and access

FeatureCapabilities linkedLinked test results
Onboarding and first work 0/5 No mapped journey
Login, invitations, and access 0/5 No mapped journey
Companies and portability 0/8 No mapped journey

Tasks and conversations

FeatureCapabilities linkedLinked test results
Task creation and lifecycle 0/6 No mapped journey
Delegation, dependencies, and signoff 0/4 No mapped journey
Inbox, decisions, and search 0/5 No mapped journey
Agent conversations and project handoff 0/4 No mapped journey
Questions and approvals 0/8 No mapped journey
Steering and queued messages 0/7 No mapped journey
Documents, attachments, and work products 0/5 No mapped journey

Agents and reusable capabilities

FeatureCapabilities linkedLinked test results
Hiring, configuration, and organization 0/9 No mapped journey
Runs, harnesses, and model accounts 0/5 No mapped journey
Skills and Skill Studio 0/5 No mapped journey
Team packages and installation 0/4 No mapped journey

Projects and execution

FeatureCapabilities linkedLinked test results
Projects and repositories 0/5 No mapped journey
Goals and work alignment 0/3 No mapped journey
Workspaces, services, and files 0/5 No mapped journey
Execution environments 0/5 No mapped journey
Recovery 0/8 No mapped journey

Apps, channels, and extensions

FeatureCapabilities linkedLinked test results
Connection setup 0/8 No mapped journey
Assistant connections (MCP) 0/5 No mapped journey
App access and action permissions 0/5 No mapped journey
External chat and email 0/6 No mapped journey
Tool gateways and access profiles 0/5 No mapped journey
Secrets and proposals 0/5 No mapped journey
Plugins 0/4 No mapped journey

Automation and structured work

FeatureCapabilities linkedLinked test results
Routines, schedules, and triggers 0/5 No mapped journey
Cases 0/5 No mapped journey
Status cards 0/4 No mapped journey

Oversight and operation

FeatureCapabilities linkedLinked test results
Costs and budgets 0/6 No mapped journey
Dashboards and audit trails 0/6 No mapped journey
Navigation, profile, and announcements 0/6 No mapped journey
Instance operations 0/6 No mapped journey
CLI/API and local worktrees 0/6 No mapped journey

Contributor surfaces

FeatureCapabilities linkedLinked test results
Developer previews and diagnostic labs 0/4 No mapped journey

Test suite

Chat feature journeys

Executable catalog requests through Agent Chat, with independently graded persisted state.

Pass rate25.0%2/8 passed
Tokens1,702,112 (partial: 3/9 runs)285,371 (partial: 3/9 runs) input · 7,941 (partial: 3/9 runs) output
Known spendUnavailablereported LLM + model/runtime estimates estimate
Agent timeUnknown0ms lease
Execution8/80 retries · cleanup passed
Native OpenCode OpenRouter DeepSeek V4.1 Flash 1017Isolated localopencode · openrouter/deepseek/deepseek-v4.1-flash-1017
Overall failed · No behavioral checks recorded

candidate failure

Stopped waiting for chat scenario tasks.create-backlog: Chat exceeded its declared run bound
Test detailsNo checks recorded · 51s
TokensUnavailable in · Unavailable outUnavailable cached · 0/2 runs covered
LLM spendUnavailable0/2 runs provider-priced
ExecutionLocal · not meteredUnknown agent

Worker phase timing: not captured.

Feedback was not captured for this attempt.

chat-features.runner-opencode.local.create-backlog
Attempt
1
Duration
51s
Agent runtime
Unknown
Runtime
native
Provider
opencode
Model
openrouter/deepseek/deepseek-v4.1-flash-1017

No result artifact was uploaded

Startup reuse evidence
RunWarmthEnvironment/leaseWorkspaceRunnerProvider processProvider sessionProvider launches
7196b904-ddfa-431d-b43f-13284936ca80unknown unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing) unknown
9eaaba3a-7678-4795-8d22-0d3bec6bd70cunknown unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing) unknown

No matcher result was recorded.

Usage, billing and timing metadata
{
  "billing": {
    "llm": {
      "runCount": 2,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "runsWithEstimatedCost": 0,
      "runsWithPartialReportedCost": 0,
      "inputTokens": 55116,
      "outputTokens": 203,
      "cachedInputTokens": 3072,
      "totalTokens": 58391,
      "reportedCostUsd": 0,
      "estimatedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": null,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null,
  "workerTiming": null
}
Native OpenCode OpenRouter DeepSeek V4.1 Flash 1017Isolated localopencode · openrouter/deepseek/deepseek-v4.1-flash-1017
Test details9/9 checks · 1m 39s
Overall passed · 9/9 behavioral checks passed

No behavioral matcher failed.

See all behavioral checks (9)
ResultCheckDetail
Passchat.complete-runtime-observationSaved agent configuration, attribution and actual rendered replies are independently required.
Passchat.saved-requested-managerMorgan reports to the real same-company Riley.
Passchat.reporting-tree-validEvery reporting chain terminates without a cycle or missing company manager.
Passchat.unrelated-config-and-tasks-retainedChanging the manager preserves accounts, permissions and existing tasks.
Passchat.attributed-agent-changeThe saved reporting-line change is independently attributed to the actual agent.
Passchat.catalog-complete-conditionsNo known catalog condition is omitted.
Passchat.catalog-original-human-queryThe persisted request preserves the catalog query with only audited public-fixture identity bindings; all human turns were declared.
Passchat.catalog-query-attachment-receiptsOnly declared production attachment suffixes are allowed, backed by same-source human receipts and independently retrieved matching bytes.
Passchat.catalog-bounded-native-runsObserved provider runs satisfy declared count, company, native runtime and allowed outcome bounds.
Tokens34,876 in · 577 out128,256 cached · 1/1 runs covered
LLM spendUnavailable0/1 runs provider-priced
ExecutionLocal · not metered36s agent

Worker phase timing: not captured.

Feedback was not captured for this attempt.

chat-features.runner-opencode.local.agents.change-manager
Attempt
1
Duration
1m 39s
Agent runtime
36s
Runtime
native
Provider
opencode
Model
openrouter/deepseek/deepseek-v4.1-flash-1017

All invariants passed

Startup reuse evidence
RunWarmthEnvironment/leaseWorkspaceRunnerProvider processProvider sessionProvider launches
109183d8-b85d-4e8a-9152-53368fff786eunknown unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing) unknown
ResultMatcherExpectationDetail
Pass json_path {"kind":"json_path","path":"chat.complete-runtime-observation","expected":true} Saved agent configuration, attribution and actual rendered replies are independently required.
Pass json_path {"kind":"json_path","path":"chat.saved-requested-manager","expected":true} Morgan reports to the real same-company Riley.
Pass json_path {"kind":"json_path","path":"chat.reporting-tree-valid","expected":true} Every reporting chain terminates without a cycle or missing company manager.
Pass json_path {"kind":"json_path","path":"chat.unrelated-config-and-tasks-retained","expected":true} Changing the manager preserves accounts, permissions and existing tasks.
Pass json_path {"kind":"json_path","path":"chat.attributed-agent-change","expected":true} The saved reporting-line change is independently attributed to the actual agent.
Pass json_path {"kind":"json_path","path":"chat.catalog-complete-conditions","expected":true} No known catalog condition is omitted.
Pass json_path {"kind":"json_path","path":"chat.catalog-original-human-query","expected":true} The persisted request preserves the catalog query with only audited public-fixture identity bindings; all human turns were declared.
Pass json_path {"kind":"json_path","path":"chat.catalog-query-attachment-receipts","expected":true} Only declared production attachment suffixes are allowed, backed by same-source human receipts and independently retrieved matching bytes.
Pass json_path {"kind":"json_path","path":"chat.catalog-bounded-native-runs","expected":true} Observed provider runs satisfy declared count, company, native runtime and allowed outcome bounds.
Usage, billing and timing metadata
{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 1,
      "runsWithReportedCost": 0,
      "runsWithEstimatedCost": 0,
      "runsWithPartialReportedCost": 0,
      "inputTokens": 34876,
      "outputTokens": 577,
      "cachedInputTokens": 128256,
      "totalTokens": 163709,
      "reportedCostUsd": 0,
      "estimatedCostUsd": 0,
      "costStatus": "unpriced"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 36024,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null,
  "workerTiming": null
}
Native OpenCode OpenRouter DeepSeek V4.1 Flash 1017Isolated localopencode · openrouter/deepseek/deepseek-v4.1-flash-1017
Overall failed · No behavioral checks recorded

candidate failure

[ { "expected": "boolean", "code": "invalid_type", "path": [ "canAssignTasks" ], "message": "Invalid input: expected boolean, received undefined" } ]
Test detailsNo checks recorded · 18s
TokensUnavailable in · Unavailable outUnavailable cached · 0/1 runs covered
LLM spendUnavailable0/1 runs provider-priced
ExecutionLocal · not meteredUnknown agent

Worker phase timing: not captured.

Feedback was not captured for this attempt.

chat-features.runner-opencode.local.agents.hire-denied
Attempt
1
Duration
18s
Agent runtime
Unknown
Runtime
native
Provider
opencode
Model
openrouter/deepseek/deepseek-v4.1-flash-1017

No result artifact was uploaded

No matcher result was recorded.

Usage, billing and timing metadata
{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "runsWithEstimatedCost": 0,
      "runsWithPartialReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "estimatedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": null,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null,
  "workerTiming": null
}
Native OpenCode OpenRouter DeepSeek V4.1 Flash 1017Isolated localopencode · openrouter/deepseek/deepseek-v4.1-flash-1017
Test details10/10 checks · 1m 10s
Overall passed · 10/10 behavioral checks passed

No behavioral matcher failed.

See all behavioral checks (10)
ResultCheckDetail
Passchat.complete-company-evidencePublic company, records and audit evidence is required.
Passchat.same-company-renamedThe saved identity changes on the original company ID.
Passchat.other-company-preservedA similarly named company remains unchanged and no duplicate company is created.
Passchat.work-retainedOriginal task and project remain attached to the same company.
Passchat.agent-attributed-identity-changeThe requested mutation is recorded for the acting CEO.
Passchat.memberships-preservedThe original member IDs, principals, roles and permissions remain unchanged.
Passchat.catalog-complete-conditionsNo known catalog condition is omitted.
Passchat.catalog-original-human-queryThe persisted request preserves the catalog query with only audited public-fixture identity bindings; all human turns were declared.
Passchat.catalog-query-attachment-receiptsOnly declared production attachment suffixes are allowed, backed by same-source human receipts and independently retrieved matching bytes.
Passchat.catalog-bounded-native-runsObserved provider runs satisfy declared count, company, native runtime and allowed outcome bounds.
Tokens33,417 in · 207 out65,408 cached · 1/1 runs covered
LLM spendUnavailable0/1 runs provider-priced
ExecutionLocal · not metered33s agent

Worker phase timing: not captured.

Feedback was not captured for this attempt.

chat-features.runner-opencode.local.companies.rename-company
Attempt
1
Duration
1m 10s
Agent runtime
33s
Runtime
native
Provider
opencode
Model
openrouter/deepseek/deepseek-v4.1-flash-1017

All invariants passed

Startup reuse evidence
RunWarmthEnvironment/leaseWorkspaceRunnerProvider processProvider sessionProvider launches
a3e9c094-6b30-4287-8881-ea00012b1311unknown unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing) unknown
ResultMatcherExpectationDetail
Pass json_path {"kind":"json_path","path":"chat.complete-company-evidence","expected":true} Public company, records and audit evidence is required.
Pass json_path {"kind":"json_path","path":"chat.same-company-renamed","expected":true} The saved identity changes on the original company ID.
Pass json_path {"kind":"json_path","path":"chat.other-company-preserved","expected":true} A similarly named company remains unchanged and no duplicate company is created.
Pass json_path {"kind":"json_path","path":"chat.work-retained","expected":true} Original task and project remain attached to the same company.
Pass json_path {"kind":"json_path","path":"chat.agent-attributed-identity-change","expected":true} The requested mutation is recorded for the acting CEO.
Pass json_path {"kind":"json_path","path":"chat.memberships-preserved","expected":true} The original member IDs, principals, roles and permissions remain unchanged.
Pass json_path {"kind":"json_path","path":"chat.catalog-complete-conditions","expected":true} No known catalog condition is omitted.
Pass json_path {"kind":"json_path","path":"chat.catalog-original-human-query","expected":true} The persisted request preserves the catalog query with only audited public-fixture identity bindings; all human turns were declared.
Pass json_path {"kind":"json_path","path":"chat.catalog-query-attachment-receipts","expected":true} Only declared production attachment suffixes are allowed, backed by same-source human receipts and independently retrieved matching bytes.
Pass json_path {"kind":"json_path","path":"chat.catalog-bounded-native-runs","expected":true} Observed provider runs satisfy declared count, company, native runtime and allowed outcome bounds.
Usage, billing and timing metadata
{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 1,
      "runsWithReportedCost": 0,
      "runsWithEstimatedCost": 0,
      "runsWithPartialReportedCost": 0,
      "inputTokens": 33417,
      "outputTokens": 207,
      "cachedInputTokens": 65408,
      "totalTokens": 99032,
      "reportedCostUsd": 0,
      "estimatedCostUsd": 0,
      "costStatus": "unpriced"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 33336,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null,
  "workerTiming": null
}
Native OpenCode OpenRouter DeepSeek V4.1 Flash 1017Isolated localopencode · openrouter/deepseek/deepseek-v4.1-flash-1017
Overall failed · No behavioral checks recorded

transient infrastructure

Stopped waiting for inbox-search.archive-without-grant: Observed provider run failed
Test detailsNo checks recorded · 2m 48s
TokensUnavailable in · Unavailable outUnavailable cached · 0/1 runs covered
LLM spendUnavailable0/1 runs provider-priced
ExecutionLocal · not metered2m 27s agent

Worker phase timing: not captured.

Feedback was not captured for this attempt.

chat-features.runner-opencode.local.inbox-search.archive-without-grant
Attempt
1
Duration
2m 48s
Agent runtime
2m 27s
Runtime
native
Provider
opencode
Model
openrouter/deepseek/deepseek-v4.1-flash-1017

No result artifact was uploaded

Startup reuse evidence
RunWarmthEnvironment/leaseWorkspaceRunnerProvider processProvider sessionProvider launches
81b6e3d7-6ced-4d0c-8be1-0532a6735810unknown unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing) unknown

No matcher result was recorded.

Usage, billing and timing metadata
{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "runsWithEstimatedCost": 0,
      "runsWithPartialReportedCost": 0,
      "inputTokens": 52716,
      "outputTokens": 3815,
      "cachedInputTokens": 925312,
      "totalTokens": 981843,
      "reportedCostUsd": 0,
      "estimatedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 146925,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null,
  "workerTiming": null
}
Native OpenCode OpenRouter DeepSeek V4.1 Flash 1017Isolated localopencode · openrouter/deepseek/deepseek-v4.1-flash-1017
Overall failed · No behavioral checks recorded

transient infrastructure

Stopped waiting for inbox-search.show-my-work: Observed provider run failed
Test detailsNo checks recorded · 2m 43s
TokensUnavailable in · Unavailable outUnavailable cached · 0/1 runs covered
LLM spendUnavailable0/1 runs provider-priced
ExecutionLocal · not metered2m 20s agent

Worker phase timing: not captured.

Feedback was not captured for this attempt.

chat-features.runner-opencode.local.inbox-search.show-my-work
Attempt
1
Duration
2m 43s
Agent runtime
2m 20s
Runtime
native
Provider
opencode
Model
openrouter/deepseek/deepseek-v4.1-flash-1017

No result artifact was uploaded

Startup reuse evidence
RunWarmthEnvironment/leaseWorkspaceRunnerProvider processProvider sessionProvider launches
b59a8598-a091-4d89-8ae7-6bb59f8a0029unknown unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing) unknown

No matcher result was recorded.

Usage, billing and timing metadata
{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "runsWithEstimatedCost": 0,
      "runsWithPartialReportedCost": 0,
      "inputTokens": 40389,
      "outputTokens": 1670,
      "cachedInputTokens": 138748,
      "totalTokens": 180807,
      "reportedCostUsd": 0,
      "estimatedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 140433,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null,
  "workerTiming": null
}
Native OpenCode OpenRouter DeepSeek V4.1 Flash 1017Isolated localopencode · openrouter/deepseek/deepseek-v4.1-flash-1017
Overall failed · No behavioral checks recorded

candidate failure

Timed out waiting for Task ambiguity clarification; the observed state did not satisfy the condition. See the saved state evidence.
Test detailsNo checks recorded · 2m 50s
TokensUnavailable in · Unavailable outUnavailable cached · 0/1 runs covered
LLM spendUnavailable0/1 runs provider-priced
ExecutionLocal · not metered2m 21s agent

Worker phase timing: not captured.

Feedback was not captured for this attempt.

chat-features.runner-opencode.local.tasks.ambiguous-task
Attempt
1
Duration
2m 50s
Agent runtime
2m 21s
Runtime
native
Provider
opencode
Model
openrouter/deepseek/deepseek-v4.1-flash-1017

No result artifact was uploaded

Startup reuse evidence
RunWarmthEnvironment/leaseWorkspaceRunnerProvider processProvider sessionProvider launches
5fe31dce-4e60-4e5b-a642-368b744a0e9aunknown unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing) unknown

No matcher result was recorded.

Usage, billing and timing metadata
{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "runsWithEstimatedCost": 0,
      "runsWithPartialReportedCost": 0,
      "inputTokens": 40125,
      "outputTokens": 727,
      "cachedInputTokens": 58020,
      "totalTokens": 98872,
      "reportedCostUsd": 0,
      "estimatedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 141155,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null,
  "workerTiming": null
}
Native OpenCode OpenRouter DeepSeek V4.1 Flash 1017Isolated localopencode · openrouter/deepseek/deepseek-v4.1-flash-1017
Overall failed · No behavioral checks recorded

candidate failure

Stopped waiting for tasks.comment-with-reference: Settled scenario failed: one-attributed-note
Test detailsNo checks recorded · 1m 16s
Tokens28,732 in · 742 out89,984 cached · 1/1 runs covered
LLM spendUnavailable0/1 runs provider-priced
ExecutionLocal · not metered38s agent

Worker phase timing: not captured.

Feedback was not captured for this attempt.

chat-features.runner-opencode.local.tasks.comment-with-reference
Attempt
1
Duration
1m 16s
Agent runtime
38s
Runtime
native
Provider
opencode
Model
openrouter/deepseek/deepseek-v4.1-flash-1017

No result artifact was uploaded

Startup reuse evidence
RunWarmthEnvironment/leaseWorkspaceRunnerProvider processProvider sessionProvider launches
f86834c7-7689-4554-978d-d3f91f2ecda2unknown unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing)unknown (evidence_missing) unknown

No matcher result was recorded.

Usage, billing and timing metadata
{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 1,
      "runsWithReportedCost": 0,
      "runsWithEstimatedCost": 0,
      "runsWithPartialReportedCost": 0,
      "inputTokens": 28732,
      "outputTokens": 742,
      "cachedInputTokens": 89984,
      "totalTokens": 119458,
      "reportedCostUsd": 0,
      "estimatedCostUsd": 0,
      "costStatus": "unpriced"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 38321,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null,
  "workerTiming": null
}