Runner Full-Stack E2E

Test suite

Core Runner Compatibility

Major provider, runtime generation, and execution-environment compatibility.

Pass rate4.8%1/21 passed

Tokens102,59954,458 input · 410 output

Cost$0.000000reported LLM + runtime estimate

Agent time33s0ms lease

Execution1/210 retries · cleanup failed

Configuration matrix7 profiles · 2 environments · 21 selected

Agent profile Isolated locallocal · local Daytona sandboxdaytona · remote

Legacy Codexlegacycodex · gpt-5.6-sol

Isolated locallocal · local

Basic response failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.legacy-codex.local.message-marker

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: legacy
Provider: codex
Model: gpt-5.6-sol

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Plan, revise, accept, implement failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.legacy-codex.local.plan-revise-accept

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: legacy
Provider: codex
Model: gpt-5.6-sol

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Ask mode question failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.legacy-codex.local.ask-question

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: legacy
Provider: codex
Model: gpt-5.6-sol

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Daytona sandboxdaytona · remote

Basic response not selected

core-compatibility.legacy-codex.daytona.message-marker

Matchers and test context

Not selected

No matcher result was recorded.

Plan, revise, accept, implement not selected

core-compatibility.legacy-codex.daytona.plan-revise-accept

Matchers and test context

Not selected

No matcher result was recorded.

Ask mode question not selected

core-compatibility.legacy-codex.daytona.ask-question

Matchers and test context

Not selected

No matcher result was recorded.

Legacy Claudelegacyclaude · claude-sonnet-4-6

Isolated locallocal · local

Basic response failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.legacy-claude.local.message-marker

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: legacy
Provider: claude
Model: claude-sonnet-4-6

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Plan, revise, accept, implement failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.legacy-claude.local.plan-revise-accept

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: legacy
Provider: claude
Model: claude-sonnet-4-6

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Ask mode question failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.legacy-claude.local.ask-question

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: legacy
Provider: claude
Model: claude-sonnet-4-6

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Daytona sandboxdaytona · remote

Basic response not selected

core-compatibility.legacy-claude.daytona.message-marker

Matchers and test context

Not selected

No matcher result was recorded.

Plan, revise, accept, implement not selected

core-compatibility.legacy-claude.daytona.plan-revise-accept

Matchers and test context

Not selected

No matcher result was recorded.

Ask mode question not selected

core-compatibility.legacy-claude.daytona.ask-question

Matchers and test context

Not selected

No matcher result was recorded.

Legacy OpenCodelegacyopencode · openrouter/deepseek/deepseek-v4-flash-0731

Isolated locallocal · local

Basic response failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.legacy-opencode.local.message-marker

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: legacy
Provider: opencode
Model: openrouter/deepseek/deepseek-v4-flash-0731

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Plan, revise, accept, implement failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.legacy-opencode.local.plan-revise-accept

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: legacy
Provider: opencode
Model: openrouter/deepseek/deepseek-v4-flash-0731

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Ask mode question failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.legacy-opencode.local.ask-question

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: legacy
Provider: opencode
Model: openrouter/deepseek/deepseek-v4-flash-0731

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Daytona sandboxdaytona · remote

Basic response not selected

core-compatibility.legacy-opencode.daytona.message-marker

Matchers and test context

Not selected

No matcher result was recorded.

Plan, revise, accept, implement not selected

core-compatibility.legacy-opencode.daytona.plan-revise-accept

Matchers and test context

Not selected

No matcher result was recorded.

Ask mode question not selected

core-compatibility.legacy-opencode.daytona.ask-question

Matchers and test context

Not selected

No matcher result was recorded.

Runner Codexnativecodex · gpt-5.6-sol

Isolated locallocal · local

Basic response failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.runner-codex.local.message-marker

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: native
Provider: codex
Model: gpt-5.6-sol

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Plan, revise, accept, implement passed

Tokens54,458 in · 410 out47,731 cached · 3/3 runs covered

LLM spend$0.0000003/3 runs provider-priced

ExecutionLocal · not metered33s agent

core-compatibility.runner-codex.local.plan-revise-accept

Matchers and test context

Attempt: 1
Duration: 1m 10s
Agent runtime: 33s
Runtime: native
Provider: codex
Model: gpt-5.6-sol
Issue: RUN-1

All invariants passed

Result	Matcher	Expectation	Detail
Pass	`message_exact`	{"kind":"message_exact","expected":"PAPERCLIP_E2E_PLAN_DONE_044067473f04-1"}	matched
Pass	`message_occurrences`	{"kind":"message_occurrences","expected":"PAPERCLIP_E2E_PLAN_DONE_044067473f04-1","count":1}	matched
Pass	`issue_status`	{"kind":"issue_status","expected":"done"}	matched
Pass	`run_status`	{"kind":"run_status","expected":"succeeded"}	matched
Pass	`runtime_mode`	{"kind":"runtime_mode","expected":"native"}	matched
Pass	`environment`	{"kind":"environment","expected":"local"}	matched

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 3,
      "runsWithTokenUsage": 3,
      "runsWithReportedCost": 3,
      "inputTokens": 54458,
      "outputTokens": 410,
      "cachedInputTokens": 47731,
      "totalTokens": 102599,
      "reportedCostUsd": 0,
      "costStatus": "reported"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 33057,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": true
  },
  "rawUsage": {
    "runs": [
      {
        "runId": "703e5583-430e-4073-b5bd-ed2c7e2da11c",
        "usage": {
          "model": "gpt-5.6-sol",
          "biller": "openai",
          "costUsd": 0,
          "provider": "openai",
          "costStatus": "reported",
          "billingType": "unknown",
          "inputTokens": 15623,
          "usageSource": "per_run",
          "freshSession": true,
          "outputTokens": 149,
          "sessionReused": false,
          "rawInputTokens": 15623,
          "sessionRotated": false,
          "configFreshness": {
            "session": {
              "reset": true,
              "categories": [
                "adapter",
                "adapterConfig",
                "agentRuntimeConfig",
                "instructions",
                "issueOverrides",
                "workspaceConfig",
                "environment",
                "envBindings",
                "secrets",
                "runtimeSkills"
              ],
              "resetReasons": [],
              "nextFingerprint": "v1:sha256:d6638f6a94d321ba78de92a52f428cddd7246669b887ebd08e78446d307d5383",
              "changedCategories": [],
              "taskSessionReused": false,
              "fingerprintVersion": 1,
              "taskSessionAvailable": false,
              "storedFingerprintPresent": false
            },
            "version": 1,
            "workspace": {
              "action": "create",
              "reasons": [],
              "categories": [
                "mode",
                "projectWorkspace",
                "strategy",
                "repo",
                "lifecycleCommands",
                "runtimeServices",
                "environment",
                "realization"
              ],
              "reuseRequested": false,
              "nextFingerprint": "v1:sha256:c8de4bc9c8f18a724d973fba3d7e90fc534b7c5111fb9dd429cb5b4b24e0fdc6",
              "workspaceReused": false,
              "activeWorkspaceId": null,
              "changedCategories": [],
              "storedFingerprint": null,
              "fingerprintVersion": 1,
              "inferredFingerprint": null,
              "previousWorkspaceId": null,
              "configSnapshotRefreshed": false,
              "storedFingerprintPresent": false
            }
          },
          "rawOutputTokens": 149,
          "cachedInputTokens": 15069,
          "taskSessionReused": false,
          "persistedSessionId": "01a06b21-0a36-71d3-812a-be67a9db02c1",
          "cacheAdjustedCostUsd": 0,
          "rawCachedInputTokens": 15069,
          "sessionRotationReason": null
        }
      },
      {
        "runId": "ecf52061-9a29-4204-9a7a-aa44e7c590e5",
        "usage": {
          "model": "gpt-5.6-sol",
          "biller": "openai",
          "costUsd": 0,
          "provider": "openai",
          "costStatus": "reported",
          "billingType": "unknown",
          "inputTokens": 22155,
          "usageSource": "per_run",
          "freshSession": false,
          "outputTokens": 240,
          "sessionReused": true,
          "rawInputTokens": 22155,
          "sessionRotated": false,
          "configFreshness": {
            "session": {
              "reset": false,
              "categories": [
                "adapter",
                "adapterConfig",
                "agentRuntimeConfig",
                "instructions",
                "issueOverrides",
                "workspaceConfig",
                "environment",
                "envBindings",
                "secrets",
                "runtimeSkills"
              ],
              "resetReasons": [],
              "nextFingerprint": "v1:sha256:d6638f6a94d321ba78de92a52f428cddd7246669b887ebd08e78446d307d5383",
              "changedCategories": [],
              "taskSessionReused": true,
              "fingerprintVersion": 1,
              "taskSessionAvailable": true,
              "storedFingerprintPresent": true
            },
            "version": 1,
            "workspace": {
              "action": "create",
              "reasons": [],
              "categories": [
                "mode",
                "projectWorkspace",
                "strategy",
                "repo",
                "lifecycleCommands",
                "runtimeServices",
                "environment",
                "realization"
              ],
              "reuseRequested": false,
              "nextFingerprint": "v1:sha256:c8de4bc9c8f18a724d973fba3d7e90fc534b7c5111fb9dd429cb5b4b24e0fdc6",
              "workspaceReused": false,
              "activeWorkspaceId": null,
              "changedCategories": [],
              "storedFingerprint": null,
              "fingerprintVersion": 1,
              "inferredFingerprint": null,
              "previousWorkspaceId": null,
              "configSnapshotRefreshed": false,
              "storedFingerprintPresent": false
            }
          },
          "rawOutputTokens": 240,
          "cachedInputTokens": 16271,
          "taskSessionReused": true,
          "persistedSessionId": "01a06b21-0a36-71d3-812a-be67a9db02c1",
          "cacheAdjustedCostUsd": 0,
          "rawCachedInputTokens": 16271,
          "sessionRotationReason": null
        }
      },
      {
        "runId": "4f51788f-8fe6-4015-8aa3-01b5ab424b6c",
        "usage": {
          "model": "gpt-5.6-sol",
          "biller": "openai",
          "costUsd": 0,
          "provider": "openai",
          "costStatus": "reported",
          "billingType": "unknown",
          "inputTokens": 16680,
          "usageSource": "per_run",
          "freshSession": true,
          "outputTokens": 21,
          "sessionReused": false,
          "rawInputTokens": 16680,
          "sessionRotated": false,
          "configFreshness": {
            "session": {
              "reset": true,
              "categories": [
                "adapter",
                "adapterConfig",
                "agentRuntimeConfig",
                "instructions",
                "issueOverrides",
                "workspaceConfig",
                "environment",
                "envBindings",
                "secrets",
                "runtimeSkills"
              ],
              "resetReasons": [
                "effective run configuration changed: adapter config",
                "forceFreshSession was requested"
              ],
              "nextFingerprint": "v1:sha256:b5ac20e0fe35790e5a770d11dbec27b67d7b4684fef16f68d8ca21b96b402341",
              "changedCategories": [
                "adapterConfig"
              ],
              "taskSessionReused": false,
              "fingerprintVersion": 1,
              "taskSessionAvailable": true,
              "storedFingerprintPresent": true
            },
            "version": 1,
            "workspace": {
              "action": "create",
              "reasons": [],
              "categories": [
                "mode",
                "projectWorkspace",
                "strategy",
                "repo",
                "lifecycleCommands",
                "runtimeServices",
                "environment",
                "realization"
              ],
              "reuseRequested": false,
              "nextFingerprint": "v1:sha256:c8de4bc9c8f18a724d973fba3d7e90fc534b7c5111fb9dd429cb5b4b24e0fdc6",
              "workspaceReused": false,
              "activeWorkspaceId": null,
              "changedCategories": [],
              "storedFingerprint": null,
              "fingerprintVersion": 1,
              "inferredFingerprint": null,
              "previousWorkspaceId": null,
              "configSnapshotRefreshed": false,
              "storedFingerprintPresent": false
            }
          },
          "rawOutputTokens": 21,
          "cachedInputTokens": 16391,
          "taskSessionReused": false,
          "persistedSessionId": "01a06b21-b457-7671-bbd9-3b77c9a4bcd8",
          "cacheAdjustedCostUsd": 0,
          "rawCachedInputTokens": 16391,
          "sessionRotationReason": null
        }
      }
    ]
  }
}

Ask mode question failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.runner-codex.local.ask-question

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: native
Provider: codex
Model: gpt-5.6-sol

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Daytona sandboxdaytona · remote

Basic response not selected

core-compatibility.runner-codex.daytona.message-marker

Matchers and test context

Not selected

No matcher result was recorded.

Plan, revise, accept, implement not selected

core-compatibility.runner-codex.daytona.plan-revise-accept

Matchers and test context

Not selected

No matcher result was recorded.

Ask mode question not selected

core-compatibility.runner-codex.daytona.ask-question

Matchers and test context

Not selected

No matcher result was recorded.

Runner OpenCodenativeopencode · openrouter/deepseek/deepseek-v4-flash-0731

Isolated locallocal · local

Basic response failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.runner-opencode.local.message-marker

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: native
Provider: opencode
Model: openrouter/deepseek/deepseek-v4-flash-0731

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Plan, revise, accept, implement failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.runner-opencode.local.plan-revise-accept

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: native
Provider: opencode
Model: openrouter/deepseek/deepseek-v4-flash-0731

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Ask mode question failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.runner-opencode.local.ask-question

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: native
Provider: opencode
Model: openrouter/deepseek/deepseek-v4-flash-0731

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Daytona sandboxdaytona · remote

Basic response not selected

core-compatibility.runner-opencode.daytona.message-marker

Matchers and test context

Not selected

No matcher result was recorded.

Plan, revise, accept, implement not selected

core-compatibility.runner-opencode.daytona.plan-revise-accept

Matchers and test context

Not selected

No matcher result was recorded.

Ask mode question not selected

core-compatibility.runner-opencode.daytona.ask-question

Matchers and test context

Not selected

No matcher result was recorded.

Runner ACPX Claudenativeacpx · claude-sonnet-5

Isolated locallocal · local

Basic response failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.runner-acpx-claude.local.message-marker

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: native
Provider: acpx
Model: claude-sonnet-5

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Plan, revise, accept, implement failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.runner-acpx-claude.local.plan-revise-accept

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: native
Provider: acpx
Model: claude-sonnet-5

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Ask mode question failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.runner-acpx-claude.local.ask-question

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: native
Provider: acpx
Model: claude-sonnet-5

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Daytona sandboxdaytona · remote

Basic response not selected

core-compatibility.runner-acpx-claude.daytona.message-marker

Matchers and test context

Not selected

No matcher result was recorded.

Plan, revise, accept, implement not selected

core-compatibility.runner-acpx-claude.daytona.plan-revise-accept

Matchers and test context

Not selected

No matcher result was recorded.

Ask mode question not selected

core-compatibility.runner-acpx-claude.daytona.ask-question

Matchers and test context

Not selected

No matcher result was recorded.

Runner ACPX Codexnativeacpx · gpt-5.6-sol

Isolated locallocal · local

Basic response failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.runner-acpx-codex.local.message-marker

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: native
Provider: acpx
Model: gpt-5.6-sol

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Plan, revise, accept, implement failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.runner-acpx-codex.local.plan-revise-accept

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: native
Provider: acpx
Model: gpt-5.6-sol

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Ask mode question failed

Tokens0 in · 0 out0 cached · 0/1 runs covered

LLM spendunavailable0/1 runs provider-priced

ExecutionLocal · not metered0ms agent

core-compatibility.runner-acpx-codex.local.ask-question

Matchers and test context

Attempt: 0
Duration: 0ms
Agent runtime: 0ms
Runtime: native
Provider: acpx
Model: gpt-5.6-sol

No result artifact was uploaded

No matcher result was recorded.

Usage and billing metadata

{
  "billing": {
    "llm": {
      "runCount": 1,
      "runsWithTokenUsage": 0,
      "runsWithReportedCost": 0,
      "inputTokens": 0,
      "outputTokens": 0,
      "cachedInputTokens": 0,
      "totalTokens": 0,
      "reportedCostUsd": 0,
      "costStatus": "unavailable"
    },
    "runtime": {
      "provider": "local",
      "agentRunDurationMs": 0,
      "leaseDurationMs": null,
      "leaseCount": 0,
      "costStatus": "not_metered",
      "costSource": "local_not_metered"
    },
    "reportedCostUsd": 0,
    "estimatedRuntimeCostUsd": 0,
    "observedAndEstimatedCostUsd": 0,
    "complete": false
  },
  "rawUsage": null
}

Daytona sandboxdaytona · remote

Basic response not selected

core-compatibility.runner-acpx-codex.daytona.message-marker

Matchers and test context

Not selected

No matcher result was recorded.

Plan, revise, accept, implement not selected

core-compatibility.runner-acpx-codex.daytona.plan-revise-accept

Matchers and test context

Not selected

No matcher result was recorded.

Ask mode question not selected

core-compatibility.runner-acpx-codex.daytona.ask-question

Matchers and test context

Not selected

No matcher result was recorded.

Test suite

Local Session Integrity

Structured interaction and continuation qualification for every supported local profile.

Pass rate0.0%0/14 passed

Tokens00 input · 0 output

Cost$0.000000reported LLM + runtime estimate

Agent time0ms0ms lease

Execution0/140 retries · cleanup failed

Configuration matrix7 profiles · 1 environments · 14 selected

Agent profile	Isolated locallocal · local
Legacy Codexlegacycodex · gpt-5.6-sol	Isolated locallocal · local Structured question, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.legacy-codex.local.structured-question-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime legacy Provider codex Model gpt-5.6-sol No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Structured question, server restart, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.legacy-codex.local.structured-question-restart-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime legacy Provider codex Model gpt-5.6-sol No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null }
Legacy Claudelegacyclaude · claude-sonnet-4-6	Isolated locallocal · local Structured question, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.legacy-claude.local.structured-question-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime legacy Provider claude Model claude-sonnet-4-6 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Structured question, server restart, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.legacy-claude.local.structured-question-restart-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime legacy Provider claude Model claude-sonnet-4-6 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null }
Legacy OpenCodelegacyopencode · openrouter/deepseek/deepseek-v4-flash-0731	Isolated locallocal · local Structured question, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.legacy-opencode.local.structured-question-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime legacy Provider opencode Model openrouter/deepseek/deepseek-v4-flash-0731 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Structured question, server restart, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.legacy-opencode.local.structured-question-restart-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime legacy Provider opencode Model openrouter/deepseek/deepseek-v4-flash-0731 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null }
Runner Codexnativecodex · gpt-5.6-sol	Isolated locallocal · local Structured question, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.runner-codex.local.structured-question-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider codex Model gpt-5.6-sol No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Structured question, server restart, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.runner-codex.local.structured-question-restart-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider codex Model gpt-5.6-sol No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null }
Runner OpenCodenativeopencode · openrouter/deepseek/deepseek-v4-flash-0731	Isolated locallocal · local Structured question, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.runner-opencode.local.structured-question-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/deepseek/deepseek-v4-flash-0731 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Structured question, server restart, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.runner-opencode.local.structured-question-restart-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/deepseek/deepseek-v4-flash-0731 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null }
Runner ACPX Claudenativeacpx · claude-sonnet-5	Isolated locallocal · local Structured question, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.runner-acpx-claude.local.structured-question-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider acpx Model claude-sonnet-5 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Structured question, server restart, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider acpx Model claude-sonnet-5 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null }
Runner ACPX Codexnativeacpx · gpt-5.6-sol	Isolated locallocal · local Structured question, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.runner-acpx-codex.local.structured-question-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider acpx Model gpt-5.6-sol No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Structured question, server restart, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `local-session-integrity.runner-acpx-codex.local.structured-question-restart-resume` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider acpx Model gpt-5.6-sol No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null }

Test suite

OpenRouter Model Breadth

Weekly-ranked tool-capable OpenRouter models through native OpenCode on isolated local workspaces.

Pass rate0.0%0/10 passed

Tokens00 input · 0 output

Cost$0.000000reported LLM + runtime estimate

Agent time0ms0ms lease

Execution0/100 retries · cleanup failed

Configuration matrix4 profiles · 1 environments · 10 selected

Agent profile	Isolated locallocal · local
#1 DeepSeek V4 Flash 0731nativeopencode · openrouter/deepseek/deepseek-v4-flash-0731	Isolated locallocal · local Hello and complete failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `openrouter-model-breadth.openrouter-deepseek-deepseek-v4-flash-0731.local.hello-complete` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/deepseek/deepseek-v4-flash-0731 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Ask, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `openrouter-model-breadth.openrouter-deepseek-deepseek-v4-flash-0731.local.question-resume-complete` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/deepseek/deepseek-v4-flash-0731 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null }
#3 Tencent HY 3nativeopencode · openrouter/tencent/hy3	Isolated locallocal · local Hello and complete failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `openrouter-model-breadth.openrouter-tencent-hy3.local.hello-complete` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/tencent/hy3 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Ask, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `openrouter-model-breadth.openrouter-tencent-hy3.local.question-resume-complete` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/tencent/hy3 No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Plan, approve, complete not selected `openrouter-model-breadth.openrouter-tencent-hy3.local.plan-approve-complete` Matchers and test context Not selected No matcher result was recorded.
#4 Nemotron 3 Ultra 550B A55B (free)nativeopencode · openrouter/nvidia/nemotron-3-ultra-550b-a55b:free	Isolated locallocal · local Hello and complete failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `openrouter-model-breadth.openrouter-nvidia-nemotron-3-ultra-550b-a55b-free.local.hello-complete` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/nvidia/nemotron-3-ultra-550b-a55b:free No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Ask, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `openrouter-model-breadth.openrouter-nvidia-nemotron-3-ultra-550b-a55b-free.local.question-resume-complete` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/nvidia/nemotron-3-ultra-550b-a55b:free No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Plan, approve, complete failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `openrouter-model-breadth.openrouter-nvidia-nemotron-3-ultra-550b-a55b-free.local.plan-approve-complete` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/nvidia/nemotron-3-ultra-550b-a55b:free No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null }
#5 GPT-5.6 Lunanativeopencode · openrouter/openai/gpt-5.6-luna	Isolated locallocal · local Hello and complete failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `openrouter-model-breadth.openrouter-openai-gpt-5-6-luna.local.hello-complete` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/openai/gpt-5.6-luna No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Ask, answer, resume failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `openrouter-model-breadth.openrouter-openai-gpt-5-6-luna.local.question-resume-complete` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/openai/gpt-5.6-luna No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null } Plan, approve, complete failed Tokens0 in · 0 out0 cached · 0/1 runs covered LLM spendunavailable0/1 runs provider-priced ExecutionLocal · not metered0ms agent `openrouter-model-breadth.openrouter-openai-gpt-5-6-luna.local.plan-approve-complete` Matchers and test context Attempt 0 Duration 0ms Agent runtime 0ms Runtime native Provider opencode Model openrouter/openai/gpt-5.6-luna No result artifact was uploaded No matcher result was recorded. Usage and billing metadata { "billing": { "llm": { "runCount": 1, "runsWithTokenUsage": 0, "runsWithReportedCost": 0, "inputTokens": 0, "outputTokens": 0, "cachedInputTokens": 0, "totalTokens": 0, "reportedCostUsd": 0, "costStatus": "unavailable" }, "runtime": { "provider": "local", "agentRunDurationMs": 0, "leaseDurationMs": null, "leaseCount": 0, "costStatus": "not_metered", "costSource": "local_not_metered" }, "reportedCostUsd": 0, "estimatedRuntimeCostUsd": 0, "observedAndEstimatedCostUsd": 0, "complete": false }, "rawUsage": null }

Core Runner Compatibility

Local Session Integrity

OpenRouter Model Breadth

Campaign trends