Inside Coding Agents
v0.2 preview
EXPERIMENT · CONTROLLED · PASSED

Reference Harness controlled tool roundtrip

Can the registry-backed reference runner reproduce one read-only tool roundtrip, including the same normalized event trace, from a pinned fixture and scenario?

reference-tool-roundtrip-v12 repetitions5 metrics
Evidence boundary

This is a controlled experiment. It supports only the listed subjects, fixture, configuration, and outputs; it does not generalize to native product behavior or cross-agent ranking.

STATUSpassed
SUCCESSFUL2/2
FIXTURE8d962551cb79
PROMPT4a1376db3ab9
SUBJECTS + CONTROLS

Which variables are fixed?

Both repetitions will record the declared tool request and successful result in order, preserve the fixture, match the expected answer, and share one normalized trace fingerprint.

SDK

Reference Harness

reference-harness-2026-08-10-controlled

Model
scripted-model-v1
Provider
local-fixture
Comparability
not-comparable
Inspect snapshot →
CONTROLS

Run manifest

network
false
temperature
0
writes
false
sandbox
temporary-copy-path-boundary
RUNS + METRICS

Are repeated runs consistent?

python3 labs/runner.py reference-tool-roundtrip-v1 --check
RUN 01passed

reference-tool-roundtrip-v1-run-001

  • event-order-validpass
  • tool-result-recordedpass
  • final-answer-matchpass
  • fixture-unchangedpass
  • deterministic-replaypass
a59806d5bd056ac750112dcaf356879b72d7d87b52fac970248fa8f72b94b886Replay 9 events →
RUN 02passed

reference-tool-roundtrip-v1-run-002

  • event-order-validpass
  • tool-result-recordedpass
  • final-answer-matchpass
  • fixture-unchangedpass
  • deterministic-replaypass
a59806d5bd056ac750112dcaf356879b72d7d87b52fac970248fa8f72b94b886Replay 9 events →
SUCCESS CRITERIA
  1. Both repetitions emit the declared event sequence and a successful read_file result.
  2. Both repetitions return the exact expected final answer without modifying the fixture.
  3. Both repetitions have the same normalized Trace fingerprint.
LIMITATIONS
  • The Scripted Model is a deterministic test double and does not measure model capability.
  • The Reference Harness is not a native vendor agent, so this result cannot raise Codex, Claude Code, Pi, or Reasonix coverage.
  • The workspace boundary prevents path escape but is not an operating-system sandbox.
  • The experiment does not measure latency, token usage, cost, or behavior under failure.
GENERATED REPORT

Reference Harness controlled tool roundtrip

  • Status: PASSED
  • Mode: controlled
  • Experiment: reference-tool-roundtrip-v1

Question

Can the registry-backed reference runner reproduce one read-only tool roundtrip, including the same normalized event trace, from a pinned fixture and scenario?

Reproduction

python3 labs/runner.py reference-tool-roundtrip-v1 --check

The command rebuilds the expected artifacts in memory and exits non-zero if a committed trace, result, fixture digest, prompt digest, or report differs.

Result

2 of 2 repetitions passed all declared metrics.

  • reference-tool-roundtrip-v1-run-001: passed, 9 events, fingerprint a59806d5bd056ac750112dcaf356879b72d7d87b52fac970248fa8f72b94b886
  • reference-tool-roundtrip-v1-run-002: passed, 9 events, fingerprint a59806d5bd056ac750112dcaf356879b72d7d87b52fac970248fa8f72b94b886

Metrics

  • event-order-valid: pass
  • tool-result-recorded: pass
  • final-answer-match: pass
  • fixture-unchanged: pass
  • deterministic-replay: pass

Input digests

  • Fixture: 8d962551cb790e0401896c6bab495fbd63f25c1194b70c9f5e0a79f2909ddcaf
  • Prompt: 4a1376db3ab987fd384e91931d34930c35856f1a379a017a237db9f60f408760

Evidence boundary

  • The Scripted Model is a deterministic test double and does not measure model capability.
  • The Reference Harness is not a native vendor agent, so this result cannot raise Codex, Claude Code, Pi, or Reasonix coverage.
  • The workspace boundary prevents path escape but is not an operating-system sandbox.
  • The experiment does not measure latency, token usage, cost, or behavior under failure.
Back to Lab →