​
What this pilot measures

This pilot asks how much the execution harness can affect professional-work output when each environment requests the same model and reasoning effort. It compares three complete, configured systems: Rudder, Codex CLI, and Claude Code. Each system requested gpt-5.6-sol with high reasoning effort. Its normal orchestration, session, and tool boundaries remained part of the condition, so this is not a comparison of identical stateless wrappers. GDPval-based harness benchmark with the same requested gpt-5.6-sol model: Rudder 81.7, Codex CLI 75.7, and Claude Code 75.6.

​
Results at a glance

HarnessLocal case-equal scoreValid artifactsCandidate vs gold
Rudder81.710/109W / 0T / 1L
Codex CLI75.710/109W / 0T / 1L
Claude Code75.610/109W / 0T / 1L
Rudder led this frozen cohort by 6.0 percentage points over Codex CLI and 6.1 points over Claude Code under the local case-equal metric. All three harnesses had the same coarse candidate-versus-gold result. The score difference came from rubric coverage and penalty avoidance, not a higher pairwise win count. These are not official GDPval scores or leaderboard results. The supported conclusion is narrower: the Rudder configured system led this frozen 10-task cohort under our local case-equal rubric metric. This is directional evidence about each complete configured system, not a causal estimate of Rudder, Memory, or any individual harness component.

​
Frozen comparison

The pilot randomly sampled 10 tasks from the 220-task GDPval dataset, then froze the cohort before execution. It spans 10 occupations and requires 11 deliverables across XLSX, DOCX, PPTX, and PDF. Each task was run three times in total: once in Rudder, once in Codex CLI, and once in Claude Code. That produced 30 scored executions, with one execution for each harness-task combination. The task packages came from the openai/gdpval dataset at revision 11e7900cdcac61bc4daf59e65feb238acda98fbf. The candidate could inspect the task prompt and source files. Rubrics and gold deliverables stayed in a separate scorer workspace.

​
Conditions compared

HarnessExecution conditionPersistent state
RudderOne organization and one default managed Agent were reused sequentially across all 10 cases, with Rudder’s managed instruction bundle and toolsManaged memory was active and read and written, but the reconstructed timeline contains one unexplained continuity reset
Codex CLIEvery case used a fresh workspace and sessionNo state was reused between cases
Claude CodeEvery case used a fresh workspace and session routed through a compatibility adapterNo state was reused between cases
This is a configured-system comparison, not a memory ablation. Rudder’s organization context, tools, instructions, and managed memory changed together as part of its normal harness condition.

​
Model identity evidence

All three conditions requested gpt-5.6-sol with high reasoning effort. The available evidence for the model that actually handled each request was not the same in every harness.
HarnessVerification available
RudderRun metadata matched the requested model for all 10 cases
Claude CodeCompatibility-adapter request logs observed gpt-5.6-sol, including requests translated from Claude aliases
Codex CLIThe request was configured for gpt-5.6-sol, but the event stream did not expose the actual model identity

​
How scoring worked

A local LLM rubric judge scored every valid candidate separately without seeing the harness label. The judge inspected the source files, candidate deliverables, gold deliverables, and rubric items. It then recorded item-level evidence and a candidate-versus-gold judgment. Every result used the same judge configuration: gpt-5.6-sol with high reasoning effort. The primary measure gives every case equal weight even though GDPval rubrics have different point scales: 
case_score_i = clamp(100 × awarded_raw_points_i / positive_max_points_i, 0, 100)
macro_score  = average(case_score_1 ... case_score_10)
Negative rubric penalties can reduce a case score. Each case is bounded to 0 through 100 before averaging, so one penalty-heavy case cannot dominate the cohort. The full metric label is Local case-equal macro-normalized rubric score · 0-100. It is not a completion rate. Artifact validity is reported separately.

​
Case-level results

The values below are awarded raw rubric points over the positive maximum. A negative result means penalty items outweighed positive points. The macro formula clamps that case to zero.
CaseOccupationClaude CodeCodex CLIRudderVs gold
Audit risk metrics sampleAccountants and Auditors48/6349/6356/63all win
Enterprise UAT planComputer and Information Systems Managers83/10174/10187/101all win
Suspicious activity investigationCompliance Officers62/6958/6962/69all win
Property superintendent performance planProperty and Real Estate Managers66/8562/8567/85all win
Nature documentary scriptFilm and Video Editors50/5151/5151/51all win
BridgeMind POC monthly reportProject Management Specialists78/7978/7978/79all win
Inventory incident presentationShipping and Inventory Clerks74/8172/8176/81all win
Territory fit reportSales Managers61/10181/10180/101all win
Tiny Rod finance reportFinancial Managers62/7662/7662/76all loss
Duplex comparative market analysisReal Estate Brokers-70/50-35/5010/50all win

​
Evidence limits

The comparative market analysis case had unusually large penalty items and amplified the aggregate gap. Excluding that case, Rudder still led by about 4.4 to 4.6 percentage points. Keep these boundaries with any use of the result:
  • Each task ran three times in total, once in each harness. Because every harness-task combination still has only one execution, the pilot does not estimate run-to-run variance within a harness.
  • The 10 tasks were randomly sampled from all 220 tasks, but the sample remains small and may not represent the full dataset.
  • A local LLM judge produced the rubric scores, not GDPval’s blinded expert-human review process.
  • All three harnesses finished 9W / 0T / 1L against gold. This is important counter-evidence to the score gap.
  • The run history was recovered across setup and resume events, then audited from preserved execution and judgment records.
  • Rudder memory was active, but continuity was incomplete and no Memory on/off ablation was run. The score difference cannot be attributed to Memory.
  • Codex CLI model identity was verifiable only at request time.

​
What a stronger experiment needs

A follow-up should use 30 to 50 cases, at least three repetitions per harness and case, expert-human blind review, alternate judges, and a Rudder Memory on/off comparison with counterbalanced case order. That design would measure stability and begin separating harness effects from memory, ordering, and judge effects.