​
What this pilot measures

This pilot asks how much the execution harness can affect professional-work output when each environment requests the same model and reasoning effort. It compares three complete, configured systems: Rudder, Codex CLI, and Claude Code. Each system requested gpt-5.6-sol with high reasoning effort. Its normal orchestration, session, and tool boundaries remained part of the condition, so this is not a comparison of identical stateless wrappers. GDPval-based harness benchmark with the same requested gpt-5.6-sol model: Rudder 81.7, Codex CLI 75.7, and Claude Code 75.6.

​
Results at a glance

Rudder led this frozen cohort by 6.0 percentage points over Codex CLI and 6.1 points over Claude Code under the local case-equal metric. All three harnesses had the same coarse candidate-versus-gold result. The score difference came from rubric coverage and penalty avoidance, not a higher pairwise win count. These are not official GDPval scores or leaderboard results. The supported conclusion is narrower: the Rudder configured system led this frozen 10-task cohort under our local case-equal rubric metric. This is directional evidence about each complete configured system, not a causal estimate of Rudder, Memory, or any individual harness component.

​
Frozen comparison

The pilot randomly sampled 10 tasks from the 220-task GDPval dataset, then froze the cohort before execution. It spans 10 occupations and requires 11 deliverables across XLSX, DOCX, PPTX, and PDF. Each task was run three times in total: once in Rudder, once in Codex CLI, and once in Claude Code. That produced 30 scored executions, with one execution for each harness-task combination. The task packages came from the openai/gdpval dataset at revision 11e7900cdcac61bc4daf59e65feb238acda98fbf. The candidate could inspect the task prompt and source files. Rubrics and gold deliverables stayed in a separate scorer workspace.

​
Conditions compared

This is a configured-system comparison, not a memory ablation. Rudder’s organization context, tools, instructions, and managed memory changed together as part of its normal harness condition.

​
Model identity evidence

All three conditions requested gpt-5.6-sol with high reasoning effort. The available evidence for the model that actually handled each request was not the same in every harness.

​
How scoring worked

A local LLM rubric judge scored every valid candidate separately without seeing the harness label. The judge inspected the source files, candidate deliverables, gold deliverables, and rubric items. It then recorded item-level evidence and a candidate-versus-gold judgment. Every result used the same judge configuration: gpt-5.6-sol with high reasoning effort. The primary measure gives every case equal weight even though GDPval rubrics have different point scales:
Negative rubric penalties can reduce a case score. Each case is bounded to 0 through 100 before averaging, so one penalty-heavy case cannot dominate the cohort. The full metric label is Local case-equal macro-normalized rubric score · 0-100. It is not a completion rate. Artifact validity is reported separately.

​
Case-level results

The values below are awarded raw rubric points over the positive maximum. A negative result means penalty items outweighed positive points. The macro formula clamps that case to zero.

​
Evidence limits

The comparative market analysis case had unusually large penalty items and amplified the aggregate gap. Excluding that case, Rudder still led by about 4.4 to 4.6 percentage points. Keep these boundaries with any use of the result:
  • Each task ran three times in total, once in each harness. Because every harness-task combination still has only one execution, the pilot does not estimate run-to-run variance within a harness.
  • The 10 tasks were randomly sampled from all 220 tasks, but the sample remains small and may not represent the full dataset.
  • A local LLM judge produced the rubric scores, not GDPval’s blinded expert-human review process.
  • All three harnesses finished 9W / 0T / 1L against gold. This is important counter-evidence to the score gap.
  • The run history was recovered across setup and resume events, then audited from preserved execution and judgment records.
  • Rudder memory was active, but continuity was incomplete and no Memory on/off ablation was run. The score difference cannot be attributed to Memory.
  • Codex CLI model identity was verifiable only at request time.

​
What a stronger experiment needs

A follow-up should use 30 to 50 cases, at least three repetitions per harness and case, expert-human blind review, alternate judges, and a Rudder Memory on/off comparison with counterbalanced case order. That design would measure stability and begin separating harness effects from memory, ordering, and judge effects.