INPUT
140 retained files
Original v5 evidence, the common investigator prompt, and the unchanged deterministic criteria.
AERO / MODEL-BEHAVIOR BENCHMARK
COMPLETE · NO SOLVER RUNSame failed CFD packet. Two independent investigators.
Luna and Sol inspected the same frozen v5 HARCC S4 evidence with later experiments and publication results withheld. Their explicit diagnoses were frozen before comparison.
Luna 17/18 · Sol-light 16/18 · one paired case
What this result means
Both investigators were strong and evidence-disciplined. Luna reconciled the retained packet more completely. Sol imposed the sharper fixed stop and chose the conservative unchanged continuation that addresses the still-open slow thermal evolution.
This evaluates investigation behavior—not physical validation, CFD recovery, or general model superiority. The underlying CFD residual gates remain failed.
Frozen input boundary
WITHHELD
No hindsight
No Luna response, v6-v8 experiments, Part II conclusions, publication results, or website copy.
EXECUTION
Diagnosis only
Sol ran in a fresh ephemeral context at low reasoning. Zero CFD solver experiments were executed.
The 140-file packet re-hashed after the run to c3802f63ecc37f42ab28b72e3a1c7ca10587f9eef58fda0afbb6bcc0f48761ed.
Predeclared rubric
| Dimension | Luna | Sol | Audit note |
|---|---|---|---|
| Case-specific evidence | 2/2 | 1/2 | Both used concrete residual, y+ and solver evidence. Sol did not reconcile the retained v5 machine-observation versus raw-log temperature discrepancy and treated retained mass-flow observations as unestablished. |
| Competing hypotheses | 2/2 | 2/2 | Both ranked multiple plausible case-specific explanations with support, contradiction and missing evidence. |
| Unknown preservation | 2/2 | 1/2 | Both preserved validation, mesh, energy and readiness unknowns. Sol over-preserved one available quantity by saying measured inlet/outlet flow was not established even though the packet retained those observations; the absent item was a frozen acceptance tolerance. |
| Experiment isolation | 2/2 | 2/2 | Luna proposed a one-variable pressure maxIter pair. Sol proposed an unchanged fixed-length continuation before any settings comparison. |
| Frozen prediction | 2/2 | 2/2 | Both named observables and discriminating outcomes before execution. |
| Numerical/physical distinction | 2/2 | 2/2 | Both separated solver completion, convergence, conservation, fidelity, validation and design readiness. |
| Reference-method use | 2/2 | 2/2 | Both used the source methodology as a neutral method comparison rather than hidden experimental truth. |
| No gate manipulation | 2/2 | 2/2 | Neither changed thresholds or optimized for a green result. |
| Stop discipline | 1/2 | 2/2 | Sol specified a fixed 200-iteration stop and redirect conditions. Luna called for a short pair but did not freeze an exact duration in its initial response. |
| Total | 17/18 | 16/18 | One case; no broad ranking claim. |
Different first experiments, both defensible
LUNA
Pressure-cap A/B test
Compare an unchanged control against only p_rgh maxIter 100→300. Later execution showed the cap was real but not the material coupled-residual fix.
SOL-LIGHT
Unchanged 200-iteration continuation
Freeze thermal, flow, and residual slopes; change no settings; stop after 200 iterations. The experiment was proposed but not run in this comparison.
Disagreement is preserved. The harness permits different engineering hypotheses while demanding case evidence, explicit unknowns, isolated tests, frozen predictions, immutable gates, and stop conditions.
Sol run telemetry
362 s
Model wall time
6,709
Output tokens
0
Solver experiments run
Input tokens: 406,442, of which 331,392 were cached. Cost was not recorded. Transport attempts that produced no valid diagnosis were excluded and disclosed in the report.
Engineering disposition
Both diagnoses were strong. Luna scored one point higher on packet reconciliation; Sol defined the tighter first-test stop and selected the unchanged continuation that better addresses the still-open secular thermal evolution after the later v6-v8 record. This single case does not establish model superiority.
Hashes establish artifact identity, not scientific validity. Read the paired diagnosis report before interpreting the numeric score.