Notes on what MakerBench measures, why hardware-agent benchmarks are hard, how grading actually works, and what the current results do — and do not — prove.
Long-form companions to the docs in the repository. Each post is plain, factual, and links back to the source contract it summarizes.
Across six Code-CAD Arena rounds, blind preference votes and the deterministic mesh gate rank the same entrants almost independently: the per-round Spearman swings from −1.0 to +0.87. The five directly comparable three-entrant rounds average ≈ +0.07. Looking right and being buildable are different capabilities — the statistical case for never blending scorelines.
The four-level grading ladder, the blind vs. perception split, parameter-derived (un-memorizable) pass criteria, honest token/cost telemetry, agent self-verification as an audit signal, and the limits of an alpha leaderboard.
The design rationale, the headline numbers, and the one pattern that matters: failure is bimodal — across the blind board, models either reach full Level-4 DFM or collapse to a structural-only mesh, with the geometric and physical-constraint rungs nearly empty. Grounded in the live v0.1.0 leaderboard.
Representative failure artifacts — welded lids, flipped axes, non-manifold meshes, impossible cut paths — that show why a math-graded hardware benchmark is legible, with the safety workflow that keeps oracle answers out of every example.