Building a Harness

A harness is the script that turns a candidate CAD file into a number. This guide walks through how StoryGold structures a task and its grading harness, using the three public take-home repos (arwinai/test_example, arwinai/test_example2, arwinai/test_example3) as the worked examples. Two of them are SolidWorks tasks graded over COM on Windows; the third is a FreeCAD task graded headlessly through freecadcmd, and it is the only one of the three that ships a real harness skeleton (ALL_CRITERIA, MUST_PASS, build_state / checks / finalize). Where the repos illustrate a team rule, the rule is quoted next to it; where the repos predate or contradict a current rule, the guide says what the repos actually do and flags the difference in a !!! warning.

What a task is

A task is: a Before model (the seed, environment/input.*), a Prompt (instruction.md), an After reference solution (solution/solution.*), a checklist, and 5-10 Defective ("adversarial") examples, each with exactly one defect. Every adversarial lives at examples/adversarial_<defect>/adversarial_<defect>.<ext>; the folder name and the top-level file name match. All three repos follow this naming, e.g. examples/adversarial_widened_by_30mm/adversarial_widened_by_30mm.SLDPRT and examples/adversarial_joint_limits_swapped/adversarial_joint_limits_swapped.FCStd.

Folder layout

All three repos share the same skeleton: a repo root with tools/ and env_requirements.txt, then one directory per CAD family (SolidWorks/, FreeCAD/), then one numbered task folder per task. The tree below is a composite; the per-repo differences are called out afterwards.

<repo>/
├── README.md                         # what the task is, how to fetch assets, how to run and validate the harness
├── env_requirements.txt              # pinned pip deps for the scoring side (numpy, trimesh, shapely, scipy...; pywin32 on win32 only)
├── .gitignore                        # *.SLDPRT, *.SLDASM, *.FCStd, *.stl, *.STEP never go in git -- they live in Azure blob
├── tools/
│   ├── fetch.py                      # downloads every [[metadata.assets]] entry that is missing or fails its sha256
│   └── sync.py                       # uploads local CAD binaries to blob and rewrites the sha256/bytes pins in task.toml
├── docker/
│   └── freecad-base.Dockerfile       # (test_example3 only) FreeCAD 1.1.3 AppImage base image exposing `freecadcmd`
└── <Family>/<N>_<task_name>/         # e.g. SolidWorks/1_playstation_controller, FreeCAD/4_sling_lift
    ├── instruction.md                # the Prompt the candidate sees; names the input file and the output file to save
    ├── task.toml                     # schema 1.4: [task], [metadata] (max_score, reference, notes), [metadata.harness], assets, env
    ├── environment/
    │   ├── input.SLDPRT | input.SLDASM | input.pdf    # the Before model (seed), or the drawing for a reconstruction task
    │   ├── input.png                 # committed render of the seed
    │   └── Dockerfile                # verifier image; a stub with TODOs for SolidWorks, `FROM freecad-base:latest` for FreeCAD
    ├── solution/
    │   ├── solution.SLDPRT | .SLDASM | .FCStd         # the After reference; must score max_score
    │   ├── solution.png              # committed render of the reference
    │   └── solve.sh                  # oracle: copies the reference into place (FreeCAD) or explains why it cannot (SolidWorks)
    ├── examples/
    │   └── adversarial_<defect>/
    │       ├── adversarial_<defect>.SLDPRT | .SLDASM | .FCStd   # one defect per example, top file named after the folder
    │       ├── adversarial_<defect>.STL                        # mesh export beside the native file (SolidWorks repos)
    │       └── adversarial_<defect>.png                        # committed render so the corpus can be browsed before fetch
    └── tests/
        ├── test.sh                   # verifier entrypoint (Harbor); fetches pinned assets, runs the harness, writes reward.json
        ├── test.bat                  # (SolidWorks) Windows entrypoint: python task\harness\harness.py %*
        └── task/
            ├── harness/
            │   ├── harness.py        # the grader; prints the JSON score envelope
            │   └── measure_stage.py  # (FreeCAD) measurement stage run under freecadcmd, writes measure.json
            ├── prompt/
            │   └── input.json        # (SolidWorks parts) frozen baseline measurements of the seed
            └── reference/
                ├── baseline.json     # (SolidWorks assembly) frozen baseline measurements of the seed
                └── *.png, *.stl      # (FreeCAD) frozen copy of the reference plus render/mesh per example

Per-repo differences worth knowing:

	test_example	test_example2	test_example3
Family / task	SolidWorks/1_playstation_controller	SolidWorks/8_mouse_wheel, SolidWorks/13_nanoindenter_stage	FreeCAD/4_sling_lift
Candidate format	.SLDPRT (part)	.SLDPRT (part), .SLDASM (assembly)	.FCStd
Seed	environment/input.SLDPRT	environment/input.SLDPRT, environment/input.SLDASM	environment/input.pdf (a drawing; there is no seed model)
Baseline file	tests/task/prompt/input.json	tests/task/prompt/input.json (mouse), tests/task/reference/baseline.json (stage)	none; tests/task/reference/solution.FCStd is the ground truth
harness.py	stub (empty envelope)	stub (empty envelope)	skeleton with ALL_CRITERIA, MUST_PASS, gate, finalize()
Adversarials	8	8 + 10	5
[metadata.examples] map	yes	yes	no
max_score	4	5, 10	6
Runs in Docker	no (needs Windows + SolidWorks via COM)	no	yes (docker/freecad-base.Dockerfile)

Current team rule differs: common/ library

The team rule assumes a shared common/ package (common.parametric, HB.task_weights, scoring.ramp, capture_schema.py, lint_harness.py, tools/parametric_check.py, tests/task/HISTORY.md). None of those files exist in any of the three repos. The READMEs say so explicitly: "Any shared measurement, SolidWorks-session or scoring code is yours to write" (test_example, test_example2) and "There is no shared library: the stub is self-contained" (test_example3). The only trace is docker/freecad-base.Dockerfile, which does COPY common /opt/common and sets PYTHONPATH=/opt:/opt/common for a common/ directory that is not in the checkout. Treat the take-homes as the shape of a task; the common/ helpers come from arwinai/test_eval_set.

The files and what each does

instruction.md

The Prompt, verbatim. It is short (3-10 lines), engineering-worded, and names the input and output files. From SolidWorks/8_mouse_wheel/instruction.md:

1. Add snap-fit tabs (qty 4, symmetric) between top and bottom shell
   halves. Target min 15N disengagement force.
2. Add two buttons that follow the design language of the mouse on the
   left where the user's thumb rests.
3. Overall mouse length is 95mm. Increase to 105mm ...
4. Make the radius of the wheel 10% bigger.

The starting part: `input.SLDPRT`, with frozen baseline measurements in `input.json`.

Save your final edited part as `solution.SLDPRT`.

Every number in the prompt (4 tabs, 105mm, 10%, 15mm, 60T, M8) is a grading target. The FreeCAD variant uses container paths: "The starting file(s): /app/input.pdf. Write your final document to /app/solution.FCStd."

task.toml

Schema 1.4. The sections you touch as a harness author:

• [task] — name = "storygold/<program>-<NNNN>-<slug>", version, description, keywords.
• [metadata] — program, category (design_iteration, geometric_reconstruction), units = "mm", human_baseline_minutes, max_score, accepted_output_formats, reference = ["solution/solution.SLDPRT"], and a free-text notes string. The notes field is where the three repos keep their design rationale and audit history (mirror plane location, which checks are "UNVERIFIABLE by design", the score ladder from the last audit, known gaps such as the nanoindenter reference's own rebuild error).
• [metadata.harness] — entrypoint = "harness/harness.py", timeout_s, requires_solidworks, requires_freecad, requires_llm.
• [[metadata.inputs]] — one table per input with path, format, role (primary, baseline_measurements, reference_image).
• [metadata.examples] — (SolidWorks repos) a map from each adversarial's path to a one-line description of its defect. This is the machine-readable checklist of what each example must lose.
• [[metadata.assets]] — one table per blob-stored file or folder: path, url, sha256, bytes, and for folders files plus a manifest = [[name, sha256, bytes], ...]. tools/sync.py writes these; tools/fetch.py and tests/test.sh read them.
• [verifier], [agent], [environment], [solution.env] — Harbor runtime config (timeout_sec, os, network_mode, build_timeout_sec).

Current team rule differs: where weights live

The team rule is task.toml [metadata.weights], read in the harness via HB.task_weights. None of the three task.toml files has a [metadata.weights] table. In the repos the weights live in the harness (ALL_CRITERIA in test_example3), and task.toml carries only the total: "keeping max_score in task.toml equal to their sum" (all three READMEs). When you build on test_eval_set, put the weights in [metadata.weights] and keep max_score equal to their sum.

environment/

The Before model (input.SLDPRT, input.SLDASM) or the drawing (input.pdf), its render (input.png), and the verifier Dockerfile. For SolidWorks the Dockerfile is a comment block explaining that SolidWorks "cannot be installed, licensed, or driven headlessly in a Docker container" and pointing at the Windows workflow. For FreeCAD it is two lines: FROM freecad-base:latest and COPY input.pdf /app/input.pdf.

solution/

The After reference (solution.SLDPRT, solution.SLDASM, solution.FCStd), renders (solution.png, and for the mouse solution_iso.png, solution_left.png), and solve.sh. In test_example3 solve.sh is the oracle Harbor runs to prove the task is solvable: cp /solution/solution.FCStd /app/solution.FCStd. In the SolidWorks repos it exits 1 with a TODO because there is no headless way to open a part.

examples/

One folder per adversarial. The SolidWorks repos put the native file, an .STL export and a .png render in each folder; the FreeCAD repo keeps the native .FCStd in examples/ and the .stl/.png for every example in tests/task/reference/. Two example folders in 13_nanoindenter_stage (adversarial_guard_fasteners_unreachable, adversarial_pulley_bore_diameter_changed) contain only a .gitkeep in git; their contents are pinned in task.toml and arrive via tools/fetch.py.

tests/test.sh and tests/test.bat

The Harbor verifier entrypoint. The FreeCAD version is the one to copy:

1. mkdir -p /logs/verifier
2. fetch_asset <url> <sha256> <dest> for every pinned file the harness needs (inline Python with hashlib + urllib.request), or, in 13_nanoindenter_stage, a loop over task.toml's [[metadata.assets]] manifests with tomllib.
3. If /app/solution.FCStd is missing, write {"reward": 0.0, "score": 0, "max_score": 0, "passed": false} to /logs/verifier/reward.json and exit 0.
4. python3 /tests/task/harness/harness.py "$CANDIDATE" > /logs/verifier/harness_result.json 2> /logs/verifier/harness_log.txt
5. Reduce the envelope to Harbor's flat reward.json (reward = score / max_score; only numbers and bools allowed).

test.bat exists only for the Windows tasks: python "%~dp0task\harness\harness.py" %*.

tests/task/harness/harness.py

The grader. The stub in the SolidWorks repos only prints an envelope; the FreeCAD skeleton defines the contract (next section). Both agree on the envelope shape: {"score", "max_score", "passed", "subscores"}.

tests/task/harness/measure_stage.py (FreeCAD)

Runs inside freecadcmd, not in the harness interpreter. harness.py's run_stage() launches it with FC_STAGE_INPUT (document to measure) and FC_STAGE_OUT (directory for measure.json) set, with a TIMEOUT_S = 600 timeout. measure_stage.py implements measure(doc) -> dict, and its run() wraps it: FreeCAD.openDocument, doc.recompute(), then json.dump of the result with ok=True, or {"ok": false, "error": traceback} on exception. The file's docstring is blunt about the constraint: "Stdlib and FreeCAD only ... Anything from numpy up belongs on the scoring side."

The README gives the two reasons to keep this split: "Your scoring logic stays testable on a machine with no FreeCAD installed. And a candidate's document never gets to run its own macros inside your interpreter."

This is the team's compute(read()) rule

The team rule to split the harness so that --score-from and --selftest run offline is the same design: an expensive, CAD-bound read (run_stage / build_state in the repo) that produces plain JSON, and a pure compute (checks + finalize) that scores it. Keep every CAD import on the read side.

tests/task/prompt/input.json and tests/task/reference/baseline.json

Frozen measurements of the seed, taken once with the real CAD program and committed so the harness can grade deltas from the seed without re-measuring it. 1_playstation_controller/tests/task/prompt/input.json (schema ps-annotation-baseline/1) records, in SolidWorks API units (metres), the mirror plane plane_x_m, the global bbox and width_m, per-body centroids/volumes/areas/inertia, geometrically assigned roles (housing, dpad, face_buttons, sticks, triggers, bumpers, centre), a housing_signature whose sign flips under a whole-part mirror, the light_cluster glyph areas and side, an interference census, and a rebuild error census (features: 199, errors: 0) "so the health gate grades candidates as a delta vs this". 13_nanoindenter_stage/tests/task/reference/baseline.json is the assembly equivalent: fixture_bbox_mm, the sorted multiset fixture_hole_diameters_mm, n_side_plate_screws, and the pulley's radius/flange/thickness/bore in mm. The task.toml notes record how it was frozen: harness.py --capture-baseline prompt/input.SLDPRT.

Targets come from the instruction and the seed, never the reference

No row may compare the candidate against the reference solution. The baseline files show the sanctioned alternative: measure the seed once, freeze it, and grade the candidate as "seed + what the instruction asks for" (width_m + 0.015, n_side_plate_screws -> 0, pulley_h635_radius_mm * 60/40). The reference must still score full marks, but it earns them by satisfying the instruction, not by matching itself.

Current team rule differs: test_example3 grades against the reference

4_sling_lift has no seed model (the input is a PDF), and its README says so: "You're comparing a candidate to a reference, not to an input." Its task.toml notes read "ground truth is reference/solution.FCStd (hardcoded)", tests/test.sh fetches tests/task/reference/solution.FCStd into the verifier, and the shipped component is literally named geometry matches ground truth. That is the opposite of the current team rule. For a reconstruction task today, pull the numeric targets (link lengths, hole spacings, overall height) out of the drawing and the checklist and put them in the harness as constants; do not open the reference at grade time.

tools/fetch.py and tools/sync.py

Repo-level, identical across repos except CONTAINER = "test-example" | "test-example2" | "test-example3" (and test_example's fetch.py adds a --skip-stl flag). fetch.py walks every task.toml, reads [[metadata.assets]], and downloads anything absent or stale (sha256 mismatch), reporting MISSING, LOCKED, STALE, CORRUPT. sync.py is the inverse: it hashes local *.SLDPRT/*.SLDASM/*.FCStd/*.stl/... under the family trees, uploads them with az/azcopy, and rewrites the sha256/bytes/manifest pins. Models never go in git; .gitignore in every repo lists the CAD extensions with the comment "CAD binaries live in Azure blob storage (see tools/fetch.py, tools/sync.py)".

README.md

Present tense, one per repo. Sections the three share: Deliverable (reference scores full, adversarials lose only their own component), First step / Setup (pip install -r env_requirements.txt, python3 tools/fetch.py), The task, Layout table, The harness (geometry only, continuous scoring, envelope shape), Validation/Running (exact commands over the reference and each example). test_example3 adds Harness architecture, Scoring with a good/not-so-good table, and an LLM judge section. The team rule adds a Known limits section; the repos keep that content in task.toml's notes instead (e.g. "symbol chirality is UNVERIFIABLE by design", "joint-limit ranges are effectively ungraded").

Writing criteria/rows

A row (the repos say "component" or "criterion") is one named, weighted check that returns a score in [0, 1] and a one-sentence description. The only real definition in the three repos is in FreeCAD/4_sling_lift/tests/task/harness/harness.py:

#: Weight per component; 0 makes it a gate when it is also in MUST_PASS.
#: The sum must equal task.toml's max_score (6 today).
ALL_CRITERIA = {
    "executes and builds geometry": 0,
    "geometry matches ground truth": 1,
    "pantograph joints mated": 1,
    "mechanism articulates": 1,
    "carries dimensional constraints": 1,
    "dimensions drive the geometry": 1,
    "signature dimensions present": 1,
}
MUST_PASS = ("executes and builds geometry",)
MAX_SCORE = 6

The contract between the row functions and the envelope:

def build_state(candidate_path):
    """Measure the candidate once; the checks read what this returns."""
    measurement, error = run_stage(candidate_path)
    return {"candidate": candidate_path, "measurement": measurement or {}, "error": error}


def checks(state):
    """One entry per component: {name: (score in [0, 1], description)}."""
    err = state["error"]
    out = {"executes and builds geometry": (
        1.0 if err is None else 0.0,
        "the measurement stage ran" if err is None else f"the measurement stage failed: {err}")}
    for name in ALL_CRITERIA:
        out.setdefault(name, (0.0, TODO))
    return out


def finalize(results):
    gated = any(results[n][0] < 1.0 for n in MUST_PASS)
    score = 0.0 if gated else sum(ALL_CRITERIA[n] * results[n][0] for n in ALL_CRITERIA)
    return {
        "score": round(score, 4),
        "max_score": MAX_SCORE,
        "passed": score >= MAX_SCORE,
        "subscores": {n: {"score": s, "weight": ALL_CRITERIA[n], "description": d}
                      for n, (s, d) in results.items()},
    }


def main(argv):
    candidate = argv[1] if len(argv) > 1 else None
    print(json.dumps(finalize(checks(build_state(candidate))), indent=2))
    return 0

How to go from an instruction to rows:

1. One row per requirement in the prompt, plus one for collateral damage. The mouse prompt has four numbered requirements; 1_playstation_controller's notes lists its rows by role: widen (mirror-pair separation +15 mm), cluster swap, text/glyph orientation (the "naive-flip guard"), and a rebuild-health row graded "as a delta vs" the seed's error census. adversarial_unrequested_change_elsewhere exists precisely to fund a "nothing else changed" row.
2. Name the row by what is measured, not pass/fail: pantograph joints mated, dimensions drive the geometry. The description returned beside the score should "say what was measured, not just pass or fail" (harness docstring).
3. Derive the target from the instruction, checklist or seed baseline. 105 mm from the prompt; width_m + 0.015 from input.json; n_side_plate_screws == 0 from baseline.json.
4. Measure geometry, not names. All three READMEs: grade "mass properties, centroids, face areas" and (for assemblies) "component placement", "never feature or body names — candidates may remodel freely". The PlayStation baseline assigns roles geometrically ("housing = largest body; the two 4-body diamonds found by X-Z proximity; dpad = the diamond whose bodies carry more sub-15mm2 faces").
5. Return a float. "A bool is accepted, and it's almost always the wrong answer."

Code style for rows (team rule)

One function per row, under about 45 lines, type hints everywhere, the measurement captured into a TypedDict/NamedTuple in capture_schema.py, compute(read()) split so --score-from <measure.json> and --selftest run with no CAD program present, and lint_harness.py plus mypy clean. The repo skeleton has no type hints and bundles every row into one checks(); use its contract (name -> (score, description), finalize builds the envelope) but write each row as its own typed function that reads the capture.

Current team rule differs: MUST_PASS

The repo ships MUST_PASS = ("executes and builds geometry",) with weight 0, and finalize() zeroes the entire score when it fails. The current team rule is MUST_PASS = (): a model that does not build should lose its rows through continuous scoring and gates on individual rows, not through a global zero. Keep the tuple, keep it empty.

Worked example: rows for 8_mouse_wheel

The table below turns the four numbered requirements of SolidWorks/8_mouse_wheel/instruction.md into rows, shows where each target comes from, and names the adversarial that proves the row fires. The ramp lines are a suggestion to show the shape; pick yours from the tolerance the instruction implies.

Instruction line	Row	Target source	Ramp (full credit / zero)	Proving adversarial
"snap-fit tabs (qty 4, symmetric) between top and bottom shell halves"	tabs on the seam, paired across it	seam plane from input.json; count 4 from the prompt	4 paired tabs on the seam / 0 paired	adversarial_snap_tabs_off_the_seam, adversarial_tabs_unpaired_across_the_seam
"two buttons ... on the left where the user's thumb rests"	thumb buttons present on the left flank	left-side bbox from input.json; count 2	both in the thumb zone / neither	adversarial_thumb_buttons_incorrectly_placed
"follow the design language of the mouse"	button form matches shell (judge or curvature row)	shell surface stats from input.json	measured curvature match or judge score	adversarial_thumb_buttons_alien_to_the_shell
"Overall mouse length is 95mm. Increase to 105mm"	overall length = 105 mm	95 and 105 from the prompt; baseline length in input.json	within 0.5 mm / 10 mm off	adversarial_mid_spacer_band (right length, wrong method: a band inserted at the seam)
"Make the radius of the wheel 10% bigger"	wheel radius = 1.10 x baseline, still circular	baseline wheel radius from input.json; 10% from the prompt	radius within 1% / 10% off; circularity 1.0 / eccentric	adversarial_elliptical_wheel, adversarial_slot_widened_past_the_wheel
(implicit) model rebuilds	rebuild errors delta vs seed census	rebuild.errors in input.json	0 new errors / N new errors	adversarial_feature_tree_errors

Two things to notice. adversarial_mid_spacer_band hits the right 105 mm, so a bare length row passes it; it is caught by a shell-continuity or seam-geometry measurement, which is why the README lists it as a near-miss. And adversarial_thumb_buttons_alien_to_the_shell is the one row in this task where a script struggles, which is the case for an LLM judge (requires_llm = true) or a measured proxy such as fillet radii and draft angles matching the shell.

Worked example: rows for 13_nanoindenter_stage

The assembly task shows how tests/task/reference/baseline.json feeds targets. Each of its fields is a seed measurement; the instruction says how each should change.

Instruction line	Row	Baseline field	Expected change	Proving adversarial
"low-profile belt guard that covers the belt"	belt run enclosed	belt path from the GT2 belt component geometry	belt swept volume inside guard	adversarial_guard_leaves_belt_exposed
"low-profile"	guard height above belt	belt bbox	height within a band / tall	adversarial_tall_boxy_guard_not_low_profile
"can be easily removed for re-tensioning"	guard fasteners reachable, removal does not require disassembly	fixture/enclosure bbox	tool access cones unobstructed	adversarial_guard_fasteners_unreachable, adversarial_guard_removal_requires_partial_disassembly
"Remove the side plate"	side plate absent	n_side_plate_screws = 8	0 screws, plate volume gone	adversarial_fixture_plate_extended_but_side_plate_remains
"extend the fixture plate to accommodate m8 mounting screws"	fixture plate longer, M8 clearance holes	fixture_bbox_mm, fixture_hole_diameters_mm	bbox grows; new holes at M8 clearance (~8.4-9 mm), not M6 (~6.4-6.6 mm)	adversarial_m6_mounting_holes_instead_of_m8
"Replace the 40T GT2 pulley on the stepper motor's shaft with a 60T"	motor-shaft pulley radius = 60/40 x baseline, bore unchanged	pulley_h635_radius_mm = 12.456, pulley_h635_bore_mm = 6.35	radius 18.68 mm on the motor shaft; driven shaft stays 12.456	adversarial_48t_pulley_substituted_for_60t, adversarial_60t_pulley_fitted_to_wrong_shaft, adversarial_pulley_bore_diameter_changed
(implicit) rebuilds	rebuild errors delta	n/a (assembly)	0 new errors	adversarial_feature_tree_rebuild_errors_dangling_references

The pulley row is a good illustration of the gate question. Tooth count is discrete: 48T is not "80% of a 60T", it is the wrong part, and the belt will not mesh at the intended ratio. That is a legitimate gate and the README should say so. Pulley radius measured off the mesh, on the other hand, is continuous and a ramp is the honest choice. The task.toml notes also record a hazard to expect: the reference solution.SLDASM's own Belt Guard-1 carries "a genuine hard rebuild error", so "the reference solution itself currently fails the 'model rebuilds without errors' must-pass gate". A gate that fails the reference is the reason the team rule asks for a delta-vs-seed health row and MUST_PASS = () instead.

Scoring

Continuous by default

Every README asks for degree, not pass/fail. From test_example3:

Situation	Good	Not so good
five joints wanted, three mated	0.6	0
a link 4% short of the drawing	0.9	0
two of four signature dimensions present	0.5	0

"Someone who gets most of it right should get most of the marks, and two candidates who are wrong by different amounts shouldn't land on the same number." test_example's README phrases the mechanism: a score "scaled by relative error against a tolerance".

The team's ramp

The standard form is two lines per measured value: the error at which the row still earns full credit, and the error at which it earns zero, linear in between, via scoring.ramp. For "+15 mm widen" that is something like full credit within 1 mm, zero at 15 mm off, so that adversarial_widened_by_30mm (15 mm over) lands at 0 on that row while a 16 mm widen keeps most of the mark. The repos provide no helper ("Write whatever scoring helpers you need (a clamp, an error-to-score curve, a ratio, a band); none are provided"); scoring.ramp from common/ is that helper.

Gates

A gate is a row with no middle. The repos use exactly one: executes and builds geometry, "the one thing a stub can answer honestly", and the README justifies it: "Save a hard 0 or 1 for the components that genuinely have no middle — 'does it rebuild without errors' is one". The PlayStation baseline softens even that by freezing the seed's rebuild census (errors: 0, warnings: 0) and grading the candidate as a delta.

When a gate is allowed

Use a pass/fail gate only when any error breaks the requirement (a pulley with the wrong tooth count does not partially drive the belt), and write the reason in the README. Everything else is a ramp.

Weights and the envelope

The score is the weighted sum of row scores, max_score is the sum of weights, passed is score >= MAX_SCORE, and subscores carries {score, weight, description} per row. Harbor then reads reward = score / max_score from reward.json. Weights in the repos are integers (1 each in the sling lift; the SolidWorks stubs only carry MAX_SCORE = 4 | 5 | 10). Weights can be uneven; the nanoindenter prompt has three requirements but max_score = 10.

Keep max_score consistent everywhere

The SolidWorks repos drifted: 1_playstation_controller/task.toml says max_score = 4, its harness.py says MAX_SCORE = 4, but tests/test.sh says "max_score": 5 and solve.sh says "scores 5/5". 8_mouse_wheel has 5 in task.toml and the stub, 8 in test.sh and solve.sh. With weights in [metadata.weights] read via HB.task_weights, there is one source of truth; do not hardcode the total anywhere else.

From envelope to reward.json

Harbor does not read the envelope directly. tests/test.sh in 4_sling_lift reduces it, and the comment in the script is the reason: "Harbor's reward.json schema requires every value to be a plain number (or bool) -- no nested objects or strings. The full breakdown (subscores, task_id, ...) stays in harness_result.json alongside this file."

max_score = result.get("max_score") or 0
score = result.get("score") or 0
reward = (score / max_score) if max_score else 0.0
payload = {"reward": reward, "score": score, "max_score": max_score,
           "passed": bool(result.get("passed"))}

If the candidate file is missing the script short-circuits with {"reward": 0.0, "score": 0, "max_score": 0, "passed": false} and exits 0, so a no-output run is a scored zero rather than a verifier crash. Keep this reduction as-is; the envelope is where detail belongs.

Parametric multiplier

Team rule: common.parametric

Tasks that ask for a parametric model carry a parametric row implemented with common.parametric, applied as a multiplier on the final score rather than as an additive row. The seed's parametric signature is frozen with python tools/parametric_check.py --references <task>. The sling-lift task is the one of the three that needs this ("driven by key dimensions rather than dead geometry"); the repo instead models it as three additive rows (carries dimensional constraints, dimensions drive the geometry, signature dimensions present) and its measure_stage.py docstring describes the probe: "whether changing one changes the geometry, which is a different question from whether it exists. ... A document can carry a tidy set of named constraints that drive nothing at all". adversarial_dummy_joints_inflate_the_count and the decorative_dims adversary mentioned in the notes are built to defeat a count-only check. Neither common.parametric nor tools/parametric_check.py exists in the repos.

LLM judge

Some rows cannot be measured cleanly: "Does this actually read as a pantograph?", "thumb buttons alien to the shell's design language", "low-profile" vs "tall boxy guard". test_example3's README: "If you decide a component is better judged than measured, you may call an LLM judge of your own. Make it raise rather than score when it can't reach a model: an unavailable judge shouldn't quietly become a pass." 4_sling_lift/task.toml sets requires_llm = true; env_requirements.txt pins anthropic==0.69.0 and pydantic==2.12.3 and mentions a common/llm_judge.py that is not in the checkout, configured through AZURE_API_KEY and AZURE_CLAUDE_RESOURCE in a root .env. The SolidWorks tasks set requires_llm = false and leave design-language defects to geometry where possible; the nanoindenter notes lists what is "NOT machine-graded" (tool-free attach/detach, manufacturability).

Judge rules

The judge can only lower its row, never raise a measured score. Set requires_llm = true in task.toml whenever any row calls it, and make the row raise (not silently pass) when the model is unreachable. The judge must still let --selftest run offline: feed it from the frozen capture or skip it under --selftest with the row's deterministic floor.

Adversarial examples and tests

What an adversarial is

One model, one defect, named after the defect. The three repos together show the canonical defect families:

Family	Examples
Wrong magnitude	adversarial_widened_by_30mm, adversarial_m6_mounting_holes_instead_of_m8, adversarial_48t_pulley_substituted_for_60t
Right change, wrong place	adversarial_60t_pulley_fitted_to_wrong_shaft, adversarial_thumb_buttons_incorrectly_placed, adversarial_snap_tabs_off_the_seam
Partial completion	adversarial_only_one_button_cluster_mirrored, adversarial_fixture_plate_extended_but_side_plate_remains, adversarial_half_finished_missing_features
Right geometry, wrong structure	adversarial_dummy_joints_inflate_the_count, adversarial_joint_limits_swapped, adversarial_feature_tree_with_errors
Shortcut that fakes the result	adversarial_text_mirrored_incorrectly (naive mirror), adversarial_elliptical_wheel, adversarial_mid_spacer_band (lengthened by a spacer, not a remodel)
Collateral damage	adversarial_unrequested_change_elsewhere
Secondary requirement missed	adversarial_missing_glyphs, adversarial_pulley_bore_diameter_changed, adversarial_guard_leaves_belt_exposed

The README of test_example3 explains the purpose of each: "The first two catch a grader that never really compares against the drawing ... dummy_joints catches a grader that counts joints instead of asking what they do. joint_limits_swapped catches one that checks joints exist and are mated but never asks what they permit."

The ladder: how example scores must relate

• The reference scores max_score.
• Each adversarial loses only its own row: "every entry under examples/ loses points on the component(s) it actually gets wrong, and only those."
• No two example totals tie. Different defects must produce different totals so a regression in one row is visible from the ladder alone.
• The untouched seed scores 0. Running the harness on environment/input.* must give 0: every row's target is a change from the seed, so the seed earns nothing.

The repos' recorded ladder has a tie

4_sling_lift/task.toml notes record the last audit: "solution 6 > half_finished 5 = joint_limits_swapped 5 > mirrored 4 > one_link_missing 3 > decorative_dims 2.5 > dummy_joints 2 > double_scale 0.5 > build-fail 0/6". Two adversarials tie at 5. Under the current rule that is a defect to fix by reweighting or by splitting a row so the two defects are distinguishable. The same notes mention adversaries (decorative_dims, double_scale, broken_mates, dead_geometry, no_joints, partial_weld, rigid, wrong_geometry) that are not in this checkout's examples/.

How adversarials are wired in

The repos wire adversarials in three places:

1. task.toml [metadata.examples] (SolidWorks repos): "examples/adversarial_<defect>/adversarial_<defect>.SLDPRT" = "<one-line defect>". This is the list a self-test iterates and the description a reviewer reads.
2. task.toml [[metadata.assets]]: one folder pin per example with a manifest of [name, sha256, bytes], so tools/fetch.py can materialise them.
3. README.md Validation / Test corpus: a table of each example and what is wrong with it, and the exact commands to run the harness over the reference and each example.

What the repos do not have is an automated self-test; validation is "run it over the reference and every .SLDPRT under examples/" by hand and paste the envelopes into your notes. The team's --selftest formalises that loop: load the frozen capture of the reference, the seed and every examples/adversarial_*, run compute() on each, and assert the ladder (reference full, seed 0, each adversarial short only on its row, no ties).

The judgement calls

Not every adversarial is unambiguous, and the repos say so rather than hide it. adversarial_assembly_mirrored_relative_to_ground_truth: "is the mirror a defect, or is our reference just one of two valid answers? ... Make a call and defend it." Under the no-reference-comparison rule a mirrored-but-correct lift must score full marks unless the drawing fixes handedness, so that example either needs a real defect or should be dropped. adversarial_joint_limits_swapped is "convicted by geometry at the delivered pose, not by the articulation probe", which the notes call a "known attribution nuance". Record these in the README's Known limits.

The .png and .STL beside each example

The renders are committed so "you can see the whole corpus first" before fetching gigabytes of models. The .STLs let the scoring side (trimesh, manifold3d, shapely are all pinned in env_requirements.txt) check mesh-level facts without the CAD program. In test_example3 the notes warn that pinned STLs can "predate these docs' current exports and tessellate differently; the FCStd is the graded artifact".

Running and self-testing

Fetch assets

pip install -r env_requirements.txt      # pywin32 installs on Windows only
python3 tools/fetch.py                   # everything missing or stale
python3 tools/fetch.py FreeCAD           # one subtree
python3 tools/fetch.py --check           # report, download nothing
python3 tools/fetch.py --skip-stl        # test_example only: skip the large meshes

Run the harness (FreeCAD, headless)

export FREECAD_CMD=/usr/bin/freecadcmd   # or put FREECAD_CMD=... in a .env at the repo root
cd FreeCAD/4_sling_lift
python3 tests/task/harness/harness.py solution/solution.FCStd
python3 tests/task/harness/harness.py examples/adversarial_joint_limits_swapped/adversarial_joint_limits_swapped.FCStd

The output is the envelope, e.g. (stub, before any rows are written):

{
  "score": 0.0,
  "max_score": 6,
  "passed": false,
  "subscores": {
    "executes and builds geometry": {"score": 1.0, "weight": 0, "description": "the measurement stage ran"},
    "geometry matches ground truth": {"score": 0.0, "weight": 1, "description": "not implemented -- see the docstring at the top of harness.py"}
  }
}

Run the harness (SolidWorks, Windows)

SolidWorks is driven over COM (win32com.client.Dispatch("SldWorks.Application")) and needs a running, licensed session. With no argument the harness grades the active document.

cd SolidWorks\1_playstation_controller
python tests\task\harness\harness.py solution\solution.SLDPRT
python tests\task\harness\harness.py examples\adversarial_widened_by_30mm\adversarial_widened_by_30mm.SLDPRT
python tests\task\harness\harness.py                 REM grades whatever is open in SolidWorks
tests\test.bat solution\solution.SLDPRT              REM same thing via the Windows verifier entrypoint

cd SolidWorks\13_nanoindenter_stage
python tests\task\harness\harness.py solution\solution.SLDASM
python tests\task\harness\harness.py examples\adversarial_m6_mounting_holes_instead_of_m8\adversarial_m6_mounting_holes_instead_of_m8.SLDASM

Freeze a baseline

python tests\task\harness\harness.py --capture-baseline prompt\input.SLDPRT    REM from 1_playstation_controller's task.toml notes

Docker (FreeCAD only)

docker build -f docker/freecad-base.Dockerfile -t freecad-base:latest .
docker build -f FreeCAD/4_sling_lift/environment/Dockerfile -t sling-lift FreeCAD/4_sling_lift

Inside the container tests/test.sh expects the candidate at /app/solution.FCStd and writes /logs/verifier/reward.json, /logs/verifier/harness_result.json and /logs/verifier/harness_log.txt.

Self-test offline

Team rule: the offline loop

Because the harness is split into read() (CAD) and compute() (pure Python), the scoring side runs anywhere:

python tests/task/harness/harness.py --score-from measure.json      # score a saved capture
python tests/task/harness/harness.py --selftest                     # replay frozen captures of reference, seed and every adversarial; assert the ladder
python tools/parametric_check.py --references <task>                # freeze the seed's parametric signature
python lint_harness.py tests/task/harness/harness.py && mypy tests/task/harness

The repos do not have --score-from or --selftest; they have the pieces (run_stage writes measure.json to a temp dir and checks(build_state(...)) reads a dict), so wiring the flags is: keep the measure.json instead of deleting it, and let --score-from skip run_stage.

Step by step: from an empty task folder to review

The order below is the one the repos imply: freeze what the harness will compare against before writing any check, and build the adversarials before the rows so every row has something to fail.

1. Branch. git checkout -b <you>/<task> in arwinai/test_eval_set.
2. Lay out the task folder as in the tree above: <Family>/<N>_<slug>/{instruction.md, task.toml, environment/, solution/, examples/, tests/task/harness/, tests/test.sh}. Copy task.toml from the nearest sibling and change [task] name, description, keywords, [metadata] category, human_baseline_minutes, accepted_output_formats, reference.
3. Write instruction.md in the style of the repos: numbered requirements, concrete numbers, the input file and the output file name.
4. Place the seed and the reference. environment/input.<ext> and solution/solution.<ext>, each with a .png render. Confirm the reference satisfies every line of the instruction before anything else; the nanoindenter notes show what happens when it does not (its own Belt Guard-1 has a rebuild error and fails the health gate).
5. Build 5-10 adversarials, one defect each, at examples/adversarial_<defect>/adversarial_<defect>.<ext> plus .png. Pick defects from the families in the table above: wrong magnitude, right change in the wrong place, partial completion, right geometry with wrong structure, a shortcut that fakes the result, collateral damage. Add each to [metadata.examples] with a one-line description.
6. Freeze the seed. Capture the baseline (harness.py --capture-baseline ... in the PlayStation notes) into tests/task/prompt/input.json or tests/task/reference/baseline.json, and register it under [[metadata.inputs]] with role = "baseline_measurements". For a parametric task, also python tools/parametric_check.py --references <task>.
7. Write the capture schema. A TypedDict/NamedTuple per measured thing in capture_schema.py; read() fills it from the CAD program (or measure_stage.py under freecadcmd) and writes it as JSON.
8. Write the rows. One typed function per row, under ~45 lines, each returning (score, description). Targets from the instruction, checklist and baseline only. Ramps via scoring.ramp; gates only where any error breaks the requirement, with the reason written down. Weights into task.toml [metadata.weights], read with HB.task_weights. MUST_PASS = ().
9. Wire compute(read()) so that harness.py <candidate> does both, --score-from <json> does only compute, and --selftest replays frozen captures of the reference, the seed and every adversarial.
10. Run the ladder. Reference at max_score, seed at 0, each adversarial short only on its row, no ties. Fix rows and weights until it holds; record the final totals.
11. Upload models and pin them. python3 tools/sync.py <Family>/<N>_<slug> uploads to blob and rewrites [[metadata.assets]]; python3 tools/fetch.py --check should then report nothing missing. Models stay out of git.
12. Write the README (present tense, Known limits) and the tests/task/HISTORY.md entry; make lint_harness.py and mypy clean.
13. Push and ping Igor with the envelopes for the reference and every example in the PR description.

Mistakes the corpus is built to catch

The adversarials across the three repos are a catalogue of grader bugs as much as modelling bugs. Each row below is a way a harness can be wrong, and the example that exposes it.

Grader mistake	Exposed by
Checking that the output exists and is well-formed, and stopping there	adversarial_half_finished_missing_features, adversarial_one_link_missing_joints_bridge_gap
Counting things instead of asking what they do	adversarial_dummy_joints_inflate_the_count ("Plenty of joints. None of them constrain anything.")
Checking existence of constraints but not whether they drive geometry	decorative_dims (in the sling-lift notes: "25 decoy constraints pass the count; volume-changed probes convict")
Passing every shape metric because everything scaled together	double_scale (in the notes: "every shape metric passes at 2x; only the ratio gate convicts")
Grading the final dimension without checking how it was reached	adversarial_mid_spacer_band (105 mm via an inserted band), adversarial_slot_widened_past_the_wheel
Mirroring by a naive geometric flip and calling it a layout swap	adversarial_text_mirrored_incorrectly, guarded by the housing_signature.sign and light_cluster.side in input.json
Checking one half of a symmetric requirement	adversarial_only_one_button_cluster_mirrored, adversarial_tabs_unpaired_across_the_seam
Measuring the right change on the wrong body	adversarial_60t_pulley_fitted_to_wrong_shaft
Ignoring secondary attributes of the changed part	adversarial_pulley_bore_diameter_changed, adversarial_missing_glyphs
Not looking at anything the instruction did not mention	adversarial_unrequested_change_elsewhere
Trusting the feature tree as evidence of health	adversarial_feature_tree_with_errors, adversarial_feature_tree_rebuild_errors_dangling_references
Comparing pose to the reference and failing a valid mirror image	adversarial_assembly_mirrored_relative_to_ground_truth
Reading object or feature names	every adversarial, since the READMEs forbid name-based grading and candidates "may remodel freely"

Three of those (decorative_dims, double_scale, the ratio gate) exist only in the task.toml notes of test_example3, not in its examples/. They are still worth building for any parametric task.

Review checklist before pinging Igor

Work on a branch <you>/<task> in arwinai/test_eval_set; push everything except the models. Do not run tools/sync.py or edit the blob pins yourself: upload new or rebuilt models to your Google Drive, make sure the link is shared, and send it to Igor, who uploads to blob and re-pins task.toml. Then check:

Task assets

• environment/input.* (Before), instruction.md (Prompt), solution/solution.* (After), checklist, and 5-10 examples/adversarial_<defect>/adversarial_<defect>.<ext> with matching folder and file names, one defect each.
• .png render beside every model; renders committed, models pinned in [[metadata.assets]] and absent from git (.gitignore covers the CAD extensions).
• python3 tools/fetch.py --check reports nothing missing.

task.toml

• [metadata.weights] present; max_score equals their sum; no other copy of the total in test.sh, solve.sh or the harness.
• [metadata.harness] has the right entrypoint, timeout_s, requires_solidworks / requires_freecad, and requires_llm = true iff a judge is called.
• [metadata.examples] lists every adversarial with a one-line defect.
• [[metadata.inputs]] names the seed and the baseline file.

Harness

• No row opens or measures the reference; targets come from the instruction, checklist, environment files and seed baseline.
• Rows are continuous (scoring.ramp, full-credit and zero lines) except documented gates; MUST_PASS = ().
• Parametric row (if any) via common.parametric as a multiplier; seed frozen with tools/parametric_check.py --references <task>.
• Judge rows can only lower; they raise when the model is unreachable.
• Geometry only: no body, feature, sketch or object names anywhere in a check.
• Weights read via HB.task_weights; MAX_SCORE not hardcoded.
• Type hints everywhere; captures as TypedDict/NamedTuple in capture_schema.py; one function per row, under ~45 lines; compute(read()) split; lint_harness.py and mypy clean.

Ladder

• Reference scores max_score.
• Untouched seed scores 0.
• Each adversarial loses only its own row.
• No two example totals tie.
• --selftest passes offline; --score-from works on a saved capture.

Docs

• README in present tense: what is graded, why each gate is a gate, exact run commands, a Known limits section (the repos' equivalents: "symbol chirality is UNVERIFIABLE by design", "joint-limit ranges are effectively ungraded", "NOT machine-graded: tool-free attach/detach").
• tests/task/HISTORY.md has an entry for this change (the repos keep this in task.toml notes as dated AUDIT lines; move it to HISTORY.md).
• Envelopes for the reference and every example pasted into the PR description.

Then ping Igor for review.

Glossary

Term	Meaning in the repos
Task	One folder <Family>/<N>_<slug>/ with instruction.md, task.toml, environment/, solution/, examples/, tests/.
Seed / Before	environment/input.*; the model the candidate starts from. Frozen into input.json or baseline.json.
Reference / After	solution/solution.*; must score max_score.
Candidate	The file passed as argv[1] to harness.py (or the active SolidWorks document when omitted).
Row / component / criterion	One named weighted check returning (score in [0, 1], description). Keys of ALL_CRITERIA.
Gate	A row with weight 0 in MUST_PASS that zeroes the total when it fails (repo); under the team rule, a per-row pass/fail with a documented reason.
Ramp	Linear map from measurement error to [0, 1] between a full-credit line and a zero line (scoring.ramp).
Envelope	{"score", "max_score", "passed", "subscores"} printed by finalize().
reward.json	Harbor's flat {"reward", "score", "max_score", "passed"} written by tests/test.sh.
Adversarial	examples/adversarial_<defect>/; a near-miss with exactly one defect, used to test the harness.
Ladder	The ordered list of totals: reference, each adversarial, seed. Must be strictly decreasing with no ties and end at 0.
Baseline	Frozen seed measurements: tests/task/prompt/input.json (parts) or tests/task/reference/baseline.json (assembly).
Measurement stage	measure_stage.py run under freecadcmd with FC_STAGE_INPUT/FC_STAGE_OUT; writes measure.json.
Asset pin	A [[metadata.assets]] table (path, url, sha256, bytes, optional manifest) that tools/fetch.py materialises.
Harbor	The runner that builds environment/Dockerfile, runs solution/solve.sh as the oracle and tests/test.sh as the verifier.

Reference repos

• arwinai/test_example — SolidWorks/1_playstation_controller: part task, 8 adversarials, tests/task/prompt/input.json baseline with geometric role assignment and a mirror-signature guard, stub harness.
• arwinai/test_example2 — SolidWorks/8_mouse_wheel (part, 8 adversarials) and SolidWorks/13_nanoindenter_stage (assembly .SLDASM, 10 adversarials, tests/task/reference/baseline.json, manifest-driven test.sh), stub harnesses.
• arwinai/test_example3 — FreeCAD/4_sling_lift: headless, 5 adversarials, the ALL_CRITERIA / MUST_PASS / build_state / checks / finalize skeleton, measure_stage.py under freecadcmd, requires_llm = true, Harbor test.sh writing reward.json, docker/freecad-base.Dockerfile.