results/reviewer_form.md

# Reviewer Form: Synthetic Mine Throughput

**Submission:** `2026-04-27__001_synthetic_mine_throughput__gsd2__gemini-3-1-pro-preview__customtools`
**Reviewer:** Independent human reviewer (opus subagent)
**Date:** 2026-04-27

## Automated report

- Automated report file: `results/evaluation_report.json`
- Runtime seconds: not recorded (null)
- Python LOC: 338 code lines (single `sim.py`, 415 total lines)
- Required scenarios present: all 6
- Behavioural checks passed: 53/53
- Token usage method: not supplied

## Human quality score

| Category | Max | Score | Notes |
|---|---:|---:|---|
| Conceptual modelling | 20 | 15 | `conceptual_model.md` is clear and well-structured (system boundary, entities, resources, events, state, assumptions split into derived vs introduced, plus limitations and performance measures). It correctly enumerates which constrained edges are modelled as resources. Loses points because entities are minimal (only "trucks" — payloads not separately treated), state variables are listed only in skeletal form, and there is no discussion of warm-up handling or steady-state behaviour. |
| Data and topology handling | 15 | 12 | The graph is built from `edges.csv` with `nx.DiGraph`, weighted by base travel time; routes are computed with Dijkstra (`sim.py:65-70`) and re-evaluated on dispatch. Capacity-constrained edges (`capacity<999`) are turned into SimPy resources (`sim.py:88-90`), and `edge_overrides` correctly close/upgrade edges via scenario YAML (`sim.py:46-53`). Slight deductions: `closed=true` parsing relies on string check rather than robust YAML/CSV bool handling; the WASTE/MAINT nodes are present but never modelled as alternatives; the routing never touches E03 even in baseline (bypass is faster), so the model "incidentally" handles ramp closure rather than from a robust topology perturbation — a more sophisticated reviewer would note this fragility. |
| Simulation correctness | 20 | 15 | Genuine SimPy DES: trucks are processes (`run_truck`), loaders/crusher/constrained roads are `simpy.Resource` (capacities sourced from data), and tonnes are recorded per completed dump (`sim.py:269`). Truck cycle: travel-empty → load → travel-loaded → dump. Constrained edges are correctly held during the timeout. Concerns: (1) `edge_resources` is keyed by `edge_id` only, so the same resource is used for both directions only if edge_ids differ — correct here, but fragile. (2) When the empty-truck routing loop finds no path it `break`s silently rather than failing loudly as the prompt asks. (3) Cycle-time semantics include the initial PARK-to-loader leg, biasing the first cycle. (4) The "ramp_closed = baseline exactly" outcome arises because baseline trucks already prefer the bypass (E03 was never on the chosen Dijkstra path) — the model is technically correct but never exercises ramp logic; this would not be caught without inspecting the event log. |
| Experimental design | 15 | 11 | 30 replications per scenario, all 6 required scenarios, deterministic seeds (`base_seed + rep`), 95% CI computed with t≈1.96 SE (`sim.py:379-382`). Stochasticity applied to load, dump, and travel (CV 0.10) using `numpy.default_rng`. Common Random Numbers across scenarios (same `base_random_seed: 12345`) is good practice for variance reduction. Loses points: warm-up declared `0` in baseline but never discussed/justified given an 8-hour shift; no additional scenario despite README naming one ("Upgrade Crusher") — the prompt allows it, the agent listed it but didn't run it; CI uses normal rather than t-distribution at n=30 (minor); no sensitivity beyond the supplied scenarios. |
| Results and interpretation | 15 | 12 | All six decision questions are answered concisely in the README with numerical evidence and operational reasoning (crusher is the binding constraint, system saturates at ~12.5kt, ramp not a bottleneck, ramp closure absorbed by crusher buffering). Numbers in `summary.json` align with `results.csv`. The interpretation that ramp closure has "virtually no impact" is technically supported but should have been flagged as a routing-quirk artefact (E03 was never used even in baseline). No quantified bottleneck ranking populated in `top_bottlenecks` (left empty in summary.json). Minor overclaim in trucks_12 reading ("explodes" for 15-min queue is reasonable). |
| Code quality and reproducibility | 10 | 6 | Single 415-line `sim.py` with all responsibilities (data loading, graph building, simulation, experiment loop, output writing) in one file — opposite of the "many small files" guideline. Hard-coded relative paths (`'data/nodes.csv'`, output to CWD) mean it must be run from the submission root. No type annotations, no docstrings, no logging (uses `print`), no CLI arguments, no `requirements.txt` or `pyproject.toml`. README install instruction is clear. Variable names are reasonable. Reproducibility is functionally adequate via seed control. |
| Traceability and auditability | 5 | 4 | `event_log.csv` contains 256k rows across all replications/scenarios with the required columns (time_min, replication, scenario_id, truck_id, event_type, from/to/location, loaded, payload, resource_id, queue_length). Truck movements can be reconstructed end-to-end (verified by tracing T01 in baseline and ramp_closed). Loses a point because `from_node`/`to_node` are blanked for non-travel events (they could carry the resource node), and there is no separate per-replication summary or visualisation derived from the log. |
| **Total** | **100** | **75** | |

## Automated context

All 53 automated checks pass, including all 6 behavioural sanity checks (trucks_12 > trucks_4, ramp_upgrade ≥ baseline, crusher_slowdown < baseline, ramp_closed ≤ baseline, saturation plausible). No bonus/penalty adjustment indicated; runtime and token usage are unrecorded so no efficiency context.

**Final score: 75 / 100**

## Top 3 strengths

1. **Genuine SimPy DES with correct resource modelling**: trucks are active processes, loaders/crusher/narrow roads are SimPy `Resource`s with capacities driven from data, and tonnes are recorded only on completed dump events.
2. **Sound experimental hygiene**: 30 reps × 6 scenarios with reproducible seeds, 95% CIs in `summary.json`, stochastic service and travel times via truncated normal, and Common Random Numbers across scenarios.
3. **Clear, decision-focused interpretation**: README answers all six operational questions with concrete numbers and the correct top-line insight that the crusher (~91% utilisation, queue time growing dramatically under crusher_slowdown) is the binding constraint.

## Top 3 concerns / gaps

1. **Ramp scenarios essentially no-ops because of routing geometry**: in baseline the Dijkstra path J2→J7→J5 (bypass) is already shorter than via E03_UP, so the narrow ramp is never traversed. The agent did not detect or comment on this; results for `ramp_closed` and `ramp_upgrade` are therefore byte-identical (or near-identical) to baseline, which is technically correct but a notable modelling blind spot for a decision-support artefact.
2. **Single monolithic file with poor separation of concerns**: `sim.py` mixes I/O, graph building, simulation, experiments, and output serialisation in one 415-line module with hard-coded paths, no type hints, no CLI, and no dependency manifest — runs only from the submission root.
3. **Soft failure modes and missing rigour**: silent `break` when no path exists (prompt explicitly asks for clear failure), no warm-up justification, `top_bottlenecks` left empty in summary.json, the proposed "Upgrade Crusher" scenario was named but never executed, and the conceptual model entity list is thin.

## Final recommendation

**Marginal-to-solid submission.** The simulation is correct, reproducible, and gives the operator the right top-line answer (crusher is the bottleneck). However, code organisation is weak, the ramp scenarios coincidentally produce no signal because of how baseline routing already avoids the ramp — and the agent did not catch or report this artefact. Trust this **partially** as a first-pass decision-support artefact: enough to focus management attention on the crusher, but a code refactor and an explicit re-examination of when E03 is actually used should happen before relying on the ramp-related conclusions.