The Harness: Engineering Notes, Quantified

The research story lives in analysis/*.md. This document is about the machine that produced it, the data-collection, labeling, and transport infrastructure , with every claim measured from the repo, the logs, and the archive. First measured 2026-07-06; all figures re-measured 2026-07-14 (addenda dated below).

What it is

One detector, many collectors. The detector is a linear probe that consumes (features, label) rows and reports AUROC; it is deliberately harness-agnostic. Everything else, four generations of collectors, a three-model LLM jury, an unattended model×corpus grid scheduler, and a self-draining mirror with off-site backup, exists to feed it labeled rows at scale without a human in the loop.

The pipeline spans five systems (the fifth added 2026-07-10, see the addendum):

By the numbers

Code

The harness is many small, single-purpose tools sharing file-format contracts (CSV + npz), not a framework.

Data

Coverage

That last row is the core economic decision: labels are a property of the claim text vs ground truth, not of the model. Teacher-forcing the same claims through each new model means the jury bill (the slow, rate-limited, quota-capped stage) was paid one time, and the marginal cost of a 26th model is GPU-hours only.

The transport layer: auto_mirror.sh

The M2 Ultra produces ~2 GB npz files faster than its disk should accumulate them, and the remote box is a loaner with a deadline. auto_mirror.sh is a 59-line daemon that drains it continuously:

pull (rsync) → hash-verify → back up to Drive (rclone, hash-verified) → delete remote
    

The ordering is the invariant: every file exists in ≥2 places at every step of its life. Measured from its own log (lifetime, re-measured 2026-07-14):

• 177 files / 183.3 GB pulled, verified, backed up, and remote-deleted
• 0 bytes lost. Every verification failure along the way (10 as of the first measurement) was the gate working as designed: the collector was still writing the npz, the hash didn't match, the remote copy was kept and retried on the next cycle. No failure required human intervention. (One corrupt npz did reach Drive in July, an ENOSPC mid-savez on the producer side, upstream of the mirror; it was caught at score time, quarantined, and the queue's done-markers became rc-gated so a failed write can never mark itself done.)
• Drive backup was originally a human-run command and silently went ~18 h stale once; it was folded into the daemon (commit 633bae5) so staleness is now structurally impossible.

The label oracle: the Groq jury

Human labels are the source of truth (qa_grading.xlsx); the jury audits and extends them, and never silently overwrites a human judgment. Design points, each earned by a failure:

• Three cross-vendor jurors, majority vote; 2-1 splits and abstains flagged for human review. 98.7% human agreement on the M1 audit, and the audit caught a juror hallucinating a fact to grade an answer, which is why humans stay final authority. The lineup has turned over under quota pressure (current: Meta MoE @Groq, OpenAI gpt-oss-120b @Cerebras, zai-glm-4.7 @Cerebras-paid); every seat change is bridge-validated against the outgoing juror's stored verdicts on an overlap slice before the new seat's votes count (96.6% and 95.3% binary agreement on the two switches to date).
• Wikipedia-grounded grading for long-form claims: the jury reads the cached article (armb_wiki_cache.json), not its own memory. No self-grading, no within-family grading.
• Resumable + checkpointed: every grader can be killed at any point and re-run; 56k verdicts were accumulated across daily free-tier quota windows.
• Failure-mode handling in code, not lore: gpt-oss 400s on JSON mode ~8-11% of the time → automatic retry with text-parse fallback. Reasoning jurors burn variable tokens before emitting JSON → per-juror token budgets (glm thinks ~7× longer than gpt-oss and gets ~7× the budget). Groq has a per-day cap, not just TPM → throttled workers (--workers 3 --throttle 0.4) after one 6-worker burst exhausted the org's daily quota mid-run, and per-model daily caps, discovered when one juror's 100k TPD turned a re-grade into a weeks-long drip (the grade chain became a 3-hourly retry loop that harvests whatever quota rolled off, continues past per-model storms, and exits only on full completion). Cerebras daily-cap 429s carry multi-hour Retry-After values that looked like socket hangs until the client capped them at 60s. A storm-abort guard (15 consecutive all-abstain calls → clean exit 3) keeps a dead-quota session from poisoning checkpoints with abstains.

The collection layer: four backends, one schema

Each milestone forced a backend change, and each change kept the downstream contract stable so no analysis code had to be rewritten:

1. Ollama top-10 logprobs (M1), cheap, already built; 1 feature.
2. llama.cpp full-vocab (M2), 152k-logit rows, byte-reproducible at temp 0. Negative result, but it proved the output side was exhausted, and revealed llama.cpp cannot expose intermediate layers, forcing the pivot.
3. HF transformers residual stream (M3), per-layer activations at selected token positions, fp16 on MPS. One collector (run_armb_teacherforce.py, 208 lines) handles all 25 panel models including base-mode (--raw), Mamba, and MoE variants.
4. Multi-turn crescendo (run_crescendo.py, 181 lines), history-carrying N-turn collection, with fallbacks for architectures (Gemma-4) that break the standard config path. Extended to a live adaptive attacker (run_crescendo_adaptive.py): a second model reads the defender's answers and escalates in a closed loop, 12 defender architectures collected unattended.
5. K-sample resampling (run_semantic_entropy.py), the semantic-entropy baseline collector: 10 sampled generations + sequence logprobs per prompt, same prompt build as the activation collectors so the existing greedy-answer labels apply with zero new grading.

The unattended grid run is the payoff for the schema discipline: 65 model×corpus cells scheduled by 64 lines of shell, each producing an npz that the analysis side scores with a one-line registry entry, while the mirror drains the disk behind it and the operator sleeps.

Scientific hygiene, enforced by tooling

• Pre-registration: two experiments (PREREG_gold_transfer.md, PREREG_freshrep.md) had their pass/fail bars committed to git before data collection. One recorded an honest near-miss; the other passed all 5 bars.
• Falsification gates: the pipeline caught its own inflated result once (0.98 cross-regime → 0.85–0.87 after the within-regime control) and the correction is the headline number, not a footnote.
• Confound controls as standing checks: answer-length, entity-fame (within-bio splits), style artifacts (an n-tokens-only baseline exposed a "perfect" HaluEval number as an artifact, flagged do not cite in the write-up), and label-convention sensitivity (clean vs collapsed reported side by side).
• Label desync made impossible-ish: labels are edited in exactly one place (qa_grading.xlsx) and flow through sync_labels.py; the two rows that desynced before this rule existed are named in CLAUDE.md as the reason.
• Public retraction discipline: a published panel finding ("distillation degrades the signal, 0.811") was traced to a transformers-5.x tokenizer bug that silently dropped every space from one model's inputs. The finding was retracted in place (the write-up carries the ⚠️ block, not a silent edit), the clean re-run (0.885) told a duller, truer story, and a round-trip tokenizer guard (tok_fix.py) now runs in every collector so the class of bug cannot recur silently.

The one-sentence version

In 26 days and ~19,500 lines of small, resumable scripts, this harness moved ~200 GiB of activations across 25+ models and a dozen labeled corpora, issued ~75,000 jury verdicts against Wikipedia ground truth with humans as final authority, lost zero bytes across 183 GB of automated pull-verify-archive cycles, and turned "test a new model" into a one-line registration, which is what made a 25-row cross-architecture panel affordable at all.

Addendum (2026-07-10): the two-agent layer

Everything above described one agent (Claude Code, in this repo) driving the pipeline. The project now runs as two AI agents with disjoint capabilities, coordinating exclusively through Google Drive, no shared filesystem, no live channel, no human relaying state between them.

What Claude Science is

Claude Science is a second, independent Claude instance running in the claude.ai app. Its only I/O is a Google Drive connector with a 10 MB per-file download cap; it has no shell, no repo checkout, and no access to the GPU box. It serves two roles the box-side agent shouldn't hold alone: independent analysis (it re-derives probe numbers from raw activation slices with its own code, the 0.932-vs-0.934 config discrepancy it caught is exactly the class of drift a second implementation exists to catch) and experiment design (it writes collection scripts and specs offline; Phase 2 of the self-generation study runs entirely on its side). Claude Code (this repo's agent) remains the sole executor: it holds the box, the repo, rclone, and the collection queue, and is the explicit gatekeeper, nothing Claude Science drops can divert the box without a deliberate decision here.

The Drive protocol: three contracts, no relay

Casey used to paste status prose between the two agents by hand. Three small file contracts (all committed as specs) eliminated that in both directions:

The remaining human bandwidth between the agents is decisions only, the first live exchange over this protocol (Phase 0, below) needed exactly two relayed messages: the initial file handoff and the one-time STATUS.json id.

First production run of the protocol: Phase 0 self-generation

The probe's flagship numbers are teacher-forced; the obvious critique is that self-generated text might read differently. Claude Science designed the experiment and wrote the collector blind (never having seen the box); Claude Code reviewed, fixed, and ran it. The division of labor caught real bugs in both directions:

• Claude Science's smoke-verified plumbing (sub-10 MB slicing, byte-exact reassembly) worked unmodified. Its two flagged unknowns, the exact Arm B prompt and segmenter, were resolved on the box side: the real system prompt passed verbatim via flag, and split_sentences transplanted from run_armb_bios.py, verified AST-identical, so sent_idx semantics match the teacher-forced set exactly.
• Box-side review found what an off-box author couldn't: a transformers-5.x API break (torch_dtype→dtype, dict-returning apply_chat_template), an EOS-token leak that would have minted ~300 junk claims-with-activations, the MPS long-context leak (the hs_mixtral lesson), and a missing raw-generations dump, the jury's irreplaceable input.
• Result: 300 entities → 2,068 fresh self-gen claims (vs 2,067 teacher-forced, same segmenter, same prompt regime), 821 MB pack, collected concurrently with the crescendo queue on the 192 GB box, smoke-to-launch in under 40 minutes, handed off through pull_phase0.sh (pull → integrity-check → Drive upload → only then flip ready:true).

Reliability postscript (2026-07-08 → 10)

The box became shared (a co-user who power-cycles it) and the operator's network became intermittent, which forced the last robustness tier: heavy runs chained overnight behind .done_markers skip-guards (npz existence stopped being a valid guard once the mirror started deleting them); ssh heartbeat monitors that distinguish reboot / queue-death / console-change; every transport loop (mirror, status sync, phase0 handoff, grade chain) built as retry-until-window with resumable partials, a 5.5 GB pull crossed a dozen link drops without restarting; and a disk rule with teeth after data/ filled 228 GB to literal zero: score a pack → verify it on Drive by name+size → delete the local npz (82.6 GB reclaimed in one audited sweep, nothing lost).

Addendum (2026-07-14): the audit turnaround, and grading goes sovereign

Outside audit → answered experiments, same week

An external audit (2026-07-13) critiqued the project from this document alone. The harness's response time is itself a measurement of the harness:

• Same day ($0, local): calibration closed (probe OOF ECE 0.031 → 0.016 nested-isotonic), the nonlinear-probe question falsified (MLP/GBT lose to linear on every metric at the same representation), and the jury-prior channel quantified, with an 84-row blind human-audit sheet built for the human check.
• Same night ($0, remote): the long-tail critique met head-on, 500 PopQA questions, 50 per popularity decile, through 3 architectures, labels free via gold alias match. The night queue ran in 15 minutes.
• Next night ($0, remote): the "no semantic-entropy baseline" critique closed with a K-sample collector (10 resamples per prompt, 4 model×corpus arms, 1,800 prompts) reusing the existing greedy-answer labels, zero new grading. Headline: SE scores 0.82 overall on the short-QA set but 0.49, exact chance, on the correct-vs-confident-hall split where the internal probe holds 0.72+; 17.4% of that set's hallucinations are unanimous across 10 resamples, i.e. structurally invisible to any resampling method. The complementarity the paper argued from theory is now measured.

Every audit item that could be answered with existing labels or free computation was answered within ~36 hours of reading the audit. The items that cost money or human time were triaged to a human decision, not silently dropped.

Batch C: the jury moves onto the box

The label oracle has been the pipeline's only rented dependency, rate-limited, quota-capped, and (since the audit's jury-prior finding) known to leak juror world-knowledge into "supported" verdicts one way. The Batch C build moves grading onto the box's own GPU with two integrity mechanisms the API jury could not enforce:

• Quote-requirement grading: a "supported" verdict must carry a verbatim quote from the article, and the quote is string-verified in code , unverifiable quotes demote the verdict to not_found, recorded raw so the demotion rate is itself a metric. In the first smoke test the mechanism caught exactly the target failure: a juror marking a claim "supported" on a paraphrased summary that appears nowhere in the article.
• Knowledge-conflict canary: 25 article-perturbed items (a year shifted throughout a copy of the article, the claim keeping the real year) where a grounded juror must answer "contradicted" and a prior-driven juror gives itself away with "supported", plus 25 unperturbed controls to separate prior-leak from general weakness. The canary is packaged as an ordinary (bios, sentences, wiki-cache) dataset triple, so every grader, API or local , can be canaried unchanged.
• Bridge validation before trust (the standing rule): three cached cross-vendor local jurors (no Qwen, the defenders are Qwen) grade a 60-bio overlap slice and are compared against the stored API-jury verdicts before a single local label counts.

If the bridge holds, the marginal cost of grading drops to GPU-hours, the same economics that made the 25-model panel affordable, now applied to the labeling stage.