How This Was Built, a human/AI division of labor, stated plainly
Written 2026-07-15 as the repo's statement of who did what. The first person in this document is me; the agent drafted it from my decisions and I signed off on it.
Based on the document as it was left in the project folder in July 2026. Formatting was converted for the site and long dashes were replaced. I revise it as I keep analyzing the work; the cost figures were corrected 2026-10-14. File names and figures refer to the project repo, which is not published.
This project was built by one person directing an AI coding agent (Claude, in the Claude Code CLI) over roughly a month of part-time work, using only my Claude Max subscription, $10 of API credits plus free-tier quotas. The agent wrote most of the code and ran most of the experiments. I am publishing that fact rather than hiding it, because the interesting question in 2026 is not whether AI did the labor, it is what the human had to do so that the labor produced science instead of plausible-looking noise.
The repo itself is the receipt for everything below: ~19,500 lines of code across two repos, ~93 commits in 26 days, ~200 GiB of activations across 25+ models, ~75,000 jury verdicts, and 50+ written analyses. The agent produced that volume. What I produced is harder to count and decided whether any of it was worth believing.
What the human did
1. Chose the problem, and kept it falsifiable.
The research question, confident hallucination, the wrong answer a model
states calmly, which output-confidence signals are structurally blind to ,
was mine, and it is narrower and sharper than "detect hallucinations."
Before the main experiments ran, I fixed the success criteria (beat AUROC
0.85 overall AND score ≥0.70 on the correct-vs-confident-hallucination split
AND survive cross-dataset transfer) and held the project to them. Two
experiments were pre-registered with pass/fail bars committed to git before
data collection (PREREG_*.md); one recorded an honest near-miss.
2. Set the evidence standards and did the human labor they require. Every integrity property this project can claim traces to a rule I imposed:
- Human labels are the source of truth. I hand-graded the original 300-row
label set (
qa_grading.xlsx) and personally audit label samples; the LLM jury audits human labels and is never allowed to silently overwrite them. - No model grades itself or its own family. Long-form claims are graded only against retrieved Wikipedia text, never a judge's memory, because our own audit caught a judge hallucinating a fact in order to grade one.
- No judge is trusted on arrival: every jury-seat change was bridge-validated against the outgoing seat's stored verdicts on an overlap slice before its votes counted (96.6%, 95.3% on the two swaps).
- No feature and no label exists without a real measurement or a real human/jury judgment behind it.
3. Demanded verification, including against the project's own results. I ordered an adversarial re-derivation pass over the headline claims (2026-07-09), which narrowed several of them, the corrected, smaller numbers are the ones now quoted. I solicited an outside audit of the project (2026-07-13) and had every free-to-answer critique answered with experiments within ~36 hours. When a published panel finding turned out to be a tokenizer bug, it was retracted in place, the wrong number is still visible in the write-up, struck through next to the true one, and a guard was added so the bug class cannot recur silently. A falsification gate caught our own inflated 0.98 result and replaced it with the honest 0.85–0.87; the correction is the headline, not a footnote.
4. Made every fork-in-the-road decision. A selected inventory:
| When | Decision |
|---|---|
| 06-17 | Problem framing + go/no-go bars set at a research review; pivot to residual-stream probes after establishing llama.cpp cannot expose hidden layers |
| 06-18 | Detector gets its own repo/lifecycle; API keys scrubbed and rotated |
| M1–M2 | Accept and publish the M2 negative result (full-vocab logits are blind to the target subset) as load-bearing evidence, not a failure to bury |
| Arm A | Accept rejection of the claim-localization hypothesis and say so |
| Arm B | Wikipedia-grounded claim-level grading design, the project's strongest result |
| 07-0x | "Don't believe it until it transfers": transfer arm required before trusting the probe; pooled-panel expansion to 25 architectures |
| 07-06→12 | Jury seat swaps under quota pressure, each gated on bridge validation; API usage capped at $10 |
| 07-09 | Adversarial re-derivation ordered; several claims re-scoped narrower |
| 07-13 | Outside audit solicited, triaged, and answered; 1-week finish path set; decision to publish the repo |
| 07-14 | Semantic-entropy baseline run (the predictable reviewer ask); quote-requirement + knowledge-conflict canary designed into the local jury; measured result: local jury integrity validated but below the agreement bar, so its labels were rejected, by the same standard applied to every other judge |
5. Ran the program. Hardware access (a borrowed, shared Mac Studio that could be power-cycled without warning, hence the night queues, done-markers, and resumable everything), API keys and budgets, sequencing, when to spend money versus days, and when to stop. Total cloud spend beyond the Max subscription: ~$10.
What the agent did
Wrote the collectors, probes, graders, queues, monitors, and transport
daemons; ran and babysat the overnight collection runs; diagnosed its own
infrastructure failures (a memory-pressure kill traced to eager-attention
buffers, a fake "quota storm" traced to a dead paid seat); drafted the
analyses and kept the documentation synchronized with results. It operated
overnight autonomously against checkpoints and monitors, and its work is
legible in the git history and the session log
(previous_session_summaries.md).
Correction policy (and why the history is trustworthy)
Two kinds of content live in this repo, with different rules:
- Claims (analyses, README, the paper): corrected whenever found wrong,
visibly, dated correction notes, struck-through numbers, retraction
blocks. See the R1 retraction in
analysis/xarch.mdfor the house style. - Records (session logs, decision history, and the
PREREG_*.mdpre-registrations): append-only. What the project believed or predicted at a point in time is never rewritten; corrections are added next to the original, dated. The pre-registration files are frozen by definition.
If you find an error, that policy is what you should expect to see applied to it.
Why say all this
Because the alternative, presenting agent-built work as hand-built, or hiding the division of labor, would fail the same integrity standard the project applies to its own data. And because "can direct an AI agent through a months-long research program and get auditable, reproducible, honestly-scoped results out the other end" is now a skill worth demonstrating in public. This document is the demonstration.