Detecting Confident Hallucinations from the Inside: A Negative Result on Output Confidence and a Bounded Positive Result on Residual-Stream Probes

Working paper draft, internal. Every headline number below is stated at the scope that survived adversarial review; claims the review weakened have been re-scoped or demoted, and every remaining single-seed / black-box-only / needs-white-box gap is flagged inline.

Catch-up pass 2026-07-13: folded in calibration (audit_batch_a.md), the nonlinear bake-off, the quantified jury-prior channel, the per-arch within-regime crescendo (crescendo.md), the 12-defender adaptive panel, multilingual×arch (m3_multilingual.md), and the PopQA long-tail arm (popqa.md).

Abstract

We study confident hallucinations, factual claims a model states with high token-level confidence (low entropy) that are nonetheless wrong, the failure mode that output-confidence signals are, by construction, worst-positioned to catch. We report two results.

Negative (lead result). No output-side confidence signal reliably isolates confident hallucinations from correct claims. In the short-answer closed-book regime the correct-vs-confident-hallucination split sits at or below chance for every output feature we tested; in the open-ended long-form regime aggregate confidence (mean token entropy ≈ 0.54) remains near chance, though the best single output feature partially recovers (min-max-token-probability ≈ 0.74, a weak correctness signal). The blanket "below chance everywhere" framing does not survive: it is a short-answer artifact, and we scope it accordingly.

Positive (bounded). A supervised linear probe on the mid-network residual stream separates individual true-vs-false factual claims in open-ended generation at 0.89 (collapsed, all claims) to 0.94 (clean supported-vs-contradicted subset) AUROC, and, decisively, holds 0.91–0.92 on the most-confident / calmest decile (selector-invariant across three confidence features) where every independently-measured output-confidence feature degrades to 0.61–0.73. The signal survives length, fame, sentence-position, and label-set (independent re-grade) controls. It transfers cold across topic-domain (person bios → places/events/species, ~94% retention) but is a parametric-factuality direction, not a universal truth direction: it degrades to ~0.6 on faithfulness / logical-entailment / context-free-claim regimes. Under a low-information black-box adaptive attacker (a multi-turn "crescendo" attacker fed the scalar truth projection each turn) the probe shows no measurable degradation in relative detectability, but this is a single-direction, black-box-only result at a saturated cross-regime ceiling, and the white-box / gradient / RL-obfuscation attacker classes that defeat comparable safety probes were not tested.

We claim detection of a specific, detectable subset, bounded above by hallucination-inevitability theory, not a solution to hallucination.

1. The negative result: output confidence cannot isolate confident hallucinations

1.1 The defensible claim

For open-ended factual generation, aggregate token-confidence, mean token entropy, sequence perplexity, does not separate confident hallucinations (wrong claims stated with low token entropy) from correct claims: mean entropy sits at ≈0.54, near chance. In the short-answer closed-book regime (HaluEval, ~3-token answers), every output feature we tested, max token probability, mean entropy, perplexity, and 62-column full-vocab logit features, sits at or below chance on the correct-vs-confident-hallucination split, even while the same features retain overall AUROC ≈0.70 by catching hedged, high-entropy hallucinations.

This is the honest, regime-split form. It replaces the earlier project headline "output baselines 0.42–0.47 (below chance)," which review showed to be a short-answer artifact imported under an open-ended label.

1.2 What review weakened, and the corrected scope

The "≈0.42–0.47, below chance" number traces to the short-answer M1/M2 work (HaluEval closed-book, ~3-token answers; m2_fullvocab.md). On the project's flagship open-ended dataset (Arm B free-form bios), the same hard split does not collapse to sub-chance:

• The output 4-feature probe scores ≈0.71 on the confident subset (project's own m3_armb.md reports 0.62 for it; either way, above chance).
• Raw min-max-token-probability works as a positive correctness signal at 0.739 on the clean split, the opposite sign from "inverted."

So output confidence is not uniformly blind: it partially recovers to 0.62–0.71 in the open-ended regime it is asked about, catching the easy hedged hallucinations. The correct statement is therefore two-regime:

"Below chance / inversion" is dropped as a general claim. True inversion (confident ⇒ more likely wrong) is regime- and feature-dependent: it appears cleanly only in the fabricated-entity crescendo regime, not in open-ended factual generation, where min-maxprob points the correct way. Any use of "inversion" in the paper must be pinned to the fabricated-entity setting and defended separately, it is a stronger and more surprising claim than the nearest prior work (CHOKE) makes, and our own open-ended data contradict the blanket version.

1.3 Why this is the right framing (theory + prior work)

The negative result is exactly what a mature literature predicts. Kalai & Vempala (STOC 2024) show a well-calibrated model must hallucinate at a rate tied to rarely-seen facts, confident wrong outputs are a consequence of good calibration, so output confidence cannot in principle separate them. CHOKE / "Trust Me, I'm Wrong" (Simhi et al., EMNLP 2025 Findings) is the closest analog: it defines our exact subset (the model has the knowledge yet confidently emits the wrong answer) and reports token-probability methods catch only ~16–43% of these, while residual-activation logistic probes do materially better, independent prior confirmation of both halves of our story. Azaria & Mitchell (2023) supply the mechanism for why output confidence conflates fluency/frequency with truth (sentence log-prob is confounded by length and word frequency; pairs with our length-alone AUROC 0.58 control).

The scope guardrails, each tied to a cite: - Not a claim about all token-level data. Kadavath et al. (2022) show that on well-formed in-distribution multiple-choice, large models are calibrated and P(True) works. Our claim is scoped to open-ended / short-answer generation, and explicitly excludes the MC regime. - Semantic entropy (Farquhar et al., Nature 2024; Kuhn et al. 2023; Semantic-Entropy Probes, Kossen et al. 2024) is a different class, targeting confabulation (unstable meaning across resamples). A confident hallucination is the low-variability case the model re-states under resampling, so SE is blind to it by construction, this is complementary, not competing. State "different class," not "worse." SEP is the sharpest reviewer trap: it reads an activation but estimates sampling variability, so it inherits the blind spot; distinguish our probe as a supervised truth-direction. The blind spot is now MEASURED, not argued (2026-07-14, se_baseline.md): K=10 Farquhar-config resampling across 4 model×corpus arms scores SE at 0.81–0.90 overall (0.904 on PopQA long-tail, SE's home regime, and a genuinely strong baseline there) but at exact chance (0.486–0.508, every metric variant) on the correct-vs-confident-hall split where the probe holds 0.72–0.76. 17.4% of the short-QA hallucinations are unanimous across 10 resamples (3–7% on PopQA), zero SE signal at any threshold, definitionally indistinguishable from unanimous-correct answers. Quote the pairing in both directions: SE tops the probe's cold transfer on the deep tail; the probe covers SE's structural blind subset at 1/10th the inference cost. - Logit/perplexity detectors (EPR, LLM-Check, RePPL) report non-trivial overall AUROC, they work on the easy hedged hallucinations and fail on the confident subset. Frame as "blind on the hard split," never "useless," or these papers falsify us.

2. The positive result: an internal probe catches a detectable subset

2.1 Setup

Qwen2.5-7B-Instruct, fp16, teacher-forced over 2067 sentence-claims from 300 Wikipedia-grounded FActScore-style biographies (Arm B). Residual stream captured at the mean and last answer-token positions across 29 layers. Probe: L16–18, mean+last concatenation (21504-dim) → StandardScaler → PCA-64 → L2 logistic regression, 5-fold StratifiedGroupKFold by bio_id (no entity leaks across folds). Labels are 3-juror LLM grades vs the source Wikipedia article; label 0 = supported (TRUE), 1 = contradicted (FALSE). Base hallucination rate 0.85.

2.2 Headline, honestly bracketed

Quote 0.89 (collapsed, all claims) as the floor and 0.94 (clean subset) as the ceiling. The +0.05 clean bump is partly a favorable-subset effect: the clean split excludes 662 not_found claims (~32% of graded data), a large, non-random exclusion (unverifiable claims). Do not quote 0.94 unqualified.

Both numbers reproduce exactly on independent re-derivation. A permutation null returns 0.51; there is no group leakage and no max-flip orientation inflation.

Layer structure (re-scope). The peak is not a sharp "L16–18" spike but a broad L13–L19 plateau (0.925–0.933 single-layer mean, Qwen2.5-7B); L13 already reads 0.925. Concat(mean+last) at L18 buys a small, real +0.01 over single-layer (0.944 vs 0.933). "L16–18" is a defensible but post-hoc pick inside a wide plateau, the invariant to cite is fractional depth (~0.58–0.62 of network depth), not an absolute layer index.

Probe form is load-bearing. The learned logreg carries the headline (0.933 single-layer). The interpretable 1-D diff-means direction, the same w_diffmeans the §5 attacker was fed, is only 0.903, ~0.03 weaker. The strong result belongs to the learned probe, not the interpretable truth direction; keep the two distinct.

Linear is not a limitation (measured 2026-07-13, audit_batch_a.md). On the same PCA-64 representation and the same grouped folds, a small MLP (64×32) scores 0.826 and gradient-boosted trees 0.850 vs the linear probe's 0.892 (collapsed), the nonlinear heads lose on every metric including the fame-controlled within-bio number (0.700/0.718 vs 0.788). There is no evidence of exploitable nonlinear structure at this representation, consistent with the steering result that a rank-1 linear direction is load-bearing. (Scope: one model/corpus/representation; a raw-dimension capacity sweep was not run.)

Probe scores are calibratable (measured 2026-07-13, closes the old "no operating-point metrics" gap). Out-of-fold probabilities: ECE 0.031 raw → 0.016 after a nested grouped-isotonic layer (Brier 0.085); the output 4-feature baseline reaches similar ECE only by predicting near the 0.85 base rate (Brier 0.118), calibrated but unresolving. Caveat from the steering arm: calibration is precisely the surface a steering adversary shifts wholesale (mean p_hall 0.03↔1.00 at intact ranking), so a deployed threshold requires monitored recalibration; the ranking, not the calibrated score, is the robust object.

2.3 The money result: the calm/confident decile

This is the subset the whole project is about, claims stated calmly and confidently. On exactly these rows, output confidence should be blind and the probe should not be.

Circularity closed (2026-07-10, recall_vs_truth.py §1b). The old version selected the decile on an output-confidence feature and then evaluated that same feature inside it, restriction of range mechanically drives the selected feature toward 0.5. The fix is a cross-selector design: select the top-10% calm/confident decile on each of three confidence features in turn, and quote only the probe and the two non-selected features on each decile (all numbers CLEAN split):

The honest, still-decisive statement: on the calm/confident decile, however it is selected, the internal probe holds 0.91–0.92 while every independently-measured output-confidence feature degrades to 0.61–0.73. "Output ≈ chance" was a selection artifact and is dropped; the ~0.2–0.3 AUROC gap is real, selector-invariant, and it is the point, the probe is not a repackaged logit signal.

2.4 Confounds ruled out

(All numbers in this section are CLEAN split unless marked; the collapsed counterpart is given wherever it differs materially.)

• Length. n_tokens alone 0.61 (clean) / 0.58 (collapsed). Probe survives length stratification: 0.93 (short) / 0.95 (long) within strata (clean), where n_tokens itself falls to ~0.52–0.57. Residual note: within a bio, false claims are somewhat longer (within-bio n_tokens 0.767), length is not perfectly orthogonal to the label, but the probe (0.929 within-bio, clean) dominates it.
• Fame / cross-bio popularity. Within-bio discrimination (entity held constant, 98 mixed bios): 0.9293 clean / 0.79 collapsed, the clean figure barely below cross-bio 0.9416 (clean). The collapsed within-bio number is the conservative floor of the fame control; quote the pair, not the clean number alone. The signal is intra-bio claim ranking, not entity-popularity separation. Trivial baselines inside a bio: output 4-feat 0.748, entropy 0.480 (chance).
• Sentence position. sent_idx alone 0.582; probe within early/late strata 0.932 / 0.948. Ruled out.
• Label set. An independent v2 jury re-grade reproduces 0.9292, the result is not an artifact of one juror panel.

2.5 Standing caveats on the positive result

1. The clean-vs-collapsed gap is a 32% non-random exclusion, not just noise reduction. Report both numbers.
2. Class imbalance 3.6:1 (85% false). AUROC is imbalance-robust, and calibration is now measured (§2.2: ECE 0.016 recalibrated; AUPRC 0.978); recall@FPR figures exist in m3_rigor.py. What remains open is precision at a chosen deployment threshold under distribution shift, in-domain operating points are now quotable.
3. Labels are LLM-jury-graded vs Wikipedia, not human-adjudicated, and the specific risk is now named and bounded (2026-07-13): a correlated-prior channel, a plausible false claim graded supported because jurors share the defender's misconception, corrupts labels in one direction only (deleting hard positives). Visible surface on Arm B: 1 of 301 supported rows carries a contradicted dissent (14 more a not_found dissent); all 96 v1→v2 label flips ran supported→false. Headline AUROCs move <0.01 under worst-case assumptions, but confident-tail absolutes should be read as upper bounds: on "fooled-a-juror" falses (≥1 juror voted supported, n=26) the probe drops to 0.60, the honest hard tail, itself partly label noise. A targeted 84-row blind human audit (armb_prior_audit.xlsx: all dissent rows + top-20 probe-flagged supported rows + 50 controls) is built and awaiting adjudication; structural fixes (verbatim-quote-required grading, knowledge-conflict juror canary) are queued for the local-jury build.
4. Architecture generality is not established from this analysis alone. The re-analysis here covers only Qwen2.5-7B and 14B, same family. That demonstrates scale-generality (7B plateau peak 0.933 at depth 0.62; 14B peak 0.953 at depth 0.58, same fractional locus), not architecture-generality. Broader cross-vendor claims (phi-4, Gemma, Llama, Mistral, Falcon-Mamba, etc.) live in the separate xarch panel and must be cited from there, not inferred from this two-model sweep. Flagged: the cross-architecture claim needs the panel to carry it; do not over-read "architecture-general" from Qwen-only data.

3. Transfer: a factuality direction, not a universal truth direction

3.1 What holds (cross-domain, solid)

An L16–18 Arm B probe applied cold (no alignment, same Qwen2.5-7B) to Arm C (places / structures / events / species) scores:

• 0.73–0.76 binary / 0.80–0.82 clean, within-Arm-C ceiling ~0.80–0.81 → ~94% retention of the in-domain ceiling. Reproduces essentially exactly and is robust to the training-label convention.
• Falsification control: a 5-seed label-shuffle collapses the cold transfer to 0.431 ± 0.027 (≈0.57 oriented). sent_idx-alone on Arm C is 0.579. So the transfer is driven by label-ordered signal, not claim position or a geometry that survives label destruction.
• Long tail (PopQA, 2026-07-13, popqa.md), the "famous entities only" objection tested head-on. 500 closed-book questions stratified 50-per-popularity-decile (subject page views 2 → 15M), gold-alias labels (no jury). Cold Arm B → PopQA: 0.76–0.83 overall across Qwen-7B / phi-4 / Llama-8B, in line with TriviaQA cold (0.836) and far above the FEVER/SciFact cold band. Within-corpus, the deepest deciles are as separable as the head (Qwen-7B: deciles 0–2 = 0.901 vs 7–9 = 0.879; knowledge-relations∩tail 0.902), the truth signal does not require entity fame. Two qualifiers: deep-tail accuracy is guessability-inflated (country-from-toponym, sport base rates, all numbers reported with a guessable/knowledge relation split), and the hard slice is model-dependent (knowledge∩tail 0.90 / 0.73 / 0.75 within-corpus). Behavioral corroboration of §1: two of three models refused zero of 500 long-tail questions at 72–80% hallucination rates; only phi-4 ever abstained (14×, all low-popularity).
• Language (multilingual×arch, 2026-07-11, m3_multilingual.md). The EN-trained direction transfers cold to zh/es/sw at 95.6–99.1% best-layer retention in every cell of a 4-architecture × 3-language grid (Qwen-7B, phi-4, Llama-8B, Falcon-Mamba, including the non-transformer). Use best-layer, not a fixed band, for deep stacks.

The domain shift is real but modest: Arm C's own ceiling (~0.80) is genuinely lower than Arm B's (~0.89), so "domain-general" costs ~0.08–0.10 and the not_found-heavy Arm C labels add noise. The 0.80–0.82 "clean" figure the write-up leads with is computed on 795/1604 claims (not_found dropped, the easier subset); the full-set binary transfer is 0.73–0.76. Report both.

3.2 What review weakened

• The cross-model leg is not established by the transfer analysis itself. The cited "cross-model ~0.79–0.91" figures are within-model CV (each model's own probe on its own claims), which is not cross-model cold transfer. Genuine cross-model transfer needs CCA/Procrustes alignment; the "99.2% aligned-grid retention" is a ratio to own-space, not a transferred absolute AUROC. Flagged: any cross-model transfer sentence must be sourced from the aligned-grid / CCA work and labeled as alignment-dependent, or dropped.
• The shuffle control proves less than "~90% real truth signal." Label-shuffle destroys all label-correlated structure, so it can only show the direction is label-driven, it cannot separate a genuine truth direction from a non-truth feature that correlates with the label and is shared across Arm B and Arm C (length, fame/frequency, specificity, named-entity density). That shared-correlate scenario is exactly the literature's warning. Only one concrete confound (sent_idx) was tested here; the project's real fame/length controls live in the bias audit and must be invoked to make the "truth, not correlate" case, the shuffle alone does not license it.
• True cross-dataset cold transfer to context-free gold corpora is weak: ~0.62–0.66 (FEVER / SciFact / HaluEval-dialogue), well below the ~0.83–0.91 in-domain band, with output baselines ~chance.

3.3 The honest one-line scope of transfer

A cold L16–18 probe carries a parametric-factuality direction that transfers across topic-domain (~94% retention), but it is not a universal truth direction: it degrades to ~0.6 on faithfulness/grounding truth, logical entailment (FOLIO), and bare teacher-forced claims the model never chose to assert (FEVER/SciFact cold). Cross-model transfer is alignment-dependent and must be cited from the aligned-grid work, not asserted here.

This lands us on the pessimist side of the generalization debate and is consistent with Track B's own finding (factuality subspace, cold HaluEval/FOLIO 0.57–0.67). Position as "we independently reproduce the brittleness and localize it to a factuality subspace," not as a refutation of it. Track B's summarization within-CV 1.000 is disclosed as a length/style artifact (n_tokens alone 0.833; marked "do not cite").

4. §5, Adaptive robustness under a black-box attacker

4.1 The surviving claim

We ran a probe-aware black-box adaptive attacker: a multi-turn "crescendo" attacker (phi-4) escalating specificity against a Qwen2.5-7B defender over 40 fictional-entity × 10-turn threads, fed the scalar L16 diff-means truth projection each turn and instructed to push its fabrications to read TRUE. Three arms: blind (no feedback slot), probe_aware (real projection), shuffle (same scaffold, scrambled projection). 4/4/3 seed-runs.

The probe-aware attacker does not reduce the probe's relative detectability of its fabrications. Seed-averaged, probe_aware ≈ blind ≈ shuffle across diff-means and logreg L16/17/18; every cross-arm delta is |Δ| ≤ 0.009, inside the seed SD (0.006–0.022).

(Full 4/4/4 grid, re-scored 2026-07-10 after re-pulling the intact shuf_s3 from Drive, see §4.2 #5. Deltas: logreg probe_aware−blind +0.001 (min −0.017 / max +0.020 over seeds), probe_aware−shuffle −0.003; diff-means +0.006 / −0.010. All inside the seed SD, sign unstable.)

Mechanistically confirmed, not merely "AUROC unchanged." The real-feedback attacker's per-turn projection trajectory (T1 −10.8 → T10 −1.0) is identical to the shuffled-feedback control (−10.8 → −1.4): the climb toward TRUE is a crescendo-structure artifact (later turns are more specific), not evasion. Fabrication projections stay strongly negative in every arm (−2.3 to −5.4) against the +8.2 TRUE reference. The attacker never pushed a fabrication to read TRUE.

Panel extension (2026-07-12, crescendo.md §5). The live black-box attacker arm now covers 12 defenders (5 vendors, 0.5B–27B, including a state-space model and a no-think reasoning model): each defender's own-dimension probe scores its own attacker-elicited fabrications at 0.890–0.996 with every output baseline at or below chance, and a same-family control (Qwen attacker → Qwen defender, 0.964) matches the cross-family arms, the attacker elicits more fabrication but opens no hole in any defender's truth direction. Same caveats as above: cross-regime ceiling, low-information attacker class.

A published number is corrected in the robust direction. The §5 headline "probe_aware 0.963 / shuffle 0.967 / blind 0.980" reproduces exactly, but these are single-seed L16 logreg values, and blind's 0.980 is the top of its 4-seed spread (mean 0.970). The documented "probe_aware < blind by 0.017" is lucky-blind-seed noise: seed-averaging collapses it to +0.001, aligned scoring to +0.006. Neither "robust (0.963 vs 0.980)" nor "the attacker degrades the probe" survives as a point estimate, only "no measurable degradation" survives, and that is the claim to make.

4.2 The honest caveats (all load-bearing)

1. Single direction, single feedback channel. The attacker saw one scalar (L16 diff-means projection), the lowest-information probe surface. This is a weak attacker class.
2. Black-box only; no white-box arm. FLAGGED, needs the white-box arm. The adjacent safety-probe literature shows this monitor class falls to (a) prompted-LLM red-teaming with per-token feedback (2511.00554: 0.91 probes → 60–88% failure), (b) gradient/embedding-suffix obfuscation (2412.09565, near-total bypass), (c) RL-finetuned latent obfuscation (2506.14261). We tested none of these. The defensible claim is "robust to a low-information black-box text attacker that never sees per-token probe scores or gradients," not "robust to adaptive attackers." Their mitigation that survives attack is max-pool/attention over per-token scores, worth checking whether our mean+last pooling is the aggregation they warn is vulnerable.
3. Absolute AUROCs are regime-inflated; treat them as relative only. Fabrications are scored against Arm B supported bios (cross-regime negatives). The same diff-means direction's within-Arm-B true-vs-false AUROC is only 0.772, so ~0.16 of the ~0.93/0.97 is bio-vs-crescendo domain separation, not falsity. Corroboration: Arm B false claims project at +1.87 (positive!) while crescendo fabrications sit at −3.75; the ~5-point drag is regime, not falsity. The design structurally cannot be scored within-regime (fictional entities have no true facts for within-regime negatives). Report §5 as a relative/cross-arm robustness result at a saturated ceiling, not as an absolute detection number. The prior framing "0.965 > in-distribution 0.89" markets the confound as a virtue and is backwards. The within-regime absolute is now measured separately on the paired real-entity crescendo with full Wikipedia-jury labels (completed 2026-07-14, crescendo.md): true-vs-fabricated inside the multi-turn regime scores Qwen-7B 0.84–0.87 / phi-4 0.77–0.80 / Gemma-4-12B 0.69–0.74 / Qwen2.5-14B 0.71–0.75 (own-dimension probes; output baselines always below the probe's best layer but not always at chance, Qwen-14B's min-maxprob reaches 0.715). Architecture-general in sign, but the 0.69–0.87 spread is model-dependent, NOT vendor- or size-monotonic: Qwen-14B lands near the bottom, well below its own 7B sibling, do not tell a per-vendor story. These, not the 0.93–0.97 cross-regime figures, are the absolute multi-turn detection numbers to quote.
4. The matched control is shuffle, not blind. probe_aware − blind mixes a probe-score change with an attacker-instruction change (blind is not told to push false-reads-true). The clean same-instruction control is probe_aware − shuffle, which is negative on all four metrics (−0.009, −0.003, −0.005, −0.002). The consistent negative sign is the single thread hinting at partial evasion, but it is far below noise. State it; do not over-spin it as "cleaner than documented."
5. Thin, partly-corrupt control arm. CLOSED 2026-07-10. Shuffle is now a full 4-seed arm. The s3 npz that was truncated (17 MB) was only the local copy, the Drive backup was intact (265 MB); re-pulled, it loads clean (400, 29, 4, 3584) and the seed grid re-scores 4/4/4 with the conclusion unchanged (probe_aware ≈ blind ≈ shuffle; see §4.1). The shuffle control is no longer thin.
6. Non-paired positive sets. nfab varies 67–115 across arms (substantive/refusal filter), so arms are compared distributionally, not paired.

Net §5 claim: No measurable degradation of relative detectability under a low-information black-box adaptive attacker, mechanistically confirmed as failed evasion, single-seed-direction, black-box-only, at a saturated cross-regime ceiling, with the white-box/gradient/feedback attacker classes untested.

5. What this cannot do, limitations and the guaranteed blind spot

• It is a detectable subset, provably not all of it. Hallucination-inevitability results (Xu et al. 2024, diagonalization; Kalai–Vempala STOC 2024, calibration; OpenAI 2025) guarantee a residual set of confident wrong outputs no detector removes. Karbasi et al. (2025) frame detection as Gold–Angluin identification, impossible from positive examples alone, but feasible with labeled true and false examples: precisely our supervised external-probe regime. We are in the tractable corner, and it is still only a corner.
• Not a universal truth direction (§3). Faithfulness, logical entailment, and context-free claims fall to ~0.6.
• Cross-model transfer is alignment-dependent and not demonstrated cold in the transfer analysis.
• §5 robustness is not general robustness (§4.2). No white-box, gradient, RL-obfuscation, or per-token-feedback attacker was tested; the tested attacker is the weakest class in the adjacent literature.
• Absolute numbers are regime-sensitive. The fabricated-entity crescendo AUROCs are cross-regime-inflated (within-regime absolutes: 0.69–0.87, model-dependent and not vendor/size-monotonic, §4.2 #3); the confident-decile "output ≈ chance" was a selection artifact, closed 2026-07-10 with a cross-selector re-derivation (probe 0.91–0.92, independent output features 0.61–0.73, selector-invariant). Calibration and AUPRC are now measured in-domain (§2.2); what remains unmeasured is a deployment threshold's precision under distribution shift.
• Detection strength is model-dependent even where the signal is universal. Every architecture tested carries the direction, but the hard slices spread: within-regime multi-turn 0.86/0.80/0.74 (Qwen/phi-4/Gemma), PopQA knowledge∩tail 0.90/0.73/0.75. Headline numbers are Qwen-7B, the most-studied model and the top of the range; do not quote them as panel-typical.
• Labels are LLM-jury vs Wikipedia, not human-adjudicated. The risk is now named and bounded (§2.5 #3: correlated-prior channel, one-directional, visible surface ~1–15 rows of 301, confident-tail absolutes = upper bounds); the targeted 84-row blind audit is built and awaiting human adjudication.
• The interpretable diff-means truth direction is ~0.03 weaker than the learned probe, mechanistic and detection claims should not be conflated.

6. Related work and scoop positioning

Decomposing our positive contribution into three cells:

The defensible wedge: a black-box multi-turn (crescendo) attacker escalating while a factual-hallucination probe is monitored, asking whether the probe survives. We could not find this cell occupied for factuality. But the technique is thoroughly established one domain over, so frame this as "port a known safety-probe red-team method to the factuality-probe setting," not a wholly new idea, and state upfront which attacker classes were and were not tested, because the safety literature has already shown stronger attackers win. Citation check (2026-07-10): all 9 arXiv IDs cited in this section (and all 7 in adaptive_attacks_bibliography.csv) resolve on arxiv.org with titles matching the descriptions used here, including the previously-unconfirmed LAD (2604.28129). The scoop verdict still rests on search coverage, an unfound occupant remains possible, but no cited ID is fabricated.

Must cite and distinguish: (1) CHOKE / 2502.12964, convergent prior confirmation of the negative-plus-internal-recovery story; differentiate on our cross-domain transfer and the adaptive-attacker test. (2) Farquhar, Nature 2024, different failure class (confabulation); distinguish, do not compare. (3) Kadavath 2022, the calibration counter that forces our regime scoping. (4) Orgad 2410.02707, corroborates the internal-vs-verbalized discrepancy (encode-correct, generate-wrong), so our confidence-inversion finding is replication of an established phenomenon in a new confident-subset regime, not a standalone scoop.

7. Open threads

1. White-box / stronger-attacker arm for §5, gradient-suffix, RL-obfuscation, and per-token-feedback prompted attackers (the classes that defeat safety probes). The single biggest gap between our claim and "robust to adaptive attackers." Needs new runs.
2. Within-regime scoring for the crescendo, CLOSED 2026-07-14: all four architectures jury-labeled and scored (Qwen-7B 0.84–0.87 / phi-4 0.77–0.80 / Gemma 0.69–0.74 / Qwen-14B 0.71–0.75; §4.2 #3). Still unmeasurable: Gemma's output baseline (its collector emitted non-finite output features on all turns, fp16 overflow).
3. Genuine cross-model cold transfer, surface an aligned absolute transfer AUROC (CCA/Procrustes), not a within-model CV or a retention ratio.
4. Truth-vs-shared-correlate, bring the bias-audit fame/length controls to bear on the transfer claim; the shuffle control alone cannot license "truth, not correlate."
5. Human adjudication, two sheets now await grading: the 100-row random armb_human_audit.xlsx (general error bound) and the targeted 84-row armb_prior_audit.xlsx (2026-07-13; measures the correlated-prior channel specifically).
6. Operating-point precision, calibration (ECE/Brier, §2.2) and AUPRC now measured in-domain; residual gap is threshold precision under distribution shift, plus the steering caveat that calibration is the adversarially fragile surface.
7. Confirm the scoop cell, closed 2026-07-10: all 16 cited arXiv IDs resolve with matching titles (see §6 citation check). Residual risk is search coverage, not citation validity.
8. Single-seed flags to close: §5 single feedback direction (per-token-feedback attacker arm smoke-tested, full run pending). Cross-arch adaptive, closed 2026-07-12: 12-defender panel, §4.1. Qwen-only "architecture-general" wording, now sourced from the panel with the honest strength-spread (§5 model-dependence bullet). Confident-decile selection circularity, closed 2026-07-10 (recall_vs_truth.py §1b). Thin shuffle arm, closed 2026-07-10 (full 4-seed grid, §4.2 #5).
9. Nonlinear capacity sweep (2026-07-13): same-representation MLP/GBT lose to linear (§2.2); the remote-scale version (raw 3584-dim, deeper heads, multi-layer concat) is future work, not required for the claim.
10. Semantic-entropy baseline, closed 2026-07-14 (se_baseline.md, folded into §1.3): SE 0.81–0.90 overall / exact chance (0.49) on the confident-hall split; 17.4% of short-QA halls unanimous under 10 resamples = structurally invisible to any resampling method. Residual (cheap, non-blocking): probe-score on the unanimous-wrong subset specifically (needs an npz re-pull; the Arm B confident-false 0.90 already covers the claim-level version of the question).