Measured, not claimed
The false-positive benchmark
A scanner that cries wolf is worse than none, so this page publishes how often xlogs does. Two corpora of pages and repositories a careful reviewer would call clean, each written by reading a detector and attacking a construct in it, are run through the shipped scanner. What came back is below, with every false positive named by case id.
The result, 2026-10-20, rules 1.14, engine 676f49a
Read it as: of 65 clean inputs whose expected outcome is settled, 65 produced no loud finding. The 0 open cases are product decisions not yet made and count for neither side. The controls are real leaks and real exposures planted beside the clean cases; if any stopped firing, a clean score here would mean nothing.
How far the counts reach: with 65 decided cases, 0 false positives is still consistent with a false-positive rate as high as 4.6%, and 32 of 32 controls firing is consistent with a scanner that reports as few as 91.0% of exposures like them. Both are exact one-sided 95% bounds (Clopper-Pearson), rounded away from the claim. Both treat the cases as a random draw of inputs like them, which a written corpus is not, so they state the uncertainty the counts carry and no more.
Round 1: secret patterns
- 10 cases, of which 0 are open judgement calls and 10 are decided.
- 10 of the 10 decided cases produced no loud finding; 0 still do, a count consistent with a false-positive rate as high as 25.9%.
- 1 of 1 true-positive controls fired, consistent with a scanner that reports as few as 5.0% of exposures like them.
Round 2: every other detector
- 55 cases, of which 0 are open judgement calls and 55 are decided.
- 55 of the 55 decided cases produced no loud finding; 0 still do, a count consistent with a false-positive rate as high as 5.4%.
- 31 of 31 true-positive controls fired, consistent with a scanner that reports as few as 90.7% of exposures like them.
What this number is, and is not
- It is a floor. Every case was written to make a detector fire on something clean. A random sample of real sites is a different instrument and would give a different figure.
- It is not a score. There is no accuracy percentage here on purpose; the two percentages above are bounds on what the counts can show. The counts, the denominator, each false positive by case id and the bounds are published so you can compute whatever you find meaningful and see exactly what it leaves out.
- It changes when the engine changes, and a deploy cannot skip that. Each run is stamped with the rules version, the commit and a hash of every file the corpora load, and the deploy gate fails while the rules version or those files differ from what is being deployed. Fixing a detector alters comparability, so earlier runs are kept in our repository rather than overwritten.
- The corpora are not public yet.
test/false-positive-corpus.test.mjsandtest/false-positive-corpus-2.test.mjsrun in the full test suite before every deploy, in xlogs's own repository, which is private. Round 2 joined the default suite on 2026-09-06, the day its last open call was decided, so a red result there fails the build rather than being allowlisted. Whether to publish them is an open decision, because round 2 shows what each repository check tolerates; until it is made, nobody outside xlogs can re-run these numbers.
What would make it outside evidence
The corpora, the harness and the decisions are all ours, so this is a self-test made public, not an independent measurement. None of these has been done yet, and each would change that:
- Publish the corpora and the script, so anyone can run the same cases against the published engine and get this file back.
- A held-out set: cases written by someone who has not read the detectors, run once, and reported before any detector is changed to pass them.
- Agreement with another scanner: the same inputs run through an independent tool, with each check both make compared one by one.
- A field sample: real sites drawn at random with every finding judged by hand, which would give a field rate where this gives a floor.