Plain record › Study 31

Can simple automatic checks catch three planted data defects without flagging clean copies?

Study 31 · Changed a rule or plan · recorded 1 October 2026 · answer: Partly · full paper with figures

The question

An earlier test planted, in copies of one study, duplicated rows, a single extreme row and a subset chosen after looking, and existing checks missed them. We built the smallest check for each, two for the extreme row, and scored each on 200 copies with its defect and 600 clean or real-effect copies, where any flag is a false alarm.

Rules fixed before the result

Before any check was computed, we fixed each check and its threshold, including eight standard deviations for the largest-deviation check and half the registered sample for the coverage check. A check catches cleanly when the lower end of the 95% interval of its catch rate is at least 90% and the upper end of its false-alarm interval at most 2%. A defect is covered when one of its checks catches cleanly.

Data and method

The copies are the earlier test's, built from 548 past events at a five-day horizon: 200 for each of three clean or real-effect cases and four defect cases. The fourth defect, rows dated past our development data into the period set aside for testing, has no check of its own and is scored only on the other checks' flags. Each check ran blind to the case; its measures are the shares of defect copies and of the 600 others flagged.

The answer

The duplicate-row check and the coverage check, for a subset chosen after looking, each flagged every copy with its defect and none of the 600 clean or real-effect copies, meeting the bar of catching at least 90% with at most 2% false alarms. Neither check for a single extreme row met it: removing one row at a time caught fewer than one copy in ten, and the largest-deviation check caught 198 of 200 but raised false alarms on 88 of 600, about 15% against the 2% limit. All as expected.

What it changes

It changed the planning rules. The duplicate-row and coverage checks are proposed for the project's shared tools, each to be built in a later, separately reviewed step. Because no check covers a single extreme row, every later study must name that risk among its limits and report its result with each row left out in turn beside its main result.

What remains uncertain

Each check was scored against the defects exactly as the earlier test planted them, so two checks were nearly certain to catch them; their false-alarm rates are the informative figures. The coverage check catches a subset chosen without being written into the plan; a subset the plan names as its population escapes it. The checks' author knew which case was which, so the blinding binds the code, not that person, and one horizon and one kind of event were used.

How it was made

Run took 27 minutes; independent code review: approve; the recorded calculation reruns to the same result.

Fingerprints

plan 8f6ef45c17b96de6a196bb4bae9aee17d6315c27bdb68c13e47e5dc799f5c013
result cbc081e491fe8946c0144be3c7cc4c4451cb6461a6ef7640f428a644e32e1ca9

Exploratory work on development data; not a performance claim.