Does our bootstrap test keep its false-alarm rate on made-up bell-curve values at other sample sizes?
Study 42 · Checked a method · recorded 7 October 2026 · answer: Yes · full paper with figures
The question
An earlier check found our bootstrap test raised false alarms too often on real returns, mostly in the lower tail (about 3.6% at 221 events and 2.9% at 417, against a 2.5% target), but not on made-up bell-curve values at one size. Repeating the made-up check at three more sizes tells whether the test or the returns caused the excess.
Rules fixed before the result
The plan fixed the cases, reading rules and expected answer before any repeat was drawn. A rate failed if its 95% range lay wholly above target (5% overall, 2.5% per tail), and the tails were unbalanced if the range of their difference lay wholly on one side of zero. The excess was to be laid on the test if, at 221 or 417 events, the lower tail failed or the imbalance leaned to it; otherwise on the returns.
Data and method
We drew made-up bell-curve values and grouped them by company like the earlier check's events, at its four sample sizes: 130, 221, 417 and 548 events. Each of six cases was repeated ten thousand times; each repeat drew whole companies at random with replacement, as the earlier check did, and counted how often the unchanged bootstrap test wrongly rejected, in total and in each tail. A control at 221 events matched the earlier check's own made-up result.
The answer
Yes. On the three cases that decide, at 221 and 417 events, the test wrongly rejected in the lower tail in 2.4% to 2.5% of repeats, against a target of 2.5%, and neither tail led the other beyond chance. Across all six cases and four sizes, none of the 24 checks on the total rate, each tail and the balance between tails departed from target, where up to about one was expected by chance alone. So the earlier one-sided excess belongs to the real returns, not to the test. This is the answer the plan expected, with high confidence.
What it changes
No rule or plan changed. The earlier limit on using this test at 221 to 417 events keeps its stated reason, now shown to lie in the returns rather than the test, and the earlier check that the test itself is sound now covers four sample sizes instead of one.
What remains uncertain
The values were made up, never market data, and some group structures were assumed; each structure carried one fixed set of values, though a second set at 417 events read the same. Excesses smaller than about 0.3 percentage points per tail would not be reliably seen. The result says the earlier excess comes from the returns, not which property of them, such as their skew, causes it.
How it was made
Run took 33 minutes; independent code review: approve; the recorded calculation reruns to the same result.
Exploratory work on development data; not a performance claim.