Does our bootstrap test keep its 5% false-alarm rate on stock split data with ten times the simulations?
Study 2 · Changed a rule or plan · recorded 26 September 2026 · answer: Not at the size tested · full paper with figures
Correction, 30 September 2026. Technical wording rewritten in shorter sentences under the readability rule; the meaning is unchanged.
Correction, 1 October 2026. The outcome label now follows the study's value record, which counts this study as having changed a rule or a plan.
Correction, 15 October 2026. This text was written and reviewed after the study finished, in a catch-up of the earliest studies; the result, outcome and dates are unchanged.
The question
An earlier methods study judged on 1,000 simulations per case that our bootstrap test held false alarms in each direction, a wrongly found fall or rise, each at its 2.5% target on stock split data, so the planning rules chose it. We repeat the check with 10,000 fresh simulations per case, enough to see a one-point excess, changing nothing else.
Rules fixed before the result
We fixed in advance six cases of 10,000 simulations: all events at 221 and 548, the 417 reverse splits, the 130 forward splits, bell-curve values to check our code, and a planted 2.28-point shortfall. With the code check passed, the test raised too many false alarms if the 95% range of its total rate at 221 or on the reverse splits lay wholly above 5%, and held if no tail's range lay wholly above 2.5% there or at 548.
Data and method
This is a methods study on the five-day returns of the 548 earlier stock split events, shifted to a mean of zero so that every rejection is a false alarm, and redrawn in whole companies. Each case runs five tests on the same draws, the bootstrap test among them, and counts how often each rejects in each tail against its 2.5% target, with a 95% range on every rate.
The answer
At 221 events the test raised false alarms above its 5% target, the excess mostly in the lower tail, and on the 417 reverse splits its lower tail ran above its 2.5% target; only at 548 events did both tails stay on target. A check on made-up bell-curve values with the same company and date structure kept the target, which gives no sign that the excess comes from our simulation code. None of the five tests compared kept both tails from running above target at 221 events and on the reverse splits. The earlier study's rates are not contradicted: at 1,000 simulations their ranges were too wide to rule out the targets.
What it changes
It changed the planning rules. The bootstrap test is no longer the test chosen for stock split studies of 221 to 417 events and no other test replaces it; any stock split study with fewer than 548 events must set its threshold by simulation. A proposed rule to build confidence intervals with the bootstrap test was withdrawn, and a one-tailed design on the 130 forward splits may use only the tests whose lower tail did not run above target there.
What remains uncertain
Only the five-day horizon was tested, on the 548 split events the earlier study had used, so the rates hold for that sample. Even with 10,000 simulations a tail rate slightly above 2.5% could go unseen, and redrawing by company does not treat companies that move together on the same dates. The check on our code was run at 221 events only, and designs that compare each split with ordinary days of the same share are not covered.