Skip to content
All apps

p-hacking & multiple testing

About this app

Many tests with no real effect yield "significant" hits purely by chance. The p-value distribution, the family-wise error rate FWER = 1−(1−α)^m, the garden of forking paths and the Bonferroni correction — beware of data dredging and many KPIs.

The statistics

Under the null hypothesis, when there is no real effect, a test’s p-value is uniformly distributed on [0, 1]. A fraction α of all tests therefore falls below the significance threshold purely by chance. Running m independent tests gives at least one false alarm with probability FWER = 1 − (1 − α)ᵐ; for m = 20 and α = 0.05 that is about 64 %. P-hacking means trying variants, subgroups or analyses until something turns significant and reporting only that. The Bonferroni correction tests each hypothesis at α/m and keeps the family-wise error rate below α.

What you see

The app simulates m tests and shows each p-value as a dot, with the “significant” ones below the threshold highlighted. A histogram of the p-values is roughly flat in the no-effect scenario; with a real effect small p-values pile up. A values panel lists the number of tests, the threshold, the hits found, the expected number α·m, the family-wise error rate, the Bonferroni threshold α/m and the hits after correction. The keep-searching button plays out the garden of forking paths and counts the attempts until the first hit.

Things to try

Stay in the no-effect scenario and re-run the simulation several times: with 20 tests you usually find one or two “significant” results although there is nothing to find. Increase m and watch the family-wise error rate approach 1. Switch on the Bonferroni correction; the false alarms largely disappear. Then switch to the real-effect scenario and set the effect size d: now hits appear quickly and rightly, and you can see how much power Bonferroni costs.

Frequently asked questions

What is p-hacking?
P-hacking means trying many analysis variants, subgroups, outcome measures or exclusion rules until a result drops below the significance threshold, and then reporting that result selectively. It pushes the false-positive rate far above the nominal α. Remedies include stating hypotheses in advance (preregistration) and correcting for multiple testing.
What is the family-wise error rate (FWER)?
The family-wise error rate is the probability of making at least one type I error in a group of tests. For m independent tests at level α it is 1 − (1 − α)ᵐ. With just 20 tests at α = 0.05 it is already about 64 %.
How does the Bonferroni correction work?
The Bonferroni correction tests each individual hypothesis at the stricter threshold α/m instead of α. This keeps the family-wise error rate at α or below. The price is lower power, so real effects are missed more easily.
What can I set in the p-hacking app?
You choose the number of tests m, the significance level α and the scenario (no effect, or a real effect with effect size d), switch on the Bonferroni correction and re-run the simulation. The keep-searching button shows how quickly enough attempts produce a chance hit.

Subject: Statistics

More from Statistics

A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps