t-test (group comparison)
About this app
Two groups as overlaid distributions + boxplots: t-statistic, p-value (exact t-distribution) and Cohen's d. The catch — the same mean difference becomes significant with a smaller σ or a larger n, while the effect stays the same.
The statistics
The two-sample t-test compares the means of two independent groups. The t statistic t = (x̄₁ − x̄₂) / (s_p · √(1/n₁ + 1/n₂)) relates the mean difference to the pooled standard deviation s_p and the sample size. The two-sided p-value states how likely a difference this large would be if the true means were equal. Cohen’s d = (x̄₁ − x̄₂)/s_p measures effect size independently of n. A significant difference is therefore not automatically a large one.
What you see
At the top are the two distributions N(μ₁, σ) and N(μ₂, σ) with their mean lines; a tinted area marks their overlap. Below are box plots of two drawn samples. A values panel lists both sample means, their difference, the pooled standard deviation, the degrees of freedom, the t statistic, the two-sided p-value from the exact t distribution and Cohen’s d. Two presets show a clear difference and a strong overlap.
Things to try
Keep the means μ₁ and μ₂ fixed and reduce the spread σ: the distributions overlap less, t grows and p falls. Reset σ and increase n per group instead; again the difference becomes significant, even though Cohen’s d stays roughly the same. Draw new samples several times and see how much p fluctuates when n is small. Choose the overlap preset and find the n at which the test detects the difference.
Frequently asked questions
- When do you use a t-test?
- The two-sample t-test checks whether the means of two independent groups differ, such as a treatment and a control group. It assumes roughly normal data or large enough samples; the classic version assumes equal variances. With unequal variances Welch’s test is used.
- What is Cohen’s d?
- Cohen’s d is a measure of effect size: the difference between the means divided by the pooled standard deviation. Following Cohen, values around 0.2 count as small, around 0.5 as medium and around 0.8 as large. Unlike the p-value, d does not depend on sample size.
- Why does a difference become significant with larger n?
- The standard error of the mean difference shrinks with 1/√n. The same difference therefore spans many standard errors when n is large, so t rises and p falls. With enough data even a tiny effect becomes significant, which is why the effect size should always be reported too.
- What can I set in the t-test app?
- You set the means μ₁ and μ₂, the common spread σ and the sample size n per group, draw new samples and choose between the clear and overlap presets. The app shows t, degrees of freedom, p-value and Cohen’s d.
Subject: Statistics
More from Statistics
- Anscombe's quartet
- Bias-variance / overfitting
- Box plot workshop
- Geometric mean
- Law of large numbers
- Histogram workshop
- Hypothesis test & p-value
- Confidence interval
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps