Try it while you read
Open the free Hypothesis Tests tool and click Load sample: it reproduces the case study below.
Theory: signal or noise?
Two samples never have exactly the same mean, even from the same process. A hypothesis test asks whether the observed difference is larger than the random variation would explain. It starts from the null hypothesis (no difference) and calculates the p-value: the probability of a difference at least this large if the null hypothesis were true.
| Question | Test |
|---|---|
| Mean versus a target | One-sample t-test |
| Two independent groups | Two-sample t-test (Welch) |
| Same parts before and after | Paired t-test |
| Three or more groups | One-way ANOVA |
| Spread of two groups | F-test |
| Counts in categories | Chi-square test |
Key point
Step by step
- Formulate the question, e.g. “does machine B produce a different mean than machine A?”.
- Choose the test from the table.
- Paste the data and name the groups.
- Set α = 0.05 unless the risk calls for 0.01.
- Read the verdict and check the dot plot for outliers or unusual shapes.
Case study: two machines, same part
Seven parts from machine A and eight from machine B are measured (the Load sample data of the two-sample test). The Welch test gives t = −3.78, p = 0.0023: the difference is significant. The 95 % confidence interval of the difference A − B is −0.71 to −0.19, so machine B runs between 0.19 and 0.71 units higher.
Whether that matters depends on the tolerance: with a tolerance of ±1 the shift uses up to a third of the half tolerance, so the team centres machine B. With a tolerance of ±10 it would be irrelevant.
Common mistakes
- Using a two-sample test for paired data (same parts measured twice).
- Running many t-tests instead of one ANOVA for several groups.
- Reading “not significant” as “no difference”: small samples miss real differences, see the sample size calculator.
Frequently asked questions
Is p < 0.05 proof?
No. It means the observed difference would be unlikely if there were no real difference. It says nothing about the size or practical importance of the difference; the confidence interval does.