Hypothesis Testing

The logic

A hypothesis test asks: is the effect I'm seeing real, or could plain chance explain it? We start by assuming nothing is going on, then check whether the data is too surprising for that assumption to survive.

Null and alternative hypotheses

• Null hypothesis $H_0$: no effect, no difference — the status quo. For example $\mu = 100$.
• Alternative hypothesis $H_a$: the claim we're testing. For example $\mu \neq 100$ (two-sided) or $\mu > 100$ (one-sided).

We never "prove" $H_0$. We either reject it or fail to reject it.

The test statistic

For a mean, the test statistic counts how many standard errors $\bar{x}$ sits from the null value $\mu_0$:

$z = \frac{\bar{x} - \mu_0}{\sigma / \sqrt{n}}$

The p-value

The p-value is the probability — assuming $H_0$ is true — of getting a result at least as extreme as the one observed.

• Small p-value → the data would be very unlikely under $H_0$ → evidence against $H_0$.
• Large p-value → the data is consistent with $H_0$.

Significance level

Choose a threshold $\alpha$ (often $0.05$) in advance:

• If $p \leq \alpha$: reject $H_0$ — the result is "statistically significant."
• If $p > \alpha$: fail to reject $H_0$.

Worked example

A factory claims its bottles average $\mu_0 = 500$ ml. A sample of $n = 64$ bottles gives $\bar{x} = 497$ with $\sigma = 8$. Test $H_a: \mu \neq 500$ at $\alpha = 0.05$.

$z = \frac{497 - 500}{8 / \sqrt{64}} = \frac{-3}{1} = -3$

For a two-sided test, the p-value is $P(|Z| > 3) \approx 0.0027$. Since $0.0027 < 0.05$, we reject $H_0$ — the fill amount differs significantly from 500 ml.

Two kinds of error

	$H_0$ is true	$H_0$ is false
Reject $H_0$	Type I error	correct
Fail to reject	correct	Type II error

• Type I error: rejecting a true null — a false positive. Its probability is $\alpha$.
• Type II error: failing to detect a real effect — a false negative. Its probability is $\beta$.
• Power $= 1 - \beta$: the chance of catching a real effect. A larger sample raises power.

Significant is not the same as important

With a big enough sample, even a tiny, meaningless difference can come out "statistically significant." Always look at the size of the effect, not just the p-value.

Key takeaways

• $H_0$ = no effect; $H_a$ = the claim. You reject or fail to reject — never prove $H_0$.
• $z = (\bar{x} - \mu_0)/(\sigma/\sqrt{n})$; the p-value measures how surprising the data is under $H_0$.
• Reject when $p \leq \alpha$ (commonly $0.05$).
• Type I = false positive (rate $\alpha$); Type II = false negative (rate $\beta$); power $= 1 - \beta$.
• Statistical significance $\neq$ practical importance.