Linear Regression & Correlation

Scatter plots and association

A scatter plot shows pairs $(x, y)$ as points. The shape hints at a relationship:

• Up-and-right pattern → positive association.
• Down-and-right → negative association.
• No pattern → no linear association.

Correlation: how strong is the line?

The correlation coefficient $r$ measures the strength and direction of a linear relationship.

• $r$ ranges from $-1$ to $+1$.
• $r = 1$: perfect positive line. $r = -1$: perfect negative line.
• $r = 0$: no linear relationship (a non-linear one could still exist).
• $|r| > 0.7$ is usually "strong"; $|r| < 0.3$ is "weak."

Important: $r$ only measures linear association. A perfect quadratic relationship can have $r$ near zero.

The line of best fit

If a linear pattern exists, the least-squares regression line minimizes the sum of squared vertical distances from the data points:

$\hat{y} = a + bx$

• $\hat{y}$ is the predicted $y$.
• $b$ is the slope: predicted change in $y$ per unit increase in $x$.
• $a$ is the intercept: predicted $y$ when $x = 0$.

The slope satisfies:

$b = r \cdot \frac{s_y}{s_x}$

where $s_x$ and $s_y$ are the standard deviations. The line always passes through the point $(\bar{x}, \bar{y})$.

Predicting with the line

Example. A regression line for hours-studied vs. test-score is $\hat{y} = 50 + 4x$.

For a student who studies 6 hours: $\hat{y} = 50 + 4(6) = 74$. Predicted score: 74.

$r^2$ — the coefficient of determination

Squaring $r$ gives the proportion of variation in $y$ explained by the line.

• $r = 0.8 \implies r^2 = 0.64$: the line explains 64% of the variation in $y$.
• $r^2$ is always between 0 and 1.

Correlation is not causation

A strong $r$ tells you two variables move together. It does not tell you one causes the other. Ice cream sales and shark attacks both rise in summer — ice cream doesn't cause sharks.

Always ask: could a third variable explain the link?

Key takeaways

• $r$ measures the strength of a linear relationship; $-1 \leq r \leq 1$.
• Best-fit line: $\hat{y} = a + bx$; it passes through $(\bar{x}, \bar{y})$.
• $r^2$ is the fraction of variation in $y$ explained by the line.
• Correlation $\neq$ causation. Watch for lurking variables.