Simple Linear Regression

The slope m measures how much y changes for each one-unit increase in x — for example, m = 2.3 means y rises by 2.3 units when x increases by 1. The intercept b is the predicted value of y when x = 0. Practically, b is meaningful only if x = 0 falls within a plausible range of your data; otherwise it is just a mathematical anchor for the line. For example, in a regression of weight on height, the intercept (weight at zero height) is biologically meaningless but mathematically necessary. Always interpret the intercept in the context of your specific data and only at values of x that make physical or business sense.

R² = 0.85 means that 85% of the total variability in your y values is explained by the linear relationship with x. The remaining 15% is unexplained variance — measurement noise, omitted variables, non-linear effects, or random fluctuation. According to NIST/SEMATECH guidelines, R² ≥ 0.80 is considered a very strong fit in most applied fields. However, R² is not the only quality metric: a model with R² = 0.85 can still produce poor predictions if it suffers from outliers, heteroscedasticity, or systematic patterns in residuals. Always combine R² with residual plots, cross-validation, and field-specific judgment about whether the residuals are tolerable for your use case.

A commonly cited rule of thumb is at least n = 10–20 observations for simple linear regression with one predictor. With n < 5 the estimates are highly unstable and confidence intervals are extremely wide. For inference (hypothesis testing on the slope), a minimum of n = 10 ensures the t-distribution approximation is reasonable. For prediction with moderate accuracy, n = 30+ is preferred. If you need to estimate confidence intervals around predictions for unseen x values, n = 50+ is recommended. For machine-learning contexts and complex industrial calibrations, n = 100+ becomes the working minimum. Statistical power for detecting a moderate slope (effect size ~0.5) typically requires n = 25-30 at α = 0.05.

Technically yes — R² can be negative if you compute it for a model that was not fit by OLS to your specific data. For example, if you impose a fixed slope or intercept from external knowledge, or if you compute R² on a held-out test set using a model trained on different data, you can get a negative value. A negative R² means the model fits worse than simply predicting the mean ȳ for every observation. However, when you use the standard OLS formulas on the same data you fit the model on, R² is mathematically constrained to be between 0 and 1 by construction. Negative R² on a test set is a strong signal of overfitting or model misspecification.

Simple linear regression uses exactly one predictor (x) to model y: ŷ = mx + b. Multiple linear regression uses two or more predictors: ŷ = b₀ + b₁x₁ + b₂x₂ + … + bₖxₖ. Adding more predictors almost always increases R², even if the new variables are irrelevant — which is why adjusted R² is used in multiple regression to penalize unnecessary predictors. Multiple regression also introduces complications: multicollinearity (predictors correlated with each other), interaction effects (where the slope of one predictor depends on another), and increased risk of overfitting when the number of predictors approaches the number of observations. Start with simple regression to understand the relationship, then add predictors only if theory or data justifies it.

Not necessarily. In social sciences and economics, R² values of 0.30–0.50 are common and considered meaningful because human behavior is inherently noisy. In physics or chemistry, R² < 0.99 might indicate a measurement or model problem. Context matters: an overfitted model on small data can show R² ≈ 1.0 yet predict new observations poorly — the classic warning sign of overfitting. Cross-validation (split data into train/test, or use k-fold) gives a more honest assessment of true predictive power. Also, a high R² combined with systematic patterns in residuals (curvature, increasing variance) indicates that despite the good fit number, the model is structurally wrong and should be replaced (e.g., with polynomial regression or log-transformed variables).

OLS regression makes four classical assumptions, often summarized as LINE: (1) Linearity — the true relationship between x and y is linear in the parameters. (2) Independence — observations are not correlated with each other (a critical violation occurs in time series data with autocorrelation). (3) Normality of residuals — residuals are approximately normally distributed (important for hypothesis tests and confidence intervals, less critical for point estimates). (4) Equal variance (homoscedasticity) — the variance of residuals is constant across all x values (heteroscedasticity is common in real data: residuals fan out as x grows). Violations of these assumptions can bias slope estimates or invalidate p-values and confidence intervals. Always check assumptions via residual plots, Q-Q plots, and tests like Durbin-Watson (autocorrelation) or Breusch-Pagan (heteroscedasticity).

In simple linear regression with one predictor, R² is exactly the square of the Pearson correlation coefficient: R² = r². For example, if r = 0.90, then R² = 0.81. If r = -0.7, then R² = 0.49. The sign of r tells you the direction of the relationship (positive or negative slope), which R² alone does not — R² is always ≥ 0 regardless of direction. This relationship breaks down in multiple regression, where R² generalizes but is no longer a simple square of a single correlation coefficient. In simple regression, you can compute one from the other interchangeably. Pearson r is preferred when you want to communicate a symmetric measure of association; R² is preferred when you want to communicate the fraction of variance explained.

Use it with caution. OLS is sensitive to outliers because it minimizes squared residuals, giving disproportionate weight to extreme points. A single outlier can shift the slope dramatically and inflate or deflate R². Best practice: (1) Always plot your data first — a scatter plot reveals outliers immediately. (2) Identify potential outliers using residual plots, leverage values (hat matrix diagonal), Cook's distance (combined leverage and residual), or studentized residuals. (3) Investigate before deleting — is the outlier a data-entry error, a genuine but unusual observation, or a sign of a different underlying mechanism? Never delete data just because it spoils your model. (4) Robust regression methods (Theil-Sen estimator, RANSAC, Huber regression, quantile regression) are less sensitive to outliers if they cannot be removed and provide a sanity check on OLS results.

Correlation measures the strength and direction of a linear association between two variables, using a symmetric metric like Pearson r (range -1 to +1). It tells you whether x and y move together but does not produce a predictive equation. Regression fits a specific equation y = mx + b that lets you predict y from x. Regression is asymmetric — regressing y on x gives a different line than regressing x on y. Use correlation when both variables are exchangeable and you only want to quantify association. Use regression when one variable is naturally the predictor (cause, independent variable, treatment) and the other is the response (effect, dependent variable, outcome) and you want to make predictions or estimate effect sizes.

Use log-transformations when: (1) the relationship is multiplicative rather than additive — common in economic data where percent changes matter more than absolute changes. (2) the data spans several orders of magnitude — for example, income, city populations, or response times. (3) residuals are heteroscedastic with variance proportional to the level — log transforms often stabilize variance. Common transforms: log y on log x (elasticity model), log y on x (exponential growth), y on log x (logarithmic growth). Use polynomial regression (y = a + bx + cx² + …) when a scatter plot shows clear curvature that a straight line cannot capture. Limit polynomial degree to 2 or 3 to avoid wild oscillations at the edges (Runge's phenomenon). If the relationship is more complex, consider spline regression or non-linear regression with theory-derived functional forms.

The slope's confidence interval gives a range of plausible values for the true population slope. A 95% CI like [1.8, 2.6] means that if you repeated the sampling and regression many times, 95% of such intervals would contain the true slope. The p-value for the slope tests the null hypothesis H₀: m = 0 (no linear relationship). A small p-value (typically p < 0.05) means the data are inconsistent with no relationship — you reject H₀ and conclude there is evidence of a linear association. Important caveats: p-values do not measure effect size (a tiny slope can have a tiny p-value with enough data); confidence intervals depend on assumptions (independence, normality, homoscedasticity); and statistical significance is not the same as practical significance. Always report both the point estimate of the slope and its confidence interval, not just the p-value.