Multiple & Polynomial Regression, Categorical Regressors & Model Building
Open in the Padhaiko app5.2 Multiple Linear Regression, Polynomial Regression & Model Building
Real-world outcomes rarely depend on a single explanatory variable or follow a purely straight-line trend. Multiple linear regression generalizes simple linear regression to predictors, polynomial regression captures curved relationships using powers of predictors while preserving linearity in parameters, and categorical encoding brings qualitative attributes into the same matrix framework.
Multiple Linear Regression: Scalar and Matrix Form
Multiple Linear Regression Model
For observations and independent predictors , the multiple linear regression (MLR) model in scalar form is:
where:
- : observed response for the -th sample.
- : value of the -th predictor for the -th sample.
- : intercept (expected value of when all ).
- : partial regression coefficients.
- : zero-mean random error term.
Stacking all equations produces the compact matrix representation:
with exact matrix dimensions:
Here is called the design matrix; its leading column of ones multiplies the intercept .
Normal Equations and Matrix OLS Estimator
To estimate by Ordinary Least Squares (OLS), we minimize the Sum of Squared Errors (SSE):
Differentiating with respect to the vector and setting the gradient to zero gives:
which is the system of normal equations. Whenever the columns of are linearly independent (so is invertible), the unique OLS estimator is:
The vector of fitted values is , and the residual vector is .
Polynomial Regression
Polynomial Regression Model
Polynomial regression models a curved (non-linear in ) relationship between a predictor and a response by adding integer powers :
Although the regression curve is nonlinear as a function of , the model remains linear in the parameters . Defining columns turns the polynomial model into a special case of multiple linear regression:
Categorical Regressors & Coefficient Interpretation
Regression predictors can be either continuous (numerical measurements such as age, area, or temperature) or categorical (discrete labels such as region, fuel type, or education tier).
Interpreting Continuous and Categorical Regressors
- Continuous Predictor Coefficient (): Represents the expected change in the dependent variable for a one-unit increase in , holding all other predictors constant (ceteris paribus).
- Categorical Predictor (Dummy / Indicator Encoding): A qualitative variable with distinct categories is encoded using binary dummy variables , omitting one category as the reference (baseline) category:
The intercept (plus continuous terms) describes the baseline response for the reference group, and each dummy coefficient measures the expected difference in between category and the reference category, holding all other predictors fixed.
Model Building, Evaluation Metrics & Complexity Trade-offs
When building multiple or polynomial regression models, adding predictors always involves balancing fit quality against generalization and simplicity.
| Direction | Mechanism | Effect on Model Performance |
|---|---|---|
| Improvements | Captures genuine signal | Relevant variables supply additional explanatory information. |
| Reduces model bias | Richer predictors approximate complex underlying relationships more faithfully. | |
| Lowers training error | Training MSE decreases and raw never decreases when a predictor is added. | |
| Drawbacks | Overfitting | Weak or noisy predictors fit sample fluctuations rather than true signal, hurting test accuracy. |
| Increased variance | More parameters make sensitive to small changes in training data. | |
| Multicollinearity | Correlated predictors inflate and destabilize individual coefficient signs. | |
| Complexity & cost | Higher computational cost, harder interpretation, and greater data-collection burden without generalization gain. |
Evaluation Metrics and Complexity-Penalized Criteria
Let be the sample size, the number of predictors (so regression coefficients including ), , and :
- Mean Squared Error (MSE) and Root Mean Squared Error (RMSE):
Lower MSE and RMSE indicate smaller average prediction errors (with RMSE sharing the units of ).
- Coefficient of Determination ():
Measures the proportion of variance in explained by the model, but always increases (or stays constant) whenever any new predictor is added, even pure noise.
- Adjusted ():
Adjusts for the number of predictors and increases only if a new variable improves the fit more than expected by chance.
- Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC):
(More generally, and , where is the number of estimated parameters.) Lower AIC and BIC values indicate a better trade-off between goodness of fit and model parsimony.
| Criterion | Objective Direction | Complexity Penalty | Primary Use |
|---|---|---|---|
| Higher is better | Scales residual variance by degrees of freedom . | Comparing linear regression models on the same dataset and response scale. | |
| Lower is better | Adds to deviance / . | Selecting models optimized for out-of-sample predictive accuracy. | |
| Lower is better | Adds (stronger penalty than AIC whenever , since ). | Identifying compact, parsimonious models and avoiding superfluous predictors. |
Worked Examples
Matrix OLS Solution for Multiple Linear Regression with Two Predictors
A cloud cost analyst models monthly server cost (in s) from CPU cores and RAM blocks across deployments:
| Deployment | CPU | RAM | Cost |
|---|---|---|---|
| 1 | 1 | 2 | 10 |
| 2 | 1 | 6 | 19 |
| 3 | 3 | 2 | 17 |
| 4 | 3 | 6 | 24 |
| 5 | 2 | 4 | 20 |
- Form the design matrix and response vector , and solve the normal equations .
- Compute SSE, MSE, RMSE, , and .
Construct , , , and :
Computing and :
Invert and compute : The determinant of is:
Evaluating the adjugate divided by yields:
Multiplying by :
Hence the fitted model is .
Evaluate fitted values, SSE, MSE, RMSE, , and : With :
- : ,
- : ,
- : ,
- : ,
- : ,
, so . Accuracy metrics: , , , , and .
Fitting a Quadratic Polynomial Regression Model
Fit a quadratic polynomial regression model to the five observations below, compute , and predict at :
Form the polynomial design matrix and compute , : Adding the quadratic feature column :
Because is symmetric around , and , simplifying :
Solve the normal equations: The middle equation decouples immediately: . For , the block has determinant and inverse :
Thus the fitted quadratic curve is:
Compute and predict at : With :
- :
- :
- :
- :
- :
At :
The fitted polynomial regression model is , achieving (), with predicted value .
Model Selection Using Adjusted R², AIC, and BIC
Two candidate multiple regression models are fitted on the same dataset of observations with :
- Model A: uses predictors ( regression coefficients), achieving .
- Model B: adds extra predictors (, regression coefficients), achieving .
Compute , , , and for both models, and decide which model should be selected.
Compute and for both models: For Model A ():
For Model B ():
Notice that increases from to , whereas drops slightly from to .
Compute AIC and BIC for both models: Using , , and :
- Model A ():
- Model B ():
Compare and select the better model: Although Model B reduces SSE by and raises raw , adding three predictors costs three degrees of freedom and adds complexity penalties of (AIC) and (BIC) that outweigh the deviance reduction (). Model A achieves higher (), lower AIC (), and substantially lower BIC ().
Select Model A (). Raw favors Model B ( vs. ), but all complexity-aware criteria favor Model A: , , and .
Exercises
Exercises on Multiple & Polynomial Regression, Categorical Regressors & Model Building
Explain simple linear regression, multiple linear regression, and polynomial regression models. Why can polynomial regression be estimated using the exact same least squares matrix formula as linear regression? [Assignment-5, Q2]
Simple Linear Regression (SLR): Relates one independent variable to one dependent variable via a straight line:
It predicts or explains from a single predictor, where is the intercept and is the slope.
Multiple Linear Regression (MLR): Extends SLR to two or more independent predictors :
It models the combined additive effect of several predictors on to improve prediction accuracy and control for confounding variables.
Polynomial Regression and Linearity in Parameters: Models a curved relationship between and by incorporating higher-order powers of :
Even though are nonlinear transformations of , the predicted response is still a linear combination of the unknown parameters . Treating as columns of the design matrix casts polynomial regression directly into , so applies unchanged.
Simple linear regression uses one predictor (), multiple linear regression uses predictors (), and polynomial regression includes powers of a predictor () to fit curves. Because polynomial regression remains linear in the parameters , setting lets it use the exact same OLS matrix estimator .
Derive the general matrix form of a multiple linear regression model with observations and predictors, specify the dimensions of each matrix/vector, and derive the least squares estimator . [Assignment-5, Q6]
Scalar equations and vector-matrix stacking: For :
Defining:
gives .
Minimizing Sum of Squared Errors (SSE):
Differentiating with respect to and setting the gradient to zero:
Normal equations and OLS solution: Rearranging gives the normal equations:
Assuming has full column rank (so is invertible), left-multiplying by yields:
In matrix form, where , , , and . Minimizing yields the normal equations and estimator .
Explain how adding more predictors to a regression model can improve or worsen performance, and discuss the trade-off between model complexity and interpretability. [Assignment-5, Q7 & Q8]
How adding predictors can improve performance:
- More information: Relevant predictors capture additional systematic patterns in the outcome.
- Reduced bias: A richer feature set allows the model to approximate the true underlying relationship more accurately.
- Lower training error: Adding variables decreases (or maintains) training MSE and raises training .
How adding predictors can worsen performance:
- Overfitting: Irrelevant or noisy predictors cause the model to memorize training noise, degrading test generalization.
- Increased variance: Parameter estimates become overly sensitive to small perturbations in the sample.
- Multicollinearity: Strongly correlated predictors inflate coefficient variances and make individual effects unstable.
- Higher complexity and cost: Additional features raise data collection, storage, and compute overhead without improving out-of-sample accuracy.
Complexity vs. interpretability trade-off: Simpler models (such as small linear regressions with few predictors) are transparent, easy to audit, and straightforward to explain to stakeholders, though they may underfit complex non-linear patterns. More complex models (high-degree polynomials or many interacting features) capture intricate relationships and can yield higher predictive accuracy on rich data, but become opaque and harder to interpret or trust in high-stakes decisions.
Adding relevant predictors reduces bias and lowers training error, but adding weak or redundant predictors causes overfitting, higher variance, multicollinearity, and extra cost without generalization gain. Similarly, model complexity and interpretability trade off: simpler models are transparent and easy to explain, whereas complex models capture finer patterns at the expense of interpretability.
Discuss the roles of MSE, RMSE, , Adjusted , AIC, and BIC in regression model evaluation and selection. [Assignment-5, Q9]
Error magnitude metrics (MSE and RMSE):
- measures average squared prediction error (larger MSE indicates worse fit).
- expresses typical error magnitude in the original units of .
Explained variance ( and Adjusted ):
- measures the fraction of response variance explained, but never decreases when predictors are added, so it cannot detect useless variables on its own.
- Adjusted modifies by dividing by degrees of freedom: . It penalizes unnecessary predictors and increases only when an added feature improves fit beyond chance.
Information criteria (AIC and BIC):
- AIC () balances goodness of fit against parameter count , favoring models with lower values.
- BIC () imposes a stronger complexity penalty than AIC whenever (), favoring simpler, parsimonious models. Lower AIC or BIC indicates a preferred model.
MSE and RMSE quantify prediction error magnitude (lower is better). While raw increases monotonically with added predictors, Adjusted penalizes superfluous predictors (higher is better), and AIC and BIC balance likelihood fit against complexity penalties and (lower is better), with BIC penalizing extra parameters more strictly.
Formulate a multiple regression model with three predictors—two continuous and one categorical variable with three levels (e.g., Low, Medium, High)—explain how its parameters are estimated, and interpret the coefficients for both continuous and categorical predictors. [Assignment-5, Q10 & Q11]
Formulating the model with dummy variables: Let be the response (e.g., annual salary), and be continuous predictors (e.g., years of experience and weekly hours), and let the 3-level categorical variable be Education Tier . Choosing Bachelor as the reference category, define two dummy variables :
- if Master, otherwise.
- if PhD, otherwise.
The model is:
Parameter estimation and accuracy evaluation: Writing the observations in matrix form (with design matrix columns ), the five coefficients are estimated via OLS: . Model accuracy is evaluated using , RMSE, AIC, and out-of-sample MSE.
Interpreting continuous and categorical coefficients:
- Continuous coefficients (): is the expected change in for a one-unit increase in , holding and education tier constant.
- Intercept (): Expected value of for the reference category (Bachelor) when .
- Categorical coefficients (): Interpreted relative to the reference category. Holding and fixed, is the expected difference in between Master and Bachelor, and is the expected difference between PhD and Bachelor.
With two continuous predictors () and a 3-level categorical predictor encoded by two dummy variables () relative to a reference level, the model is , estimated via . Continuous coefficients measure expected change in per unit predictor increase (other variables fixed), while categorical dummy coefficients measure the expected shift relative to the reference category.
Given the observations for two predictors and :
| 1 | 1 | 1 | 6 |
| 2 | 1 | 3 | 13 |
| 3 | 2 | 2 | 10 |
| 4 | 3 | 1 | 11 |
| 5 | 3 | 3 | 15 |
- Find the OLS estimator for .
- Predict at .
- Compute SSE, MSE, , and .
Form and :
Invert and compute : With :
So .
Predict at and compute SSE, MSE, , : At :
Fitted values and squared residuals ():
- :
- :
- :
- :
- :
- .
- .
- , , , .
Fit the quadratic polynomial regression model to the dataset below, and compute SSE, , and :
Compute and : With and :
Solve for : From the decoupled middle equation, . Using on the remaining block:
Thus .
Compute SSE, , and : With :
- :
- :
- :
- :
- :
, with , , and .