Empirical Models, Simple Linear Regression, MLE & Least Squares
Open in the Padhaiko app5.1 Empirical Models, Simple Linear Regression, MLE & Least Square Estimator
Predictive modeling seeks mathematical relationships between observable input features (predictors) and an output quantity of interest (response). In data science, when first-principles physical laws are unknown, incomplete, or too complex to write down analytically, we build empirical models directly from observed measurements and estimate their parameters using Least Squares Estimation (LSE) or Maximum Likelihood Estimation (MLE).
Empirical Models vs. Theoretical Models
Empirical Model and Theoretical Model
An empirical model is a mathematical or statistical representation constructed primarily from observed sample data. It uses statistical estimation and pattern-recognition techniques to quantify relationships, trends, and correlations in real-world measurements without requiring derivation from underlying scientific laws.
A theoretical model is derived from established scientific principles, physical laws, axioms, and domain assumptions (for example, Newton's laws of motion or thermodynamic equations) rather than being fitted solely from a dataset.
| Aspect | Empirical Model | Theoretical Model |
|---|---|---|
| Basis | Relies directly on observed data and statistical estimation. | Relies on established domain theories, laws, and explicit assumptions. |
| Primary Purpose | Focuses on empirical description, pattern capture, and quantitative prediction. | Focuses on causal explanation, mechanistic understanding, and deduction. |
| Flexibility | Adapts closely to observed data patterns without rigid structural constraints. | Constrained by the functional form prescribed by underlying theory. |
| Generalization | Typically limited to the range and conditions of the studied dataset (risky to extrapolate). | Aims for broad, universal applicability wherever the underlying assumptions hold. |
| Example | Predicting house prices or monthly sales revenue via regression fitted to historical records. | Computing projectile trajectory via from classical mechanics. |
Simple Linear Regression
Simple linear regression is the foundational parametric empirical model for relating a single continuous predictor to a continuous response through a straight line plus random noise.
Simple Linear Regression Model
Given paired observations , the simple linear regression (SLR) model expresses the response as:
where:
- : observed dependent (response) variable for the -th observation.
- : independent (predictor / explanatory) variable for the -th observation.
- : population -intercept, representing the expected value .
- : population slope, representing the expected change in for a one-unit increase in .
- : unobserved random error (disturbance) capturing measurement noise and omitted factors.
The fitted regression line obtained from sample estimates is:
and the -th residual is .
For valid statistical inference and optimal estimator properties, the classical linear regression framework assumes:
- Linearity in parameters: .
- Zero-mean errors: .
- Homoscedasticity (constant variance): for all .
- Independence: for all .
- Normality (for MLE and exact / tests): .
Least Squares Estimator (LSE)
The Least Squares Estimator (LSE)—also called Ordinary Least Squares (OLS)—chooses to minimize the sum of squared vertical deviations between observed responses and predicted responses .
Least Squares Estimators for Simple Linear Regression
Let the Sum of Squared Errors (SSE) (or Residual Sum of Squares, RSS) objective function be:
Setting the partial derivatives and produces the normal equations:
Solving this linear system yields the unique closed-form OLS estimators (provided ):
where , , and in computational form:
To summarize how much variance in is explained by , we compute the coefficient of determination ():
where is the Total Sum of Squares. Values of close to indicate that most of the sample variation in is captured by the linear relationship with .
Maximum Likelihood Estimation (MLE) & Equivalence with LSE
While LSE relies purely on geometric distance minimization without assuming a probability distribution for the errors, Maximum Likelihood Estimation (MLE) adopts a probabilistic model and finds the parameter values that make the observed sample most probable.
Maximum Likelihood Estimation (MLE)
Suppose independent observations are drawn from a probability mass or density function parameterized by . The likelihood function is the joint probability (or density) of the observed sample treated as a function of :
The Maximum Likelihood Estimator chooses to maximize , or equivalently to maximize the log-likelihood :
MLE for Linear Regression with Normal Errors
Assume the simple linear regression errors are independent and identically normally distributed:
The likelihood and log-likelihood for are:
Maximizing with respect to is equivalent to minimizing , so the MLE slope and intercept coincide exactly with LSE:
Setting gives the MLE error variance:
(whereas the classical unbiased OLS variance estimator divides by degrees of freedom: ).
| Aspect | Least Squares Estimation (LSE) | Maximum Likelihood Estimation (MLE) |
|---|---|---|
| Objective | Minimizes the sum of squared residuals . | Maximizes the likelihood of the observed data. |
| Distributional Assumption | No strict parametric distribution required for point estimation. | Requires an explicit probability distribution (e.g., Normal, Binomial, Poisson). |
| Noise Variance | Not required to compute ; typically estimated afterward as . | Estimated jointly as part of the likelihood: . |
| Flexibility | Primarily suited to linear-in-parameter models with continuous responses. | General framework extending to logistic regression, count models, and non-Gaussian noise. |
| Equivalence Case | Special case of MLE when errors are i.i.d. . | Reduces to LSE for under i.i.d. Gaussian errors. |
Worked Examples
Fitting a Simple Linear Regression Line and Predicting Performance
An instructor records weekly study hours () and final exam scores () for students:
| Student | Study Hours | Exam Score |
|---|---|---|
| 1 | 2 | 52 |
| 2 | 4 | 60 |
| 3 | 6 | 71 |
| 4 | 8 | 79 |
| 5 | 10 | 88 |
Fit the simple linear regression model using Least Squares, predict the score for a student who studies hours per week, and compute the coefficient of determination .
Compute sample means and :
Compute deviations and sums of squares (, , ):
| 2 | 52 | |||||
| 4 | 60 | |||||
| 6 | 71 | |||||
| 8 | 79 | |||||
| 10 | 88 | |||||
| Sum | 0 | 0 |
Evaluate OLS slope and intercept :
Hence the fitted regression line is:
Predict score at and evaluate SSE and : At :
Computing fitted values and residuals :
- : ,
- : ,
- : ,
- : ,
- : ,
The least squares regression equation is . For study hours, the predicted exam score is , and the model explains of the sample variance in scores ().
Computational Formula & MLE Error Variance vs. Unbiased Variance
A systems engineer measures API response latency (in ms) at server load levels (in hundreds of requests per second):
| Load | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| Latency | 14 | 19 | 27 | 32 |
Using the raw-sum computational formulas, find the MLE estimates , , and under Gaussian errors , and compare with the unbiased OLS variance .
Compute raw sums and sample means:
Compute , , and MLE regression coefficients:
Under i.i.d. Gaussian errors, the MLE slope and intercept equal the OLS estimators:
Thus .
Compute SSE, , and unbiased : Fitted values and squared residuals:
- :
- :
- :
- :
Comparing the two variance estimators:
Because MLE divides by instead of , it slightly underestimates the population error variance in small samples.
, (), , , and the unbiased OLS error variance is .
Deriving an MLE Estimator Directly from the Log-Likelihood
A physical sensor obeys a zero-intercept linear calibration model , where . Given the calibration pairs , derive the MLE formula for directly from the log-likelihood, then compute and .
Write the log-likelihood : Since , the log-likelihood is:
Differentiate with respect to and solve:
Evaluate and numerically:
Residuals and SSE:
(model ), and .
Exercises
Exercises on Empirical Models, Simple Linear Regression, LSE & MLE
Define an empirical model and explain how it differs from a theoretical model across basis, purpose, flexibility, and generalization. Give one concrete example of each. [Assignment-5, Q1]
Defining empirical and theoretical models:
- An empirical model is built primarily from observed data using statistical or machine learning estimation to identify relationships, correlations, and predictive patterns without relying on underlying first-principles laws.
- A theoretical model is derived from established domain theories, scientific axioms, and structural assumptions (for example, conservation laws or circuit equations).
Comparing across the four dimensions:
- Basis: Empirical models rely on measured sample data; theoretical models rely on foundational theory and explicit assumptions.
- Purpose: Empirical models prioritize quantitative description and prediction; theoretical models prioritize causal explanation and mechanistic understanding.
- Flexibility: Empirical models adapt closely to observed patterns in data; theoretical models are tightly constrained by theoretical structure.
- Generalization: Empirical models are generally valid only within the range of the studied data (extrapolation is hazardous), whereas theoretical models generalize broadly wherever their assumptions hold.
Concrete examples:
- Empirical model: A simple linear regression equation fitted from historical monthly records to predict retail revenue from advertising spend.
- Theoretical model: Ohm's law or the ideal gas law derived from physics.
An empirical model learns predictive relationships directly from observed data, whereas a theoretical model is deduced from established scientific laws and assumptions. They differ in basis (data vs. theory), purpose (prediction vs. explanation), flexibility (data-adaptive vs. theory-constrained), and generalization (bounded by observed data range vs. broad applicability when assumptions hold).
Explain the concept of the Least Squares Estimator (LSE) for simple linear regression , and derive the closed-form estimators and mathematically. [Assignment-5, Q3]
Objective of LSE: The Least Squares Estimator selects the intercept and slope that minimize the Sum of Squared Errors (SSE) between observed values and predicted values :
First-order optimality conditions (normal equations): Taking partial derivatives of with respect to and and setting both to zero:
Solving for and : Dividing the first equation by gives:
Substituting into the second normal equation:
Using the identities and :
The Least Squares Estimator minimizes , giving:
Given the following dataset of observations:
| 1 | 2 | 3 | 4 | 5 | |
|---|---|---|---|---|---|
| 2 | 3 | 5 | 4 | 6 |
Find the simple linear regression equation using the least squares estimator, and compute the exact Sum of Squared Errors (SSE) and . [Assignment-5, Q4]
Compute sample means:
Compute , , and :
Calculate exact slope and intercept :
Thus, the fitted regression equation is:
Compute SSE and : Evaluating at each point:
- : ,
- : ,
- : ,
- : ,
- : ,
, , giving the exact regression line , with and .
Define Maximum Likelihood Estimation (MLE) and show how it relates to simple linear regression. Compare the assumptions underlying LSE and MLE. [Assignment-5, Q5]
Definition of MLE: Given i.i.d. observations modeled by a density or probability function , the likelihood function is:
MLE chooses the parameter estimate that maximizes the probability (or density) of observing the sample data.
Connection to linear regression: Assume the simple linear regression model with , so . The log-likelihood is:
Maximizing over requires minimizing . Therefore, LSE is a special case of MLE when regression errors are normally distributed with constant variance.
Comparing assumptions of LSE and MLE:
- LSE: Minimizes squared residuals directly. Requires no explicit distributional family (such as normality) to compute point estimates , and estimates variance separately via .
- MLE: Requires specifying a full probability distribution for (e.g., Gaussian for linear regression, Bernoulli for logistic regression). Estimates all parameters including jointly, and generalizes naturally to non-Gaussian models.
MLE estimates parameters by maximizing the sample likelihood . Under i.i.d. normal regression errors , maximizing the log-likelihood minimizes , making LSE identical to MLE for while MLE requires a specified probability distribution and LSE does not.
An agronomist tests the effect of nitrogen fertilizer (kg/hectare) on wheat yield (quintals/hectare) across plots:
| Fertilizer | 10 | 20 | 30 | 40 | 50 | 60 |
|---|---|---|---|---|---|---|
| Yield | 25 | 34 | 46 | 54 | 65 | 72 |
- Fit the OLS regression line .
- Predict the wheat yield when of fertilizer is applied.
- Compute the residual at .
Compute means and :
Compute and : Deviations are :
Compute OLS slope and intercept :
Thus .
Predict at and compute residual at :
At :
- (exact intercept ).
- Predicted yield at is .
- Residual at is .
For the observations below, assume with :
| 2 | 3 | 4 | 5 | 6 | |
|---|---|---|---|---|---|
| 5 | 8 | 9 | 12 | 16 |
Compute the MLE estimates , , and , as well as the unbiased OLS error variance .
Compute means, , and :
Compute and :
So .
Compute SSE, , and : Residuals:
- :
- :
- :
- :
- :
, , , and .
How does simple linear regression support predictive analytics in data science, and why should analysts be cautious when using a fitted empirical regression line for extrapolation?
Role of simple linear regression in predictive analytics:
- Point Forecasting: Substituting unseen predictor values into produces quantitative forecasts for future outcomes.
- Marginal Impact Quantification: The slope quantifies the direction and exact rate of change in per unit increase in , guiding sensitivity and "what-if" scenario analysis.
- Uncertainty & Fit Assessment: Metrics such as and residual variance quantify how reliably the predictor explains target variation.
Why extrapolation is hazardous: An empirical regression line is calibrated solely on observed pairs within . Outside that domain, physical saturation, diminishing returns, threshold effects, or structural regime shifts can make the linear relationship invalid, causing prediction errors to grow rapidly.
Simple linear regression supports predictive analytics through quantitative forecasting (), marginal sensitivity quantification (), and fit diagnostics (, ). Analysts must avoid extrapolation beyond the observed predictor range because empirical models do not guarantee that linearity persists outside the sampled regime.