PadhaikoAll subjects

Empirical Models, Simple Linear Regression, MLE & Least Squares

Foundation of Data Science

Open in the Padhaiko app

5.1 Empirical Models, Simple Linear Regression, MLE & Least Square Estimator

Predictive modeling seeks mathematical relationships between observable input features (predictors) and an output quantity of interest (response). In data science, when first-principles physical laws are unknown, incomplete, or too complex to write down analytically, we build empirical models directly from observed measurements and estimate their parameters using Least Squares Estimation (LSE) or Maximum Likelihood Estimation (MLE).

Empirical Models vs. Theoretical Models

Definition

Empirical Model and Theoretical Model

An empirical model is a mathematical or statistical representation constructed primarily from observed sample data. It uses statistical estimation and pattern-recognition techniques to quantify relationships, trends, and correlations in real-world measurements without requiring derivation from underlying scientific laws.

A theoretical model is derived from established scientific principles, physical laws, axioms, and domain assumptions (for example, Newton's laws of motion or thermodynamic equations) rather than being fitted solely from a dataset.

AspectEmpirical ModelTheoretical Model
BasisRelies directly on observed data and statistical estimation.Relies on established domain theories, laws, and explicit assumptions.
Primary PurposeFocuses on empirical description, pattern capture, and quantitative prediction.Focuses on causal explanation, mechanistic understanding, and deduction.
FlexibilityAdapts closely to observed data patterns without rigid structural constraints.Constrained by the functional form prescribed by underlying theory.
GeneralizationTypically limited to the range and conditions of the studied dataset (risky to extrapolate).Aims for broad, universal applicability wherever the underlying assumptions hold.
ExamplePredicting house prices or monthly sales revenue via regression fitted to historical records.Computing projectile trajectory via from classical mechanics.

Simple Linear Regression

Simple linear regression is the foundational parametric empirical model for relating a single continuous predictor to a continuous response through a straight line plus random noise.

Definition

Simple Linear Regression Model

Given paired observations , the simple linear regression (SLR) model expresses the response as:

where:

The fitted regression line obtained from sample estimates is:

and the -th residual is .

For valid statistical inference and optimal estimator properties, the classical linear regression framework assumes:

  1. Linearity in parameters: .
  2. Zero-mean errors: .
  3. Homoscedasticity (constant variance): for all .
  4. Independence: for all .
  5. Normality (for MLE and exact / tests): .

Least Squares Estimator (LSE)

The Least Squares Estimator (LSE)—also called Ordinary Least Squares (OLS)—chooses to minimize the sum of squared vertical deviations between observed responses and predicted responses .

Theorem

Least Squares Estimators for Simple Linear Regression

Let the Sum of Squared Errors (SSE) (or Residual Sum of Squares, RSS) objective function be:

Setting the partial derivatives and produces the normal equations:

Solving this linear system yields the unique closed-form OLS estimators (provided ):

where , , and in computational form:

To summarize how much variance in is explained by , we compute the coefficient of determination ():

where is the Total Sum of Squares. Values of close to indicate that most of the sample variation in is captured by the linear relationship with .

Maximum Likelihood Estimation (MLE) & Equivalence with LSE

While LSE relies purely on geometric distance minimization without assuming a probability distribution for the errors, Maximum Likelihood Estimation (MLE) adopts a probabilistic model and finds the parameter values that make the observed sample most probable.

Definition

Maximum Likelihood Estimation (MLE)

Suppose independent observations are drawn from a probability mass or density function parameterized by . The likelihood function is the joint probability (or density) of the observed sample treated as a function of :

The Maximum Likelihood Estimator chooses to maximize , or equivalently to maximize the log-likelihood :

Theorem

MLE for Linear Regression with Normal Errors

Assume the simple linear regression errors are independent and identically normally distributed:

The likelihood and log-likelihood for are:

Maximizing with respect to is equivalent to minimizing , so the MLE slope and intercept coincide exactly with LSE:

Setting gives the MLE error variance:

(whereas the classical unbiased OLS variance estimator divides by degrees of freedom: ).

AspectLeast Squares Estimation (LSE)Maximum Likelihood Estimation (MLE)
ObjectiveMinimizes the sum of squared residuals .Maximizes the likelihood of the observed data.
Distributional AssumptionNo strict parametric distribution required for point estimation.Requires an explicit probability distribution (e.g., Normal, Binomial, Poisson).
Noise Variance Not required to compute ; typically estimated afterward as .Estimated jointly as part of the likelihood: .
FlexibilityPrimarily suited to linear-in-parameter models with continuous responses.General framework extending to logistic regression, count models, and non-Gaussian noise.
Equivalence CaseSpecial case of MLE when errors are i.i.d. .Reduces to LSE for under i.i.d. Gaussian errors.

Worked Examples

Example 1

Fitting a Simple Linear Regression Line and Predicting Performance

An instructor records weekly study hours () and final exam scores () for students:

Student Study Hours Exam Score
1252
2460
3671
4879
51088

Fit the simple linear regression model using Least Squares, predict the score for a student who studies hours per week, and compute the coefficient of determination .

Compute sample means and :

Compute deviations and sums of squares (, , ):

252
460
671
879
1088
Sum00

Evaluate OLS slope and intercept :

Hence the fitted regression line is:

Predict score at and evaluate SSE and : At :

Computing fitted values and residuals :

  • : ,
  • : ,
  • : ,
  • : ,
  • : ,
Answer:

The least squares regression equation is . For study hours, the predicted exam score is , and the model explains of the sample variance in scores ().

Example 2

Computational Formula & MLE Error Variance vs. Unbiased Variance

A systems engineer measures API response latency (in ms) at server load levels (in hundreds of requests per second):

Load 1234
Latency 14192732

Using the raw-sum computational formulas, find the MLE estimates , , and under Gaussian errors , and compare with the unbiased OLS variance .

Compute raw sums and sample means:

Compute , , and MLE regression coefficients:

Under i.i.d. Gaussian errors, the MLE slope and intercept equal the OLS estimators:

Thus .

Compute SSE, , and unbiased : Fitted values and squared residuals:

  • :
  • :
  • :
  • :

Comparing the two variance estimators:

Because MLE divides by instead of , it slightly underestimates the population error variance in small samples.

Answer:

, (), , , and the unbiased OLS error variance is .

Example 3

Deriving an MLE Estimator Directly from the Log-Likelihood

A physical sensor obeys a zero-intercept linear calibration model , where . Given the calibration pairs , derive the MLE formula for directly from the log-likelihood, then compute and .

Write the log-likelihood : Since , the log-likelihood is:

Differentiate with respect to and solve:

Evaluate and numerically:

Residuals and SSE:

Answer:

(model ), and .

Exercises

Exercises on Empirical Models, Simple Linear Regression, LSE & MLE

Question 1

Define an empirical model and explain how it differs from a theoretical model across basis, purpose, flexibility, and generalization. Give one concrete example of each. [Assignment-5, Q1]

Defining empirical and theoretical models:

  • An empirical model is built primarily from observed data using statistical or machine learning estimation to identify relationships, correlations, and predictive patterns without relying on underlying first-principles laws.
  • A theoretical model is derived from established domain theories, scientific axioms, and structural assumptions (for example, conservation laws or circuit equations).

Comparing across the four dimensions:

  1. Basis: Empirical models rely on measured sample data; theoretical models rely on foundational theory and explicit assumptions.
  2. Purpose: Empirical models prioritize quantitative description and prediction; theoretical models prioritize causal explanation and mechanistic understanding.
  3. Flexibility: Empirical models adapt closely to observed patterns in data; theoretical models are tightly constrained by theoretical structure.
  4. Generalization: Empirical models are generally valid only within the range of the studied data (extrapolation is hazardous), whereas theoretical models generalize broadly wherever their assumptions hold.

Concrete examples:

  • Empirical model: A simple linear regression equation fitted from historical monthly records to predict retail revenue from advertising spend.
  • Theoretical model: Ohm's law or the ideal gas law derived from physics.
Answer:

An empirical model learns predictive relationships directly from observed data, whereas a theoretical model is deduced from established scientific laws and assumptions. They differ in basis (data vs. theory), purpose (prediction vs. explanation), flexibility (data-adaptive vs. theory-constrained), and generalization (bounded by observed data range vs. broad applicability when assumptions hold).

Question 2

Explain the concept of the Least Squares Estimator (LSE) for simple linear regression , and derive the closed-form estimators and mathematically. [Assignment-5, Q3]

Objective of LSE: The Least Squares Estimator selects the intercept and slope that minimize the Sum of Squared Errors (SSE) between observed values and predicted values :

First-order optimality conditions (normal equations): Taking partial derivatives of with respect to and and setting both to zero:

Solving for and : Dividing the first equation by gives:

Substituting into the second normal equation:

Using the identities and :

Answer:

The Least Squares Estimator minimizes , giving:

Question 3

Given the following dataset of observations:

12345
23546

Find the simple linear regression equation using the least squares estimator, and compute the exact Sum of Squared Errors (SSE) and . [Assignment-5, Q4]

Compute sample means:

Compute , , and :

Calculate exact slope and intercept :

Thus, the fitted regression equation is:

Compute SSE and : Evaluating at each point:

  • : ,
  • : ,
  • : ,
  • : ,
  • : ,
Answer:

, , giving the exact regression line , with and .

Question 4

Define Maximum Likelihood Estimation (MLE) and show how it relates to simple linear regression. Compare the assumptions underlying LSE and MLE. [Assignment-5, Q5]

Definition of MLE: Given i.i.d. observations modeled by a density or probability function , the likelihood function is:

MLE chooses the parameter estimate that maximizes the probability (or density) of observing the sample data.

Connection to linear regression: Assume the simple linear regression model with , so . The log-likelihood is:

Maximizing over requires minimizing . Therefore, LSE is a special case of MLE when regression errors are normally distributed with constant variance.

Comparing assumptions of LSE and MLE:

  • LSE: Minimizes squared residuals directly. Requires no explicit distributional family (such as normality) to compute point estimates , and estimates variance separately via .
  • MLE: Requires specifying a full probability distribution for (e.g., Gaussian for linear regression, Bernoulli for logistic regression). Estimates all parameters including jointly, and generalizes naturally to non-Gaussian models.
Answer:

MLE estimates parameters by maximizing the sample likelihood . Under i.i.d. normal regression errors , maximizing the log-likelihood minimizes , making LSE identical to MLE for while MLE requires a specified probability distribution and LSE does not.

Question 5

An agronomist tests the effect of nitrogen fertilizer (kg/hectare) on wheat yield (quintals/hectare) across plots:

Fertilizer 102030405060
Yield 253446546572
  1. Fit the OLS regression line .
  2. Predict the wheat yield when of fertilizer is applied.
  3. Compute the residual at .

Compute means and :

Compute and : Deviations are :

Compute OLS slope and intercept :

Thus .

Predict at and compute residual at :

At :

Answer:
  1. (exact intercept ).
  2. Predicted yield at is .
  3. Residual at is .
Question 6

For the observations below, assume with :

23456
5891216

Compute the MLE estimates , , and , as well as the unbiased OLS error variance .

Compute means, , and :

Compute and :

So .

Compute SSE, , and : Residuals:

  • :
  • :
  • :
  • :
  • :
Answer:

, , , and .

Question 7

How does simple linear regression support predictive analytics in data science, and why should analysts be cautious when using a fitted empirical regression line for extrapolation?

Role of simple linear regression in predictive analytics:

  1. Point Forecasting: Substituting unseen predictor values into produces quantitative forecasts for future outcomes.
  2. Marginal Impact Quantification: The slope quantifies the direction and exact rate of change in per unit increase in , guiding sensitivity and "what-if" scenario analysis.
  3. Uncertainty & Fit Assessment: Metrics such as and residual variance quantify how reliably the predictor explains target variation.

Why extrapolation is hazardous: An empirical regression line is calibrated solely on observed pairs within . Outside that domain, physical saturation, diminishing returns, threshold effects, or structural regime shifts can make the linear relationship invalid, causing prediction errors to grow rapidly.

Answer:

Simple linear regression supports predictive analytics through quantitative forecasting (), marginal sensitivity quantification (), and fit diagnostics (, ). Analysts must avoid extrapolation beyond the observed predictor range because empirical models do not guarantee that linearity persists outside the sampled regime.

Past exam questions for Foundation of Data Science

← Feature EngineeringMultiple & Polynomial Regression, Categorical Regressors & Model Building →