PadhaikoAll subjects

Correlation and Regression

Probability and Statistics (ENSH 304)

Open in the Padhaiko app

4.1 Correlation analysis and test of linear correlation

Correlation analysis measures the direction and strength of the statistical relationship between two quantitative variables and .

4.1.1 Karl Pearson's Correlation Coefficient ()

Definition

Sample Covariance and Pearson Correlation Coefficient

Given paired observations with sample means , define the sums of squares and cross-products:

Properties of :

  1. Boundedness: . denotes a perfect positive linear relation, a perfect negative linear relation, and no linear correlation (though a strong nonlinear curve like can still have ).
  2. Scale and Shift Invariance: is dimensionless and unchanged by adding constants or multiplying and by positive constants.
  3. Symmetry: .
  4. Correlation Causation: A high may reflect a lurking confounding variable rather than direct cause-and-effect.

4.1.2 Spearman's Rank Correlation and Test of Linear Correlation

Theorem

Student's t-Test for Population Correlation (H0: rho = 0)

Assuming follow a bivariate normal distribution, to test whether a true linear correlation exists in the population ( vs. , or one-tailed):

Reject at level if (two-tailed).

Example 4.1

4.1

Five temperature-versus-power measurements are recorded on an embedded processor:

Ambient Temp ()
Leakage Power ()
  1. Compute Karl Pearson's correlation coefficient .
  2. Test at whether there is a significant linear correlation ().

Compute means and centered sums ():

Sum—

Compute :

Test vs. ():

Since , reject .

Answer:
  1. (very strong positive linear correlation); 2. : reject , confirming significant linear correlation.

4.2 Simple regression analysis, the concept of explained, unexplained, and total

While correlation measures the strength of a linear relationship, regression analysis builds a mathematical equation to predict a dependent (response) variable from one or more independent (predictor) variables .

4.2.1 Ordinary Least Squares (OLS) Simple Linear Regression

Definition

Simple Linear Regression Model and OLS Estimators

The simple linear regression model is:

where is the true -intercept, is the true slope, and is the random error term.

Using Ordinary Least Squares (OLS), the fitted line minimizes the sum of squared residuals . Setting and yields the normal equations:

Solving gives the OLS regression line of on :

(Similarly, the regression line of on has slope . Both lines intersect at , and .)

4.2.2 Concept of Explained, Unexplained, and Total Variation

Every observation's deviation from the sample mean decomposes orthogonally into two parts:

Theorem

ANOVA Variation Decomposition and Coefficient of Determination

Squaring and summing over all observations (the cross-term vanishes because ) gives the fundamental identity:

where:

  1. Coefficient of Determination (): The proportion of total variation in explained by :
  1. Standard Error of Estimate (): Estimates the residual standard deviation :
Example 4.2

4.2

For concrete curing tests relating curing time (days) to compressive strength ():

  1. Fit the OLS regression line .
  2. Compute Total (), Explained (), and Unexplained () variation, Coefficient of Determination (), and Standard Error of Estimate ().

Compute means and sums of squares:

  • .
  • .
  • .

Fit the regression line of on :

Decompose variation () and compute :

  • Explained variation: .
  • Unexplained variation: Checking residuals gives:
  • Coefficient of determination: ( of variation in strength is explained by curing time).
  • Standard error of estimate: .
Answer:
  1. ; 2. , , , (), .

4.3 Multiple regression analysis

In practice, an engineering response often depends on two or more predictors simultaneously (e.g., server latency as a function of CPU load and concurrent users).

4.3.1 Multiple Linear Regression Model and Normal Equations

Definition

Multiple Linear Regression Model (k Predictors)

Here () is a partial regression coefficient: it measures the expected change in for a one-unit increase in while holding all other predictors constant.

For two predictors (), the fitted plane minimizes , yielding the three normal equations:

(In matrix notation , the OLS estimator is .)

4.3.2 Fit Assessment: , Adjusted , and Overall -Test

Adding any extra variable to a model never decreases , even if the variable is pure noise. Therefore, multiple regression relies on Adjusted and an overall ANOVA -test:

Theorem

Multiple R-squared, Adjusted R-squared, and Overall F-Test

For observations and predictors ():

  1. Multiple Coefficient of Determination: .
  2. Adjusted : Penalizes model complexity using error degrees of freedom :
  1. Standard Error of Estimate: .
  2. Overall Significance -Test ( vs. : at least one ):
Example 4.3

4.3

Six trials measure heat dissipation () as a function of fan speed and fin count :

Trial

Fit using the normal equations, and compute , , and the overall -statistic.

Compute the required sums (, ):

  • , , .
  • Centered sums:

Solve the centered normal equations:

Since , the two equations decouple cleanly:

Thus the fitted regression plane is .

Compute , , , , , and :

Predicted values give residuals :

Answer:

; , , and .


Practice Exercises — Correlation & Regression Analysis

Question 1

Explain the meaning of Explained (), Unexplained (), and Total () variation in simple linear regression, and distinguish between and Adjusted in multiple regression. Why is correlation not equivalent to causation?

Variation Decomposition ():

  • Total Variation (): Overall spread of the response variable around its mean.
  • Explained Variation (): Portion of spread captured by the fitted linear regression line.
  • Unexplained Variation (): Residual spread caused by random noise or unmodeled factors.

vs. Adjusted , and Correlation vs. Causation:

  • always increases (or stays flat) when any predictor is added, even noise; Adjusted divides and by their degrees of freedom ( and ), so it drops if a useless variable is added.
  • A high correlation measures linear co-variation, which may result from a third confounding variable or coincidence rather than a causal mechanism.
Answer:

splits total response variation into regression-explained () and residual-unexplained () components. Adjusted penalizes unnecessary predictors via degrees of freedom, unlike . High correlation shows linear association, not necessarily causation.

Question 2
  1. (Spearman Rank Correlation) Two engineers independently rank software prototypes with rank differences . Compute Spearman's rank correlation .
  2. (Test of Linear Correlation) In another study of circuit runs, Pearson's correlation is . Test against at ().

Part 1 — Spearman's ():

Part 2 — Student's -test for (, ):

Since , reject .

Answer:
  1. ; 2. : reject , confirming significant linear correlation.
Question 3

For the data pairs : , find the OLS regression line , decompose , and compute .

Compute means and sums (, ):

  • .
  • .
  • .

Compute regression coefficients, variation breakdown, and :

Answer:

; , , ; ().

Question 4

In a bivariate engineering dataset, the two regression lines are and .

  1. Find the sample means and .
  2. Identify which line is on and which is on , and find the correlation coefficient .
  3. If the standard deviation of is , find .

Find as the intersection of the two lines: Multiply the first equation by : . Subtracting :

Substituting into .

Identify the lines using : Suppose is on (, so ) and is on (, so ). Then:

(The reverse assignment gives , impossible.) Since both slopes are positive, .

Find from :

Answer:
  1. , ; 2. is on , is on , ; 3. .
Question 5

A multiple linear regression model with predictors is fitted to observations, yielding and . Compute , , Adjusted , and the overall -statistic, and test overall model significance at ().

Compute , , and :

Compute overall -statistic ():

Since , reject .

Answer:

, , , and (reject ; the overall model is highly significant).

← Statistical Inference (part 2: 3.4–3.5)Statistical Quality Control →