Dimensionality Reduction, Linear Factor Model & PCA
Open in the Padhaiko app3.8 Dimensionality Reduction
In modern machine learning applications—such as genomics, natural language processing, computer vision, and customer telemetry—datasets often feature hundreds, thousands, or even millions of measured dimensions (). While collecting a vast array of attributes provides rich descriptive information, operating directly in ultra-high-dimensional spaces introduces severe mathematical, computational, and statistical challenges known collectively as the Curse of Dimensionality.
The Curse of Dimensionality
The Curse of Dimensionality refers to the compounding geometric and probabilistic phenomena that arise when analyzing and modeling data in high-dimensional vector spaces ( for large ):
- Exponential Volume Expansion and Data Sparsity: The volume of a unit hypercube is , but the volume of an inscribed hypersphere of radius contracts to zero as :
Almost all the hypercube volume concentrates in the distant corners. Data points become isolated and sparse; the sample size required to maintain constant sampling density grows exponentially ().
- Distance Metric Concentration: As dimensionality grows, the ratio of the distance to the nearest neighbor () versus the furthest neighbor () approaches unity for arbitrary distance metrics:
In high dimensions, all points become virtually equidistant from one another, rendering distance-based algorithms (-NN, -Means, density estimation) ineffective.
- Overfitting (The Hughes Phenomenon): With fixed training observations , adding features initially increases model predictive power, but beyond an optimal threshold, additional dimensions provide parameters that fit random sample noise, causing test generalization to collapse.
Taxonomy of Dimensionality Reduction
Dimensionality reduction maps data from an original high-dimensional space into a lower-dimensional manifold (where ) while preserving maximal statistical information and underlying structure.
Practitioners divide dimensionality reduction into two primary methodological branches:
| Aspect | Feature Selection | Feature Extraction |
|---|---|---|
| Operational Mechanism | Selects a subset of original features and discards the remaining features | Constructs completely new, synthetic features by combining the original dimensions |
| Interpretability | High: original physical meanings and measurement units are retained | Moderate to Low: new composite variables (e.g., principal components) are linear/non-linear mixtures |
| Information Loss | Complete for discarded attributes: any unique signal in dropped features is lost | Minimized: information from all dimensions is compressed into the top components |
| Canonical Techniques | Filter methods (correlation, ANOVA), Wrapper methods (RFE), Embedded (Lasso ) | Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Autoencoders |
The Linear Factor Model
The Linear Factor Model
A Linear Factor Model is a statistical latent-variable model that represents a -dimensional vector of observed continuous variables as a linear combination of a smaller number of unobserved (latent) common factors (), plus an idiosyncratic error term:
where:
- is the vector of observed attributes with population mean .
- is the factor loading matrix, where entry indicates the sensitivity or weight of observed variable to common factor .
- is the vector of standardized common latent factors, assumed to satisfy:
- is the vector of unique errors (uniquenesses), assumed uncorrelated with and mutually uncorrelated:
Under these assumptions, the covariance matrix of the observed data decomposes into common variance and unique variance:
The primary purpose of the linear factor model is to simplify complex covariance structures, uncover hidden drivers behind observed correlations, and filter out measurement noise.
Principal Component Analysis (PCA)
Principal Component Analysis (PCA)
Principal Component Analysis (PCA) is an unsupervised, non-parametric linear transformation technique that maps a dataset of possibly correlated variables into a set of linearly uncorrelated orthogonal variables called principal components.
The principal components are ordered such that:
- The first principal component () accounts for the maximum possible variance of the dataset along any single direction.
- Each succeeding component () accounts for the maximum possible remaining variance under the mathematical constraint that it is strictly orthogonal (uncorrelated) to all preceding components.
Mathematical Derivation of PCA
Let denote a centered data matrix () with sample covariance matrix:
We seek a unit projection vector () such that the sample variance of the projected coordinates is maximized:
Formulating the Lagrangian with multiplier :
Setting the partial derivative with respect to to zero:
This is the standard eigenvalue equation of linear algebra:
- The optimal projection direction is an eigenvector of the covariance matrix .
- The variance along that projection axis is the corresponding eigenvalue:
To maximize variance, is selected as the eigenvector corresponding to the largest eigenvalue .
Criteria for Selecting the Number of Principal Components ()
- Cumulative Explained Variance Ratio: Retain the smallest number of components such that the cumulative explained variance meets a predetermined threshold (typically , , or ):
- Scree Plot Elbow Criterion: Plot eigenvalues in descending order against component index. Look for the "elbow"—the inflection point where the rate of variance decline sharply flattens into a scree of noise.
- Kaiser-Guttman Rule: When PCA is conducted on standardized correlation matrices ( for all original features), retain only components with eigenvalues (i.e., components that capture more variance than a single original standardized feature).
Principal Component Analysis on Bivariate Dataset
Execute full PCA: center the bivariate observations, construct the covariance matrix, solve for eigenvalues and eigenvectors, and compute the variance explained by the principal axis.
Calculate mean vector : The sample size is .
Compute mean-centered coordinates: Subtract from each observation vector :
Tabulate centered vectors:
| Coordinate | ||||||
|---|---|---|---|---|---|---|
Calculate outer products and covariance matrix: Compute outer product matrices :
Sum the matrices:
Using the population scaling factor :
(Rounded to two decimals: )
Formulate characteristic equation and extract eigenvalues: Solve :
Determine explained variance ratio: Total variance is the trace of the covariance matrix:
Proportion of variance captured by the first principal component:
The second component captures only of the total variance:
Because explains of the information, discarding achieves a reduction in dimensionality with only a negligible loss in total variance.
Derive primary eigenvector and project data: Substitute into :
From the first row:
Normalizing to unit length ():
Primary unit eigenvector:
Project the centered points onto using :
The sample covariance matrix is . The eigenvalues are (or rounded) and . The first principal component accounts for of the total variance with unit eigenvector .
Determining Principal Components from Eigenvalue Spectrum
Calculate the variance explained by each eigenvalue and determine the minimum component cutoff for an retention threshold.
Compute total variance: The total variance is the sum of all eigenvalues:
Compute individual explained variance ratios:
- :
- :
- :
- :
- :
Compute cumulative explained variance ratios:
- Top 1 component:
- Top 2 components:
- Top 3 components:
- Top 4 components:
- All 5 components:
Evaluate retention threshold:
- Retaining components captures .
- Retaining components captures .
Therefore, a minimum of principal components must be retained, achieving a reduction in dimensionality (from dimensions to ) while preserving of total statistical information.
Total variance is . Individual explained variance ratios are , , , , and . To exceed the threshold, principal components must be retained, capturing cumulative variance.
Exercises: Dimensionality Reduction & PCA
Describe how feature scaling affects machine learning model performance. Explain what dimensionality reduction means and why it is important in data science.
How feature scaling affects model performance: Feature scaling standardizes numerical attributes so that no single feature dominates optimization or distance calculations:
- Gradient Descent: Features with disparate scales produce elongated, eccentric loss landscapes. Gradient descent oscillates perpendicularly, slowing convergence. Scaling creates isotropic contours, accelerating learning.
- Distance-based Algorithms: Algorithms such as -NN, -Means, and PCA compute distances. Features with magnitudes in thousands overpower features with magnitudes in single digits, destroying predictive sensitivity.
Definition of dimensionality reduction: Dimensionality reduction is the process of reducing the number of input variables in a dataset by projecting or selecting features, producing a lower-dimensional representation while preserving as much relevant variance as possible.
Why dimensionality reduction is important:
- Mitigating the Curse of Dimensionality: Prevents extreme data sparsity and distance concentration in high-dimensional hypercubes.
- Reducing Overfitting: Prunes redundant and collinear dimensions, reducing parameter count and improving model generalization.
- Computational Efficiency: Decreases memory storage and speeds up training and inference.
- Data Visualization: Enables projection of complex multi-dimensional data into or for exploratory visual inspection.
Feature scaling prevents high-magnitude features from distorting distance metrics and allows gradient descent to converge stably. Dimensionality reduction reduces feature count while preserving essential information, mitigating the curse of dimensionality, decreasing computational overhead, preventing overfitting, and enabling low-dimensional visualization.
Define the Linear Factor Model and explain its purpose and mathematical assumptions in data reduction.
Definition and Mathematical Formulation: The Linear Factor Model expresses a -dimensional vector of observed variables as a linear combination of unobserved common latent factors () plus unique error terms :
where , is the factor loading matrix, is the common factor vector, and is the uniqueness vector.
Core Statistical Assumptions:
- Common factors have zero mean and identity covariance: .
- Unique errors have zero mean and diagonal covariance: .
- Common factors and unique errors are independent: .
Purpose in Data Science:
- Latent Variable Discovery: Identifies underlying constructs that cannot be measured directly (e.g., student intelligence driving multiple subject test scores).
- Covariance Simplification: Decomposes the full covariance matrix into , compressing free parameters into parameters.
- Noise Reduction: Segregates common explanatory variance from sensor-specific noise.
The Linear Factor Model represents observed variables as linear combinations of fewer latent factors plus error: . It assumes latent factors are standard orthogonal variables uncorrelated with diagonal error terms. Its purpose is to uncover underlying drivers, simplify covariance structures, and filter noise.
Describe the main idea behind Principal Component Analysis (PCA). Outline its sequential mathematical algorithm.
Main Idea behind PCA: Principal Component Analysis (PCA) is an unsupervised linear transformation technique that identifies orthogonal directions of maximum variance in high-dimensional data. By projecting data onto the first eigenvectors of the sample covariance matrix, it captures the bulk of the variance in fewer dimensions while eliminating linear correlation among features.
Sequential Algorithmic Pipeline:
- Mean Centering: Compute the sample mean vector and center data: . (Standardize with -score if features have different units).
- Compute Covariance Matrix: Calculate .
- Eigen-decomposition: Solve the characteristic equation to obtain eigenvalues and orthonormal eigenvectors .
- Select Components: Choose the top eigenvectors corresponding to the largest eigenvalues based on scree plots or cumulative variance thresholds ().
- Project Data: Construct projection matrix and compute lower-dimensional coordinates: .
PCA finds orthogonal axes of maximum variance via eigen-decomposition of the centered covariance matrix. Its steps are: center the data, compute the covariance matrix , solve for eigenvalues and eigenvectors, sort by descending eigenvalue magnitude, select the top eigenvectors, and project data into the lower-dimensional space.
Consider three two-dimensional data points lying on a straight line:
- Compute the mean vector and the sample covariance matrix (using ).
- Solve for the eigenvalues and show that the first principal component captures of the total variance while the second eigenvalue is zero.
Calculate mean vector and centered coordinates:
Centered coordinates:
Construct sample covariance matrix: Compute sum of outer products:
Using sample denominator :
Solve characteristic polynomial:
The eigenvalues are:
Total variance is:
Proportion of variance explained by :
Because the original data points are strictly collinear along , the first principal component captures of the information without loss.
The mean is and sample covariance is . The eigenvalues are and . explains exactly of the variance, reflecting the collinearity of the data.
A 4-dimensional sensor array yields the following eigenvalues from PCA eigen-decomposition:
- Calculate the percentage of variance captured by each principal component.
- Determine the cumulative percentage of variance explained by the top two components.
- If an engineer sets an cumulative variance threshold, how many dimensions can be eliminated?
Compute total variance and individual ratios:
- :
- :
- :
- :
Compute cumulative variance for top two components:
Evaluate threshold and dimension elimination: The required threshold is . Since the top two components account for , retaining components is sufficient. The engineer can discard and , eliminating dimensions (a dimensionality reduction).
Individual variance ratios: , , , . The top two components explain cumulative variance. Retaining 2 components satisfies the threshold, eliminating 2 of the 4 dimensions.
Explain why feature standardization (-score scaling) is an indispensable prerequisite prior to performing Principal Component Analysis when features possess different measurement units.
Variance Maximization Objective of PCA: PCA seeks orthogonal axes that maximize sample variance:
The total variance equals the sum of the variances of individual original features:
Impact of Unscaled Measurement Units: Suppose feature is salary measured in cents (variance in the order of ) and feature is employee age in decades (variance in the order of ). The total variance will be overwhelmingly dominated by :
The first eigenvector will align almost parallel to the axis regardless of whether contains critical domain signals.
Correction via Standardization: Applying -score standardization () forces every feature to have unit variance (). The covariance matrix of standardized variables is the correlation matrix :
This places all features on equal statistical footing, ensuring that principal components reflect genuine correlation structures rather than arbitrary unit magnitudes.
PCA is variance-driven; features with large scales (e.g., salary in rupees vs age in years) possess artificially massive variances that dominate the covariance matrix, forcing principal components to align solely with high-magnitude features. Standardizing features to unit variance ensures PCA operates on the correlation matrix, giving equal weight to all attributes.
Contrast Principal Component Analysis (PCA) with Linear Discriminant Analysis (LDA) and Factor Analysis (FA) across objectives, supervision, and mathematical assumptions.
Comparative Evaluation:
- Supervision:
- PCA is unsupervised: it operates purely on feature matrix without knowledge of class labels .
- LDA is supervised: it utilizes categorical class labels to maximize between-class separation.
- Factor Analysis is unsupervised: it models correlations among observed variables through latent factors.
- Objective:
- PCA finds axes that maximize total data variance.
- LDA finds projection axes that maximize between-class variance relative to within-class variance ().
- FA models common variance () while explicitly segregating unique feature variance ().
- Assumptions and Interpretability:
- PCA makes no distributional assumption; principal components are orthogonal linear combinations.
- LDA assumes features within each class follow a multivariate Gaussian distribution with equal covariance matrices.
- FA explicitly assumes underlying latent factors generate the observed variables through a generative linear model.
PCA is an unsupervised technique maximizing total variance along orthogonal axes. LDA is a supervised technique maximizing between-class separability relative to within-class spread. Factor Analysis is an unsupervised latent-variable model that explicitly decomposes covariance into shared common factors and unique errors.