Linear Algebra, Matrix Factorization & Gradient Descent
Open in the Padhaiko app2.1 Introduction to Linear Algebra for Data Science
Modern data science operates on multi-dimensional, structured, and unstructured observations. At the mathematical heart of nearly all data modeling, feature transformations, optimization algorithms, and predictive systems lies linear algebra—the mathematical discipline concerned with vector spaces, linear mappings, and systems of linear equations.
In computational data science, data is rarely handled as isolated scalar quantities. Rather, collections of observations and their attributes are naturally organized into geometric vectors, rectangular matrices, and multi-dimensional tensors. Linear algebra provides the algebraic language and computational mechanics required to manipulate, rotate, project, compress, and transform these high-dimensional spaces efficiently.
Linear Algebra and Vector Spaces in Data Science
Linear Algebra is the mathematical branch studying vectors, vector spaces (linear spaces), systems of linear equations, and the linear transformations that map between them.
In data science, the foundational computational structures include:
- Scalars (): Single numerical quantities representing a magnitude or continuous measurement (e.g., learning rate , temperature, loss value).
- Vectors (): Ordered -tuples of real numbers representing a single data observation across features, or a direction and magnitude in an -dimensional Euclidean space.
- Matrices (): Two-dimensional arrays of numbers arranged in rows and columns. Matrices represent entire structured datasets, systems of linear constraints, or linear operators.
- Tensors (): Multi-dimensional arrays generalizing matrices to higher orders (e.g., an RGB image tensor of dimension , or a batch of sequence embeddings in deep learning).
| Mathematical Entity | Order / Tensor Rank | Geometric Interpretation | Data Science Manifestation | Concrete Notation Example |
|---|---|---|---|---|
| Scalar | A single zero-dimensional magnitude | Model hyperparameters, individual loss, scalar weights | ||
| Vector | A point or directed line segment in | Single observation feature vector, parameter vector | ||
| Matrix | A linear operator or 2D grid transformation | Design matrix ( samples features), weight matrix | ||
| Tensor | Multi-way spatial/temporal coordinate field | Color images (), video volumes, token embeddings |
Importance and Real-World Applications in Data Science
Every stage of the end-to-end data science lifecycle relies on linear algebraic operations:
- Tabular Data Representation (The Design Matrix): In supervised and unsupervised learning, tabular datasets are represented as an design matrix :
- Each of the rows represents an individual sample or observation (e.g., a patient, a customer, a transaction).
- Each of the columns represents an attribute or feature (e.g., age, income, blood pressure).
- Linear Regression and Normal Equations: Multiple linear regression models the dependent response vector as a linear combination of input features . Solving for optimal parameters that minimize the residual sum of squares produces the closed-form normal equations:
- Dimensionality Reduction and Principal Component Analysis (PCA): High-dimensional datasets frequently suffer from the curse of dimensionality and collinearity. PCA identifies orthogonal axes of maximum variance by computing the eigenvectors and eigenvalues of the data covariance matrix .
- Deep Learning and Artificial Neural Networks: Every dense neural network layer computes an affine linear transformation followed by an element-wise non-linear activation function :
where is the weight matrix, is the input representation, and is the bias vector.
- Optimization and Loss Minimization: Gradient descent, Newton-Raphson, and quasi-Newton optimization methods navigate multidimensional loss surfaces using gradient vectors and second-derivative Hessian matrices .
- Image and Signal Processing: Grayscale digital images are encoded as matrices of normalized pixel brightness values in or . Color digital images are tensors across Red, Green, and Blue channels. Geometric image manipulations—such as rotations, zooms, translations, and perspective corrections—are computed by multiplying pixel coordinates by homogeneous transformation matrices.
- Computational Performance, BLAS, and GPU Vectorization: Modern machine learning libraries (NumPy, PyTorch, TensorFlow) achieve extreme throughput because linear algebra operations map directly to hardware-accelerated Basic Linear Algebra Subprograms (BLAS) and matrix processing engines (NVIDIA CUDA Tensor Cores). Vectorized operations compute millions of multiply-accumulate steps in parallel, avoiding high-overhead scalar loops.
Systems of Linear Equations
A system of linear equations in unknown variables is expressed algebraically as:
Using matrix algebra, this entire system compresses into the compact matrix equation:
where:
- is the coefficient matrix:
- is the column vector of unknowns:
- is the column vector of constant terms:
Depending on the rank of matrix and augmented matrix , the system exhibits:
- A unique solution: When (system is consistent with full column rank; for square matrices, and ).
- Infinitely many solutions: When (under-determined system with free variables).
- No solution: When (over-determined inconsistent system; in data science, we approximate the best pseudo-solution via least-squares: ).
Linear Transformations
Linear Transformation
A mapping is a linear transformation if and only if it preserves vector addition and scalar multiplication for all vectors and all scalars :
- Additivity (Superposition):
- Homogeneity (Scalar Multiplication):
Combining these two conditions yields the fundamental linearity condition:
Every linear transformation can be represented uniquely as matrix multiplication by an standard matrix :
where the columns of are the images of the standard basis vectors :
| Aspect | Geometric Vector Space | Data Feature Space |
|---|---|---|
| Dimension | 2 or 3 spatial dimensions | Hundreds or thousands of feature dimensions |
| Coordinates | Spatial axes () | Numerical attributes (age, salary, sensor voltage, word counts) |
| Linear Mapping | Geometric rotation, reflection, scaling, shearing | Feature projection, latent factor encoding, dimensionality reduction |
| Distance Meaning | Physical Euclidean distance | Metric dissimilarity between observations or customers |
| Origin | Spatial reference point | Mean-centered feature baseline (all features zeroed) |
2.2 Vectors, Matrices, and Matrix Factorization
Vectors in Data Science
A vector represents a single point or directed segment in an -dimensional space:
Fundamental Vector Operations
Let and be vectors in , and :
- Vector Addition:
- Scalar Multiplication:
- Dot Product (Inner Product):
The dot product returns a scalar. Two non-zero vectors are orthogonal if and only if .
Vector Norms and Distances
Norms measure the length or magnitude of vectors:
- Norm (Manhattan / Taxicab Norm):
Used in Lasso regression ( regularization) to enforce feature sparsity by driving irrelevant coefficients to exact zeros.
- Norm (Euclidean Norm):
Used in Ridge regression ( weight decay) to prevent weights from exploding.
- Norm (Chebyshev / Maximum Norm):
Cosine Similarity
The geometric definition of the dot product links the algebraic sum to the angle between two vectors:
Solving for defines the Cosine Similarity:
In Natural Language Processing (NLP) and recommender systems, documents and user preferences are encoded as high-dimensional term-frequency or embedding vectors. Cosine similarity evaluates orientation rather than magnitude: two documents discussing identical topics with different word lengths produce collinear vectors ().
Matrices in Data Science
A matrix is a rectangular array:
Taxonomy of Matrices
- Square Matrix: Number of rows equals columns ().
- Row Matrix: A single horizontal row ().
- Column Matrix: A single vertical column ().
- Diagonal Matrix: Square matrix where all off-diagonal entries are zero ( for ).
- Identity Matrix (): Diagonal matrix with diagonal entries equal to ().
- Zero Matrix (): All entries are zero.
- Symmetric Matrix: Square matrix equal to its own transpose (, so ).
- Orthogonal Matrix (): Square matrix whose columns and rows are orthonormal: , implying and .
Matrix Operations
- Matrix Addition: Commutative and associative for matrices of identical dimension :
- Matrix Multiplication: Matrix multiplies to yield :
Crucial rule: Inner dimensions must match (). Matrix multiplication is associative () and distributive, but not commutative ( in general).
- Matrix Transpose: Reflects matrix entries over the main diagonal:
- Determinant and Inverse: For an square matrix, the determinant quantifies the volumetric scaling factor of the transformation. A matrix is invertible (non-singular) if and only if . Its inverse satisfies:
Eigenvalues and Eigenvectors
For a square matrix , a non-zero vector is an eigenvector with corresponding eigenvalue if:
Non-trivial solutions exist if and only if the characteristic determinant vanishes:
Eigenvalues and eigenvectors identify the invariant axes of linear operators, serving as the foundational engine for Principal Component Analysis (PCA) and Spectral Graph Clustering.
Matrix Factorization
Matrix Factorization
Matrix Factorization (or matrix decomposition) is the process of decomposing a large, potentially incomplete data matrix into a product of two or more lower-rank matrices:
where:
- represents the row entities across latent dimensions (e.g., user-feature matrix).
- represents the column entities across latent dimensions (e.g., item-feature matrix).
- is the latent dimensionality (the rank of the factorization).
Why Factorize Matrices in Data Science?
- Dimensionality Reduction: Compresses massive feature spaces into compact latent dimensions while discarding high-frequency noise.
- Latent Feature Discovery: Discovers underlying unobserved patterns (e.g., identifying film themes like "romance", "action", "sci-fi" from user rating records).
- Handling Missing Values & Collaborative Filtering: Real-world interaction matrices (such as customer ratings on Netflix or Amazon) are over sparse. Factorization fits parameters on known entries to predict missing ratings .
| Factorization Technique | Mathematical Formulation | Primary Constraint | Major Use Case |
|---|---|---|---|
| Singular Value Decomposition (SVD) | orthogonal; diagonal | PCA, Latent Semantic Analysis (LSA), noise filtering | |
| Non-negative Matrix Factorization (NMF) | Topic modeling, image part decomposition, gene expression | ||
| Alternating Least Squares (ALS) | Alternating quadratic convex optimization | Large-scale recommender systems with explicit/implicit feedback | |
| Stochastic Gradient Descent (SGD) | Iterative loss minimization over observed ratings | Netflix Prize collaborative filtering |
Singular Value Decomposition (SVD)
Singular Value Decomposition (SVD) Theorem
Any real rectangular matrix of rank can be factored into the product of three matrices:
where:
- is an orthogonal matrix whose columns are the left singular vectors of (the orthonormal eigenvectors of ).
- is a diagonal matrix containing the non-negative singular values arranged in descending order along its main diagonal:
- is an orthogonal matrix whose columns are the right singular vectors of (the orthonormal eigenvectors of ). is its transpose.
Algorithmic Procedure for Computing SVD
To compute the full SVD of an matrix :
- Form the symmetric matrix : Solve the characteristic equation for eigenvalues .
- Compute singular values :
Construct the diagonal matrix with diagonal entries .
- Find right singular vectors : For each eigenvalue , solve , normalize to unit Euclidean length , and assemble the orthogonal matrix .
- Compute left singular vectors : For each non-zero singular value :
This ensures directional alignment (). If , complete the remaining columns of using Gram-Schmidt orthogonalization on the nullspace of .
- Verify the decomposition: Check that .
| Aspect | Singular Value Decomposition (SVD) | Non-Negative Matrix Factorization (NMF) |
|---|---|---|
| Factor Constraints | Orthonormal singular vectors () | Non-negativity () |
| Interpretability | Factors have mixed positive and negative signs (hard to explain physically) | Purely additive parts-based representation (readily interpretable) |
| Uniqueness | Mathematically unique up to column sign flips | Non-convex; local minima depending on initialization |
| Data Requirements | Operates on arbitrary real matrices | Input matrix entries must be strictly non-negative () |
2.3 Gradient Descent for Optimization
In machine learning and data science, learning is formulated as an optimization problem: finding the model parameter vector that minimizes a real-valued empirical cost or loss function :
Except for simple models like ordinary least squares regression, real-world loss functions have no analytical closed-form solution. We must instead use iterative numerical optimization algorithms, among which Gradient Descent is the primary engine.
Gradient Descent Optimization
Gradient Descent is a first-order iterative optimization algorithm used to find a local minimum of a differentiable objective function by taking successive steps proportional to the negative of the gradient of the function at the current point.
Geometric Intuition and the Gradient Vector
Consider an objective function defined over parameter vector .
The gradient vector is the vector of all first-order partial derivatives:
From multivariable calculus, the directional derivative of in unit direction is:
where is the angle between and :
- is maximized when (): the gradient points in the direction of steepest ascent.
- is minimized when (): the negative gradient points in the direction of steepest descent.
Physical Analogy: Imagine standing blindfolded on a foggy, mountainous terrain. To reach the bottom of the valley, you feel the slope of the ground beneath your feet and take a step in the steepest downhill direction. Repeating this process brings you to the valley floor.
The Parameter Update Rule
Starting from an initial parameter guess (initialized randomly or with heuristics), the parameter vector is updated iteratively:
where:
- is the current parameter vector at iteration .
- is the gradient evaluated at .
- The minus sign () ensures movement in the downhill direction (decreasing loss).
- is the learning rate (or step size hyperparameter), controlling how far the parameters move along the negative gradient.
The iteration terminates when a stopping condition is reached:
- The gradient norm falls below a threshold: .
- The decrease in loss between consecutive iterations is negligible: .
- A pre-set maximum number of iterations (epochs) is completed.
Impact of the Learning Rate ()
The learning rate is one of the most critical hyperparameters in model training:
- Learning Rate Too Small (): The parameter updates are minuscule. Training requires an excessive number of iterations, leading to high computational costs and the risk of stalling on flat plateaus or saddle points.
- Learning Rate Optimal (): The parameters descend smoothly and rapidly along the error surface, converging reliably to the minimum.
- Learning Rate Too Large (): The algorithm overshoots the minimum. Successive updates oscillate wildly across the valley walls and may diverge, driving the loss to numerical infinity ().
Variants of Gradient Descent
In practical machine learning, the cost function is an average over training examples:
Depending on how many samples are evaluated before performing each parameter update, gradient descent is classified into three variants:
| Aspect | Batch Gradient Descent | Stochastic Gradient Descent (SGD) |
|---|---|---|
| Data per Update | The entire training dataset ( observations) | Exactly one randomly chosen sample |
| Update Frequency | Once per epoch (after scanning all samples) | times per epoch (after every individual sample) |
| Trajectory | Smooth, deterministic, and direct | Noisy, erratic, and fluctuating |
| Memory Footprint | High; requires entire dataset in memory () | Minimal; processes one sample at a time () |
| Local Minima | Can get trapped in shallow local minima | Gradient noise helps jump out of bad local minima |
1. Batch Gradient Descent (BGD)
Uses all training instances to compute the exact gradient:
- Advantages: Guaranteed stable and monotonic convergence toward the minimum for convex loss functions.
- Disadvantages: Computationally prohibitive for large datasets (); cannot perform online updates with streaming data.
2. Stochastic Gradient Descent (SGD)
Picks a single sample uniformly at random and updates parameters immediately:
- Advantages: Ultra-fast updates; handles continuous data streams; gradient variance helps escape shallow local minima and saddle points.
- Disadvantages: High variance causes the parameters to oscillate around the minimum without settling into it, requiring learning rate decay schedules.
3. Mini-Batch Gradient Descent
The industry standard in deep learning and modern data science. It splits the training dataset into small subsets (mini-batches) of size (typically samples):
- Advantages: Balances the smooth convergence of BGD with the computational efficiency of SGD. Efficiently utilizes GPU SIMD vectorization and tensor hardware.
| Variant | Batch Size () | Gradient Variance | Epoch Update Count | Computational Efficiency | Hardware Acceleration |
|---|---|---|---|---|---|
| Batch GD | Zero (deterministic) | Low for large | Moderate (large matrix limits) | ||
| Stochastic GD | Extremely high | Low vectorization | Poor (scalar overhead) | ||
| Mini-Batch GD | Low to moderate | Optimal | Optimal (maximizes GPU throughput) |
Worked Examples
2.1
In an artificial neural network layer, an input feature vector is transformed by a weight matrix . Compute the linear transformation , and explain the geometric effect of this transformation.
Evaluating the matrix-vector product: The transformed vector is computed by multiplying the transformation matrix by the column vector :
Geometric interpretation: The matrix is a uniform isotropic scaling operator. It stretches every vector in by a factor of radially outward from the origin without altering its orientation:
The length doubles while the direction remains identical.
The transformation scales the input vector by a factor of along all coordinate axes, doubling its magnitude while preserving its angular orientation.
2.2
A movie recommender system represents customer preferences and movie attributes across four genres (Comedy, Romance, Action, Sci-Fi) as vectors:
Calculate:
- The inner (dot) product .
- The Euclidean norms and .
- The cosine similarity between the user profile and the candidate movie .
Computing the dot product: Multiply corresponding components and sum:
Computing the Euclidean norms: Calculate the norm for each vector:
Calculating cosine similarity: Divide the dot product by the product of the norms:
The high cosine similarity () indicates strong alignment between the user's preferences and the movie's attributes.
2.3
Compute the parameter updates for the quadratic objective function using Gradient Descent. Assume:
- Initial parameter value:
- Learning rate:
- Perform iterations up to .
Analytical derivative of the objective function: Differentiating with respect to :
The theoretical global minimum occurs where .
First parameter update (): Evaluate gradient at initial position :
Update parameter:
Second parameter update (): Evaluate gradient at :
Update parameter:
Third parameter update (): Evaluate gradient at :
Update parameter:
Fourth parameter update (): Evaluate gradient at :
Update parameter:
The parameter monotonically approaches the global minimizer .
2.4
Compute the Singular Value Decomposition (SVD) of the symmetric matrix:
Find the orthogonal matrices and the diagonal singular value matrix such that .
Computing and characteristic equation: Since is symmetric, :
Set up the characteristic equation:
Singular values and matrix : The singular values are the square roots of the eigenvalues:
Right singular vectors : For :
Normalized unit vector:
For :
Normalized unit vector:
Assembling and :
Left singular vectors : Using the relation :
Assembling :
Exercises
Exercises — Linear Algebra, Matrix Factorization & Gradient Descent
Define linear algebra and explain its importance in data science with concrete examples. Explain what is meant by a linear transformation and illustrate it with an algebraic example.
Definition and scope of linear algebra: Linear algebra is the mathematical foundation of vector spaces, linear mappings, and systems of linear equations. In modern data science, datasets are modeled as matrices where rows represent observations and columns represent feature dimensions.
Key areas of importance in data science:
- Data Representation: An observation with features (e.g., age, income, credit score) is a vector . An entire dataset of observations is a design matrix .
- Machine Learning Algorithms: Linear regression relies on the normal equations ; deep neural networks compute layer-wise affine mappings ; support vector machines optimize hyperplane margins using inner products.
- Dimensionality Reduction: Techniques like Principal Component Analysis (PCA) project data along eigenvectors corresponding to the largest eigenvalues of the covariance matrix.
- Computer Vision: Digital images are stored as pixel matrices (grayscale) or tensors (RGB color). Image operations like filtering, rotation, and resizing are matrix transformations.
- Computational Speed: Linear algebra libraries leverage BLAS and GPU hardware to perform millions of parallel matrix multiplications, enabling scalable training on big data.
Linear transformation definition and properties: A mapping is a linear transformation if it satisfies two axioms:
- Additivity: for all .
- Homogeneity: for all and .
Concrete algebraic example: Consider the transformation matrix acting on vector :
This represents a uniform scaling transformation that doubles the length of any vector in while preserving its direction.
Linear algebra is the study of vectors, matrices, and linear transformations that structure data representation, regression, neural networks, PCA, and GPU acceleration in data science. A linear transformation satisfies and , exemplified by scaling to .
Explain matrix factorization with an example. Explain the working mechanism of Singular Value Decomposition (SVD) and derive the complete SVD factorization for the matrix:
Concept of matrix factorization: Matrix factorization decomposes an matrix into a product of lower-rank factor matrices, typically , where and with . It discovers latent features, removes noise, and enables missing-value prediction in recommender systems.
SVD formulation and mechanism: Singular Value Decomposition factors any real matrix into:
where contains orthonormal eigenvectors of (left singular vectors), contains orthonormal eigenvectors of (right singular vectors), and contains the singular values in descending order along its diagonal.
Computing and its eigenvalues:
Find eigenvalues by solving the characteristic equation:
Singular values and matrix : The singular values are the square roots of the eigenvalues:
Computing and right singular vectors :
For :
Normalized unit vector:
For :
Normalized unit vector:
Assembling and :
Determining left singular vectors :
Assembling :
Reconstruction verification:
where singular values are and .
State one real-world application of matrix factorization in data science and explain its mechanism in detail.
Real-world application: Recommender Systems: One of the most prominent applications of matrix factorization is in Collaborative Filtering Recommender Systems, famously deployed by streaming and e-commerce platforms like Netflix, Spotify, and Amazon.
Problem formulation: Consider an ecosystem with users and items (movies, songs, products). The user interactions form a massive utility matrix , where entry represents the rating given by user to item (e.g., to stars).
Because typical users rate only a tiny fraction of the available catalog, is extremely sparse (often empty entries). The data science goal is to predict the missing ratings and recommend items with the highest predicted scores.
Latent factor model mechanism: Matrix factorization maps both users and items into a shared latent factor space of dimensionality :
where:
- : User-feature matrix where row measures how much user prefers each latent feature (e.g., action, drama, budget, pacing).
- : Item-feature matrix where row measures how strongly item embodies those same latent features.
The predicted rating of user for item is the dot product of their latent representations:
Training via regularized loss minimization: To learn and from the observed ratings , we minimize the regularized squared error:
where is a regularization parameter preventing overfitting on sparse entries. This objective is optimized efficiently using Stochastic Gradient Descent (SGD) or Alternating Least Squares (ALS).
In collaborative filtering recommender systems (e.g., Netflix, Spotify), matrix factorization decomposes a sparse user-item interaction matrix into user latent factor matrix and item latent factor matrix such that predicted ratings are computed as , enabling high-accuracy recommendations.
Define gradient descent and explain its purpose in machine learning. Write down its iterative mathematical formula, identify all components, and describe its geometric intuition.
Definition and purpose: Gradient Descent is a first-order iterative numerical optimization algorithm designed to minimize a differentiable cost or loss function by making parameter adjustments proportional to the negative gradient.
Its core purpose in machine learning is empirical risk minimization: training algorithms cannot solve for optimal weights analytically when loss landscapes are high-dimensional, non-linear, or non-convex. Gradient descent systematically adjusts model parameters to minimize prediction error.
Iterative mathematical formula:
where:
- is the parameter vector at iteration .
- is the updated parameter vector.
- is the gradient vector of partial derivatives evaluated at .
- is the learning rate (step size hyperparameter).
- The minus sign () directs the step opposite to the gradient, descending toward the minimum.
Geometric interpretation: The gradient vector points in the direction of maximum directional derivative (steepest ascent). By taking steps in the opposite direction (), the algorithm traverses the path of steepest descent, moving toward the lowest point (local or global minimum) on the error surface.
Gradient descent minimizes machine learning loss functions by iteratively updating parameters via . The gradient indicates the direction of steepest error increase; subtracting a fraction moves parameters in the direction of steepest descent toward minimum error.
Explain the differences between Batch Gradient Descent, Stochastic Gradient Descent (SGD), and Mini-Batch Gradient Descent. Discuss their respective advantages, disadvantages, and suitability for real-world datasets.
Batch Gradient Descent (BGD): Computes the gradient of the loss function using the entire training dataset of observations at each step:
- Pros: Stable, deterministic convergence trajectory; guarantees finding the global minimum for convex functions.
- Cons: Very slow per iteration; memory-intensive (); fails when datasets exceed RAM capacity.
Stochastic Gradient Descent (SGD): Updates parameters after evaluating only a single, randomly chosen training instance :
- Pros: Ultra-fast updates; minimal memory footprint; online learning capability; stochastic fluctuations help jump out of shallow local minima.
- Cons: High gradient variance leads to noisy oscillations, preventing smooth settling without a carefully decaying learning rate schedule.
Mini-Batch Gradient Descent: Partitions the dataset into small batches of size (typically ) and updates parameters per mini-batch:
- Pros: Combines the stability of BGD with the efficiency of SGD; maps directly onto vectorized GPU matrix operations (CUDA Tensor Cores).
- Cons: Introduces batch size as an additional hyperparameter to tune.
Comparative summary table:
| Feature | Batch Gradient Descent | Stochastic Gradient Descent (SGD) | Mini-Batch Gradient Descent |
|---|---|---|---|
| Samples per step | All samples | sample | samples (–) |
| Convergence path | Direct, smooth | Highly erratic, noisy | Moderately smooth |
| Computational cost | High per step () | Very low per step () | Well-balanced () |
| Hardware acceleration | Suboptimal for massive | Poor (sequential overhead) | Optimal (fully leverages GPU SIMD) |
Batch GD updates once per epoch using all samples (smooth but memory-heavy); SGD updates after every individual sample (fast but noisy); Mini-Batch GD updates after evaluating samples (–), achieving optimal balance, fast GPU vectorization, and smooth convergence in production machine learning.
Discuss how the choice of learning rate affects the convergence of gradient descent. Describe what happens when the learning rate is too small, optimal, or too large, and explain the concept of learning rate scheduling.
Role of the learning rate (): The learning rate controls the step size taken along the negative gradient vector. It determines the balance between training stability and convergence speed.
Effects of different learning rate regimes:
- Too Small (): The model makes tiny adjustments to parameters. Training progresses at a crawl, requiring thousands of redundant epochs, consuming excessive computational time, and frequently stalling in shallow plateaus or saddle points.
- Optimal (): The parameters descend steadily and stably down the loss landscape, converging efficiently to the neighborhood of the minimum.
- Too Large (): The step size overshoots the minimum point of the cost function. Successive iterations oscillate across opposite canyon walls with increasing amplitude, ultimately causing numerical overflow and catastrophic divergence ().
Learning rate scheduling and adaptive methods: To resolve this trade-off, practitioners use dynamic learning rate strategies:
- Step Decay: Reduce by a factor (e.g., ) every fixed number of epochs.
- Exponential Decay: .
- Adaptive Optimizers (Adam, RMSprop): Compute per-parameter adaptive learning rates based on running averages of first and second moments of the gradients.
An undersized learning rate causes sluggish convergence and high computational expense; an oversized learning rate overshoots the valley floor, causing severe oscillations and divergence. An optimal learning rate ensures fast, steady convergence, often augmented by learning rate decay or adaptive optimizers (Adam) that decrease as the minimum approaches.
In an information retrieval task, two document vectors are represented in a 5-word vocabulary space as:
Compute their dot product, Euclidean norms, and cosine similarity. State whether these documents are considered similar.
Evaluating the dot product: Multiply corresponding term-frequency components:
The two documents share vocabulary terms in common.
Computing the Euclidean norms:
Calculating cosine similarity:
With a cosine similarity of (), the documents exhibit substantial semantic overlap and are considered strongly related.
Perform two iterations of multivariable gradient descent on the objective function:
starting from initial point with learning rate . Calculate the coordinates after two updates.
Deriving the gradient vector: Compute partial derivatives with respect to each coordinate:
The global minimum is at the origin where .
First iteration (): Evaluate gradient at :
Update coordinates:
Value of function: (reduced from ).
Second iteration (): Evaluate gradient at :
Update coordinates:
Value of function: . Notice that coordinates obey and .
After two iterations, the objective value drops from to and then to , moving toward the origin.