PadhaikoAll subjects

Linear Algebra, Matrix Factorization & Gradient Descent

Foundation of Data Science

Open in the Padhaiko app

2.1 Introduction to Linear Algebra for Data Science

Modern data science operates on multi-dimensional, structured, and unstructured observations. At the mathematical heart of nearly all data modeling, feature transformations, optimization algorithms, and predictive systems lies linear algebra—the mathematical discipline concerned with vector spaces, linear mappings, and systems of linear equations.

In computational data science, data is rarely handled as isolated scalar quantities. Rather, collections of observations and their attributes are naturally organized into geometric vectors, rectangular matrices, and multi-dimensional tensors. Linear algebra provides the algebraic language and computational mechanics required to manipulate, rotate, project, compress, and transform these high-dimensional spaces efficiently.

Definition

Linear Algebra and Vector Spaces in Data Science

Linear Algebra is the mathematical branch studying vectors, vector spaces (linear spaces), systems of linear equations, and the linear transformations that map between them.

In data science, the foundational computational structures include:

  1. Scalars (): Single numerical quantities representing a magnitude or continuous measurement (e.g., learning rate , temperature, loss value).
  2. Vectors (): Ordered -tuples of real numbers representing a single data observation across features, or a direction and magnitude in an -dimensional Euclidean space.
  3. Matrices (): Two-dimensional arrays of numbers arranged in rows and columns. Matrices represent entire structured datasets, systems of linear constraints, or linear operators.
  4. Tensors (): Multi-dimensional arrays generalizing matrices to higher orders (e.g., an RGB image tensor of dimension , or a batch of sequence embeddings in deep learning).
Structural Entities in Linear Algebra: Scalars, Vectors, Matrices, and Tensors
Mathematical EntityOrder / Tensor RankGeometric InterpretationData Science ManifestationConcrete Notation Example
ScalarA single zero-dimensional magnitudeModel hyperparameters, individual loss, scalar weights
VectorA point or directed line segment in Single observation feature vector, parameter vector
MatrixA linear operator or 2D grid transformationDesign matrix ( samples features), weight matrix
TensorMulti-way spatial/temporal coordinate fieldColor images (), video volumes, token embeddings

Importance and Real-World Applications in Data Science

Every stage of the end-to-end data science lifecycle relies on linear algebraic operations:

  1. Tabular Data Representation (The Design Matrix): In supervised and unsupervised learning, tabular datasets are represented as an design matrix :
    • Each of the rows represents an individual sample or observation (e.g., a patient, a customer, a transaction).
    • Each of the columns represents an attribute or feature (e.g., age, income, blood pressure).
  2. Linear Regression and Normal Equations: Multiple linear regression models the dependent response vector as a linear combination of input features . Solving for optimal parameters that minimize the residual sum of squares produces the closed-form normal equations:
  1. Dimensionality Reduction and Principal Component Analysis (PCA): High-dimensional datasets frequently suffer from the curse of dimensionality and collinearity. PCA identifies orthogonal axes of maximum variance by computing the eigenvectors and eigenvalues of the data covariance matrix .
  2. Deep Learning and Artificial Neural Networks: Every dense neural network layer computes an affine linear transformation followed by an element-wise non-linear activation function :

where is the weight matrix, is the input representation, and is the bias vector.

  1. Optimization and Loss Minimization: Gradient descent, Newton-Raphson, and quasi-Newton optimization methods navigate multidimensional loss surfaces using gradient vectors and second-derivative Hessian matrices .
  2. Image and Signal Processing: Grayscale digital images are encoded as matrices of normalized pixel brightness values in or . Color digital images are tensors across Red, Green, and Blue channels. Geometric image manipulations—such as rotations, zooms, translations, and perspective corrections—are computed by multiplying pixel coordinates by homogeneous transformation matrices.
  3. Computational Performance, BLAS, and GPU Vectorization: Modern machine learning libraries (NumPy, PyTorch, TensorFlow) achieve extreme throughput because linear algebra operations map directly to hardware-accelerated Basic Linear Algebra Subprograms (BLAS) and matrix processing engines (NVIDIA CUDA Tensor Cores). Vectorized operations compute millions of multiply-accumulate steps in parallel, avoiding high-overhead scalar loops.

Systems of Linear Equations

A system of linear equations in unknown variables is expressed algebraically as:

Using matrix algebra, this entire system compresses into the compact matrix equation:

where:

Depending on the rank of matrix and augmented matrix , the system exhibits:

  1. A unique solution: When (system is consistent with full column rank; for square matrices, and ).
  2. Infinitely many solutions: When (under-determined system with free variables).
  3. No solution: When (over-determined inconsistent system; in data science, we approximate the best pseudo-solution via least-squares: ).

Linear Transformations

Definition

Linear Transformation

A mapping is a linear transformation if and only if it preserves vector addition and scalar multiplication for all vectors and all scalars :

  1. Additivity (Superposition):
  1. Homogeneity (Scalar Multiplication):

Combining these two conditions yields the fundamental linearity condition:

Every linear transformation can be represented uniquely as matrix multiplication by an standard matrix :

where the columns of are the images of the standard basis vectors :

AspectGeometric Vector SpaceData Feature Space
Dimension2 or 3 spatial dimensionsHundreds or thousands of feature dimensions
CoordinatesSpatial axes ()Numerical attributes (age, salary, sensor voltage, word counts)
Linear MappingGeometric rotation, reflection, scaling, shearingFeature projection, latent factor encoding, dimensionality reduction
Distance MeaningPhysical Euclidean distanceMetric dissimilarity between observations or customers
Origin Spatial reference pointMean-centered feature baseline (all features zeroed)

2.2 Vectors, Matrices, and Matrix Factorization

Vectors in Data Science

A vector represents a single point or directed segment in an -dimensional space:

Fundamental Vector Operations

Let and be vectors in , and :

  1. Vector Addition:
  1. Scalar Multiplication:
  1. Dot Product (Inner Product):

The dot product returns a scalar. Two non-zero vectors are orthogonal if and only if .

Vector Norms and Distances

Norms measure the length or magnitude of vectors:

Used in Lasso regression ( regularization) to enforce feature sparsity by driving irrelevant coefficients to exact zeros.

Used in Ridge regression ( weight decay) to prevent weights from exploding.

Cosine Similarity

The geometric definition of the dot product links the algebraic sum to the angle between two vectors:

Solving for defines the Cosine Similarity:

In Natural Language Processing (NLP) and recommender systems, documents and user preferences are encoded as high-dimensional term-frequency or embedding vectors. Cosine similarity evaluates orientation rather than magnitude: two documents discussing identical topics with different word lengths produce collinear vectors ().

Matrices in Data Science

A matrix is a rectangular array:

Taxonomy of Matrices

Matrix Operations

  1. Matrix Addition: Commutative and associative for matrices of identical dimension :
  1. Matrix Multiplication: Matrix multiplies to yield :

Crucial rule: Inner dimensions must match (). Matrix multiplication is associative () and distributive, but not commutative ( in general).

  1. Matrix Transpose: Reflects matrix entries over the main diagonal:
  1. Determinant and Inverse: For an square matrix, the determinant quantifies the volumetric scaling factor of the transformation. A matrix is invertible (non-singular) if and only if . Its inverse satisfies:

Eigenvalues and Eigenvectors

For a square matrix , a non-zero vector is an eigenvector with corresponding eigenvalue if:

Non-trivial solutions exist if and only if the characteristic determinant vanishes:

Eigenvalues and eigenvectors identify the invariant axes of linear operators, serving as the foundational engine for Principal Component Analysis (PCA) and Spectral Graph Clustering.

Matrix Factorization

Definition

Matrix Factorization

Matrix Factorization (or matrix decomposition) is the process of decomposing a large, potentially incomplete data matrix into a product of two or more lower-rank matrices:

where:

Why Factorize Matrices in Data Science?

  1. Dimensionality Reduction: Compresses massive feature spaces into compact latent dimensions while discarding high-frequency noise.
  2. Latent Feature Discovery: Discovers underlying unobserved patterns (e.g., identifying film themes like "romance", "action", "sci-fi" from user rating records).
  3. Handling Missing Values & Collaborative Filtering: Real-world interaction matrices (such as customer ratings on Netflix or Amazon) are over sparse. Factorization fits parameters on known entries to predict missing ratings .
Taxonomy of Matrix Factorization Techniques in Data Science
Factorization TechniqueMathematical FormulationPrimary ConstraintMajor Use Case
Singular Value Decomposition (SVD) orthogonal; diagonalPCA, Latent Semantic Analysis (LSA), noise filtering
Non-negative Matrix Factorization (NMF)Topic modeling, image part decomposition, gene expression
Alternating Least Squares (ALS)Alternating quadratic convex optimizationLarge-scale recommender systems with explicit/implicit feedback
Stochastic Gradient Descent (SGD)Iterative loss minimization over observed ratingsNetflix Prize collaborative filtering

Singular Value Decomposition (SVD)

Theorem

Singular Value Decomposition (SVD) Theorem

Any real rectangular matrix of rank can be factored into the product of three matrices:

where:

  1. is an orthogonal matrix whose columns are the left singular vectors of (the orthonormal eigenvectors of ).
  2. is a diagonal matrix containing the non-negative singular values arranged in descending order along its main diagonal:
  1. is an orthogonal matrix whose columns are the right singular vectors of (the orthonormal eigenvectors of ). is its transpose.

Algorithmic Procedure for Computing SVD

To compute the full SVD of an matrix :

  1. Form the symmetric matrix : Solve the characteristic equation for eigenvalues .
  2. Compute singular values :

Construct the diagonal matrix with diagonal entries .

  1. Find right singular vectors : For each eigenvalue , solve , normalize to unit Euclidean length , and assemble the orthogonal matrix .
  2. Compute left singular vectors : For each non-zero singular value :

This ensures directional alignment (). If , complete the remaining columns of using Gram-Schmidt orthogonalization on the nullspace of .

  1. Verify the decomposition: Check that .
AspectSingular Value Decomposition (SVD)Non-Negative Matrix Factorization (NMF)
Factor ConstraintsOrthonormal singular vectors ()Non-negativity ()
InterpretabilityFactors have mixed positive and negative signs (hard to explain physically)Purely additive parts-based representation (readily interpretable)
UniquenessMathematically unique up to column sign flipsNon-convex; local minima depending on initialization
Data RequirementsOperates on arbitrary real matricesInput matrix entries must be strictly non-negative ()

2.3 Gradient Descent for Optimization

In machine learning and data science, learning is formulated as an optimization problem: finding the model parameter vector that minimizes a real-valued empirical cost or loss function :

Except for simple models like ordinary least squares regression, real-world loss functions have no analytical closed-form solution. We must instead use iterative numerical optimization algorithms, among which Gradient Descent is the primary engine.

Definition

Gradient Descent Optimization

Gradient Descent is a first-order iterative optimization algorithm used to find a local minimum of a differentiable objective function by taking successive steps proportional to the negative of the gradient of the function at the current point.

Geometric Intuition and the Gradient Vector

Consider an objective function defined over parameter vector .

The gradient vector is the vector of all first-order partial derivatives:

From multivariable calculus, the directional derivative of in unit direction is:

where is the angle between and :

Physical Analogy: Imagine standing blindfolded on a foggy, mountainous terrain. To reach the bottom of the valley, you feel the slope of the ground beneath your feet and take a step in the steepest downhill direction. Repeating this process brings you to the valley floor.

The Parameter Update Rule

Starting from an initial parameter guess (initialized randomly or with heuristics), the parameter vector is updated iteratively:

where:

The iteration terminates when a stopping condition is reached:

  1. The gradient norm falls below a threshold: .
  2. The decrease in loss between consecutive iterations is negligible: .
  3. A pre-set maximum number of iterations (epochs) is completed.

Impact of the Learning Rate ()

The learning rate is one of the most critical hyperparameters in model training:

  1. Learning Rate Too Small (): The parameter updates are minuscule. Training requires an excessive number of iterations, leading to high computational costs and the risk of stalling on flat plateaus or saddle points.
  2. Learning Rate Optimal (): The parameters descend smoothly and rapidly along the error surface, converging reliably to the minimum.
  3. Learning Rate Too Large (): The algorithm overshoots the minimum. Successive updates oscillate wildly across the valley walls and may diverge, driving the loss to numerical infinity ().

Variants of Gradient Descent

In practical machine learning, the cost function is an average over training examples:

Depending on how many samples are evaluated before performing each parameter update, gradient descent is classified into three variants:

AspectBatch Gradient DescentStochastic Gradient Descent (SGD)
Data per UpdateThe entire training dataset ( observations)Exactly one randomly chosen sample
Update FrequencyOnce per epoch (after scanning all samples) times per epoch (after every individual sample)
TrajectorySmooth, deterministic, and directNoisy, erratic, and fluctuating
Memory FootprintHigh; requires entire dataset in memory ()Minimal; processes one sample at a time ()
Local MinimaCan get trapped in shallow local minimaGradient noise helps jump out of bad local minima

1. Batch Gradient Descent (BGD)

Uses all training instances to compute the exact gradient:

2. Stochastic Gradient Descent (SGD)

Picks a single sample uniformly at random and updates parameters immediately:

3. Mini-Batch Gradient Descent

The industry standard in deep learning and modern data science. It splits the training dataset into small subsets (mini-batches) of size (typically samples):

Comparative Architecture of Gradient Descent Variants
VariantBatch Size ()Gradient VarianceEpoch Update CountComputational EfficiencyHardware Acceleration
Batch GDZero (deterministic)Low for large Moderate (large matrix limits)
Stochastic GDExtremely highLow vectorizationPoor (scalar overhead)
Mini-Batch GDLow to moderateOptimalOptimal (maximizes GPU throughput)

Worked Examples

Example 2.1

2.1

In an artificial neural network layer, an input feature vector is transformed by a weight matrix . Compute the linear transformation , and explain the geometric effect of this transformation.

Evaluating the matrix-vector product: The transformed vector is computed by multiplying the transformation matrix by the column vector :

Geometric interpretation: The matrix is a uniform isotropic scaling operator. It stretches every vector in by a factor of radially outward from the origin without altering its orientation:

The length doubles while the direction remains identical.

Answer:

The transformation scales the input vector by a factor of along all coordinate axes, doubling its magnitude while preserving its angular orientation.

Example 2.2

2.2

A movie recommender system represents customer preferences and movie attributes across four genres (Comedy, Romance, Action, Sci-Fi) as vectors:

Calculate:

  1. The inner (dot) product .
  2. The Euclidean norms and .
  3. The cosine similarity between the user profile and the candidate movie .

Computing the dot product: Multiply corresponding components and sum:

Computing the Euclidean norms: Calculate the norm for each vector:

Calculating cosine similarity: Divide the dot product by the product of the norms:

Answer:

The high cosine similarity () indicates strong alignment between the user's preferences and the movie's attributes.

Example 2.3

2.3

Compute the parameter updates for the quadratic objective function using Gradient Descent. Assume:

Analytical derivative of the objective function: Differentiating with respect to :

The theoretical global minimum occurs where .

First parameter update (): Evaluate gradient at initial position :

Update parameter:

Second parameter update (): Evaluate gradient at :

Update parameter:

Third parameter update (): Evaluate gradient at :

Update parameter:

Fourth parameter update (): Evaluate gradient at :

Update parameter:

Answer:

The parameter monotonically approaches the global minimizer .

Example 2.4

2.4

Compute the Singular Value Decomposition (SVD) of the symmetric matrix:

Find the orthogonal matrices and the diagonal singular value matrix such that .

Computing and characteristic equation: Since is symmetric, :

Set up the characteristic equation:

Singular values and matrix : The singular values are the square roots of the eigenvalues:

Right singular vectors : For :

Normalized unit vector:

For :

Normalized unit vector:

Assembling and :

Left singular vectors : Using the relation :

Assembling :

Answer:

Exercises

Exercises — Linear Algebra, Matrix Factorization & Gradient Descent

Question 1

Define linear algebra and explain its importance in data science with concrete examples. Explain what is meant by a linear transformation and illustrate it with an algebraic example.

Definition and scope of linear algebra: Linear algebra is the mathematical foundation of vector spaces, linear mappings, and systems of linear equations. In modern data science, datasets are modeled as matrices where rows represent observations and columns represent feature dimensions.

Key areas of importance in data science:

  1. Data Representation: An observation with features (e.g., age, income, credit score) is a vector . An entire dataset of observations is a design matrix .
  2. Machine Learning Algorithms: Linear regression relies on the normal equations ; deep neural networks compute layer-wise affine mappings ; support vector machines optimize hyperplane margins using inner products.
  3. Dimensionality Reduction: Techniques like Principal Component Analysis (PCA) project data along eigenvectors corresponding to the largest eigenvalues of the covariance matrix.
  4. Computer Vision: Digital images are stored as pixel matrices (grayscale) or tensors (RGB color). Image operations like filtering, rotation, and resizing are matrix transformations.
  5. Computational Speed: Linear algebra libraries leverage BLAS and GPU hardware to perform millions of parallel matrix multiplications, enabling scalable training on big data.

Linear transformation definition and properties: A mapping is a linear transformation if it satisfies two axioms:

  1. Additivity: for all .
  2. Homogeneity: for all and .

Concrete algebraic example: Consider the transformation matrix acting on vector :

This represents a uniform scaling transformation that doubles the length of any vector in while preserving its direction.

Answer:

Linear algebra is the study of vectors, matrices, and linear transformations that structure data representation, regression, neural networks, PCA, and GPU acceleration in data science. A linear transformation satisfies and , exemplified by scaling to .

Question 2

Explain matrix factorization with an example. Explain the working mechanism of Singular Value Decomposition (SVD) and derive the complete SVD factorization for the matrix:

Concept of matrix factorization: Matrix factorization decomposes an matrix into a product of lower-rank factor matrices, typically , where and with . It discovers latent features, removes noise, and enables missing-value prediction in recommender systems.

SVD formulation and mechanism: Singular Value Decomposition factors any real matrix into:

where contains orthonormal eigenvectors of (left singular vectors), contains orthonormal eigenvectors of (right singular vectors), and contains the singular values in descending order along its diagonal.

Computing and its eigenvalues:

Find eigenvalues by solving the characteristic equation:

Singular values and matrix : The singular values are the square roots of the eigenvalues:

Computing and right singular vectors :

For :

Normalized unit vector:

For :

Normalized unit vector:

Assembling and :

Determining left singular vectors :

Assembling :

Reconstruction verification:

Answer:

where singular values are and .

Question 3

State one real-world application of matrix factorization in data science and explain its mechanism in detail.

Real-world application: Recommender Systems: One of the most prominent applications of matrix factorization is in Collaborative Filtering Recommender Systems, famously deployed by streaming and e-commerce platforms like Netflix, Spotify, and Amazon.

Problem formulation: Consider an ecosystem with users and items (movies, songs, products). The user interactions form a massive utility matrix , where entry represents the rating given by user to item (e.g., to stars).

Because typical users rate only a tiny fraction of the available catalog, is extremely sparse (often empty entries). The data science goal is to predict the missing ratings and recommend items with the highest predicted scores.

Latent factor model mechanism: Matrix factorization maps both users and items into a shared latent factor space of dimensionality :

where:

  • : User-feature matrix where row measures how much user prefers each latent feature (e.g., action, drama, budget, pacing).
  • : Item-feature matrix where row measures how strongly item embodies those same latent features.

The predicted rating of user for item is the dot product of their latent representations:

Training via regularized loss minimization: To learn and from the observed ratings , we minimize the regularized squared error:

where is a regularization parameter preventing overfitting on sparse entries. This objective is optimized efficiently using Stochastic Gradient Descent (SGD) or Alternating Least Squares (ALS).

Answer:

In collaborative filtering recommender systems (e.g., Netflix, Spotify), matrix factorization decomposes a sparse user-item interaction matrix into user latent factor matrix and item latent factor matrix such that predicted ratings are computed as , enabling high-accuracy recommendations.

Question 4

Define gradient descent and explain its purpose in machine learning. Write down its iterative mathematical formula, identify all components, and describe its geometric intuition.

Definition and purpose: Gradient Descent is a first-order iterative numerical optimization algorithm designed to minimize a differentiable cost or loss function by making parameter adjustments proportional to the negative gradient.

Its core purpose in machine learning is empirical risk minimization: training algorithms cannot solve for optimal weights analytically when loss landscapes are high-dimensional, non-linear, or non-convex. Gradient descent systematically adjusts model parameters to minimize prediction error.

Iterative mathematical formula:

where:

  • is the parameter vector at iteration .
  • is the updated parameter vector.
  • is the gradient vector of partial derivatives evaluated at .
  • is the learning rate (step size hyperparameter).
  • The minus sign () directs the step opposite to the gradient, descending toward the minimum.

Geometric interpretation: The gradient vector points in the direction of maximum directional derivative (steepest ascent). By taking steps in the opposite direction (), the algorithm traverses the path of steepest descent, moving toward the lowest point (local or global minimum) on the error surface.

Answer:

Gradient descent minimizes machine learning loss functions by iteratively updating parameters via . The gradient indicates the direction of steepest error increase; subtracting a fraction moves parameters in the direction of steepest descent toward minimum error.

Question 5

Explain the differences between Batch Gradient Descent, Stochastic Gradient Descent (SGD), and Mini-Batch Gradient Descent. Discuss their respective advantages, disadvantages, and suitability for real-world datasets.

Batch Gradient Descent (BGD): Computes the gradient of the loss function using the entire training dataset of observations at each step:

  • Pros: Stable, deterministic convergence trajectory; guarantees finding the global minimum for convex functions.
  • Cons: Very slow per iteration; memory-intensive (); fails when datasets exceed RAM capacity.

Stochastic Gradient Descent (SGD): Updates parameters after evaluating only a single, randomly chosen training instance :

  • Pros: Ultra-fast updates; minimal memory footprint; online learning capability; stochastic fluctuations help jump out of shallow local minima.
  • Cons: High gradient variance leads to noisy oscillations, preventing smooth settling without a carefully decaying learning rate schedule.

Mini-Batch Gradient Descent: Partitions the dataset into small batches of size (typically ) and updates parameters per mini-batch:

  • Pros: Combines the stability of BGD with the efficiency of SGD; maps directly onto vectorized GPU matrix operations (CUDA Tensor Cores).
  • Cons: Introduces batch size as an additional hyperparameter to tune.

Comparative summary table:

FeatureBatch Gradient DescentStochastic Gradient Descent (SGD)Mini-Batch Gradient Descent
Samples per stepAll samples sample samples (–)
Convergence pathDirect, smoothHighly erratic, noisyModerately smooth
Computational costHigh per step ()Very low per step ()Well-balanced ()
Hardware accelerationSuboptimal for massive Poor (sequential overhead)Optimal (fully leverages GPU SIMD)
Answer:

Batch GD updates once per epoch using all samples (smooth but memory-heavy); SGD updates after every individual sample (fast but noisy); Mini-Batch GD updates after evaluating samples (–), achieving optimal balance, fast GPU vectorization, and smooth convergence in production machine learning.

Question 6

Discuss how the choice of learning rate affects the convergence of gradient descent. Describe what happens when the learning rate is too small, optimal, or too large, and explain the concept of learning rate scheduling.

Role of the learning rate (): The learning rate controls the step size taken along the negative gradient vector. It determines the balance between training stability and convergence speed.

Effects of different learning rate regimes:

  1. Too Small (): The model makes tiny adjustments to parameters. Training progresses at a crawl, requiring thousands of redundant epochs, consuming excessive computational time, and frequently stalling in shallow plateaus or saddle points.
  2. Optimal (): The parameters descend steadily and stably down the loss landscape, converging efficiently to the neighborhood of the minimum.
  3. Too Large (): The step size overshoots the minimum point of the cost function. Successive iterations oscillate across opposite canyon walls with increasing amplitude, ultimately causing numerical overflow and catastrophic divergence ().

Learning rate scheduling and adaptive methods: To resolve this trade-off, practitioners use dynamic learning rate strategies:

  • Step Decay: Reduce by a factor (e.g., ) every fixed number of epochs.
  • Exponential Decay: .
  • Adaptive Optimizers (Adam, RMSprop): Compute per-parameter adaptive learning rates based on running averages of first and second moments of the gradients.
Answer:

An undersized learning rate causes sluggish convergence and high computational expense; an oversized learning rate overshoots the valley floor, causing severe oscillations and divergence. An optimal learning rate ensures fast, steady convergence, often augmented by learning rate decay or adaptive optimizers (Adam) that decrease as the minimum approaches.

Question 7

In an information retrieval task, two document vectors are represented in a 5-word vocabulary space as:

Compute their dot product, Euclidean norms, and cosine similarity. State whether these documents are considered similar.

Evaluating the dot product: Multiply corresponding term-frequency components:

The two documents share vocabulary terms in common.

Computing the Euclidean norms:

Calculating cosine similarity:

Answer:

With a cosine similarity of (), the documents exhibit substantial semantic overlap and are considered strongly related.

Question 8

Perform two iterations of multivariable gradient descent on the objective function:

starting from initial point with learning rate . Calculate the coordinates after two updates.

Deriving the gradient vector: Compute partial derivatives with respect to each coordinate:

The global minimum is at the origin where .

First iteration (): Evaluate gradient at :

Update coordinates:

Value of function: (reduced from ).

Second iteration (): Evaluate gradient at :

Update coordinates:

Value of function: . Notice that coordinates obey and .

Answer:

After two iterations, the objective value drops from to and then to , moving toward the origin.

Past exam questions for Foundation of Data Science

← Trends, Applications, Tools & the Data ScientistProbability, Random Variables & Distributions →