PadhaikoAll subjects

Data Transformation, Encoding & Normalization

Foundation of Data Science

Open in the Padhaiko app

3.7 Data Transformation and Normalization

Numerical datasets gathered across multiple domains typically exhibit wide disparities in scale, unit of measurement, and statistical distribution. For instance, in an employee analytics dataset, salary may range from to , while employee age ranges from to years, and experience from to years. If fed directly into learning algorithms, high-magnitude features disproportionately dominate the model's loss function, leading to distorted parameter estimation and slow gradient convergence.

Definition

Data Normalization and Scaling

Data Normalization is the process of mathematically transforming numerical feature values onto a standardized, common numerical scale (typically , , or a standard normal distribution with and ) without altering the relative ordering or information content of the data.

Normalization is indispensable in data preprocessing for three primary reasons:

  1. Equalizing Feature Influence: Distance-based algorithms (such as -nearest neighbors, -means clustering, support vector machines, and principal component analysis) calculate Euclidean distance:

Without scaling, features with large absolute ranges completely overpower smaller features.

  1. Accelerating Gradient Descent Convergence: In parameterized models (linear regression, logistic regression, artificial neural networks), unscaled features produce highly eccentric, elongated elliptical cost surfaces. Gradient vectors oscillate back and forth perpendicular to the gradient path, demanding tiny learning rates. Normalized features transform the cost surface into near-spherical contours, enabling direct and rapid convergence toward the global minimum.
  2. Preventing Numerical Instability: Prevents arithmetic overflow or underflow when computing exponential operations in activation functions (e.g., sigmoid or softmax).

Mathematical Normalization Techniques

1. Min-Max Normalization (Feature Rescaling)

Min-Max normalization linearly maps the original feature values from their native range into a target interval (most commonly ):

When mapping to , the formula simplifies to:

2. -Score Standardization (Zero-Mean Scaling)

-score standardization transforms feature values based on the sample mean ( or ) and standard deviation ( or ):

The resulting standardized variable has mean and standard deviation :

3. Decimal Scaling Normalization

Decimal scaling normalizes feature values by shifting the decimal point according to the maximum absolute value observed:

where is the smallest integer such that , formally computed as:

AspectMin-Max NormalizationZ-Score Standardization
Mathematical Formula
Resulting RangeFixed bounded range (typically )Unbounded (), centered at
Distribution ParametersRange is standardized; mean and variance varyMean is strictly ; standard deviation is strictly
Outlier SensitivityVery high: outliers compress standard dataModerate: outliers still have large -scores but do not compress the bulk
Ideal Model ContextNeural networks, image processing, -NNLinear regression, logistic regression, SVM, PCA

Non-Linear Data Transformations

When feature distributions exhibit extreme right-skewness or heteroscedasticity (non-constant variance), linear rescaling alone is insufficient. Data scientists employ non-linear transformations:

Compresses large values while expanding smaller values, transforming exponential or heavy-tailed distributions into approximately Gaussian distributions.

Categorical Feature Encoding

Machine learning models require numeric tensor inputs. Categorical variables must be systematically encoded into numeric representations:

  1. Ordinal Encoding (Label Encoding): Appropriate for categorical features with an intrinsic order or rank (e.g., Education Level: "High School" , "Bachelor" , "Master" , "PhD" ).
  2. One-Hot Encoding (OHE): Appropriate for nominal features with no intrinsic order (e.g., City: "Kathmandu", "Pokhara", "Biratnagar"). For a categorical variable with distinct categories, OHE generates binary indicator columns ().
    • The Dummy Variable Trap: In linear regression models containing an intercept term (), including all binary columns creates perfect multicollinearity (). To avoid matrix singularity, exactly one category must be omitted ( columns retained as the reference baseline).

Data Discretization and Binning

Discretization converts continuous numerical features into categorical intervals (bins):

Example

Min-Max and Z-Score Normalization of Salary and Age

Perform step-by-step min-max normalization and -score standardization for both Salary and Age attributes across all five employee records.

Min-Max Normalization of Salary: Identify feature extremes:

Apply :

  • For :
  • For :
  • For :
  • For :
  • For :

Min-Max Normalization of Age: Identify feature extremes:

Apply :

  • For :
  • For :
  • For :
  • For :
  • For :

Min-Max normalized dataset summary:

FeatureEmp 1Emp 2Emp 3Emp 4Emp 5
Salary (Min-Max)
Age (Min-Max)

-Score Standardization of Salary: Compute the population mean :

Compute squared deviations from the mean:

Calculate population variance and standard deviation ():

Compute :

  • For :
  • For :
  • For :
  • For :
  • For :

-Score Standardization of Age: Compute mean :

Compute squared deviations:

Calculate population variance and standard deviation:

(Using sample correction : )

Compute :

  • For :
  • For :
  • For :
  • For :
  • For :

Standardized (-score) dataset summary:

FeatureEmp 1Emp 2Emp 3Emp 4Emp 5
Salary (-score)
Age (-score)
Answer:
  • Min-Max Normalized ():
  • -Score Standardized ():
Example

One-Hot Categorical Encoding and Decimal Scaling

Compute the decimal scaling exponent and scaled features for HoursWorked, and construct the one-hot encoded indicator matrix for Degree.

Determine decimal scaling parameter : Find the maximum absolute value in the feature:

Determine the smallest integer such that :

Divide each value by :

All scaled values lie strictly in the range .

Construct one-hot encoding for categorical attribute: The attribute has unique nominal categories:

  • Category 1: "Bachelor"
  • Category 2: "Master"
  • Category 3: "PhD"

Mitigate dummy variable trap: In linear regression with an intercept , the sum of the three dummy variables equals for all observations:

This introduces exact collinearity with the constant intercept vector , making the Gram matrix singular and non-invertible. To prevent this, one category is designated as the reference baseline (e.g., "Bachelor") and dropped, retaining indicator columns:

  • "Bachelor":
  • "Master":
  • "PhD":
Answer:

Decimal scaling uses (), yielding . One-hot encoding creates binary columns, but to prevent the dummy variable trap in regression with an intercept, columns () are retained with "Bachelor" as the reference base.


Exercises: Transformation, Normalization & Encoding

Question 1

Explain the concept of normalization and why it is used in data preprocessing. Provide an illustrative example showing how unnormalized features bias machine learning models.

Definition of Normalization: Normalization is the process of rescaling continuous numerical features into a standardized target range (such as or standard normal with ). It prevents attributes with large numerical scales from dominating those with smaller numerical ranges.

Reasons for Normalization in Preprocessing:

  1. Algorithm Fairness in Distance Calculations: Classifiers such as -nearest neighbors and -means compute geometric distances. Unscaled features with large ranges overwhelm the distance metric.
  2. Gradient Descent Optimization Stability: Neural networks and linear models optimize weights via gradient updates (). Unscaled features distort the loss surface into steep valleys, causing oscillations and slow convergence.
  3. Regularization Consistency: Penalties such as Ridge () and Lasso () penalize weights uniformly (). Without scaling, weights for large-magnitude features are artificially small and evade penalization, while weights for small-scale features are excessively penalized.

Illustrative Real-World Example: Consider predicting apartment rental prices using ( to ) and ( to ). In an unnormalized model, a difference of generates a squared difference of , whereas a -bedroom difference generates only . The model ignores bedroom count entirely unless normalized.

Answer:

Normalization scales numerical features onto a common scale. It is required to equalize feature contributions in distance-based models, accelerate gradient descent convergence, and ensure consistent regularization penalties. An example is housing prediction, where floor area () completely dominates bedroom count () unless both are normalized.

Question 2

Differentiate between Min-Max Normalization and -Score Standardization across at least four distinct comparative criteria.

Construct comparative criteria:

  1. Mathematical Formulation:
    • Min-Max:
    • -Score:
  2. Output Range:
    • Min-Max: Strictly bounded (typically or ).
    • -Score: Unbounded (), with mean centered at .
  3. Sensitivity to Outliers:
    • Min-Max: Extremely sensitive; extreme outliers compress normal data into a narrow, uninformative band.
    • -Score: Moderate sensitivity; preserves the outlier's position without compressing the remaining sample.
  4. Preservation of Statistical Properties:
    • Min-Max: Preserves the exact original distribution shape and relative differences, but mean and variance vary.
    • -Score: Forces mean to exactly and standard deviation to exactly .
Answer:

Min-Max scales data into a bounded range based on extremes () and is highly sensitive to outliers. -Score standardization maps data to an unbounded scale with and based on sample mean and standard deviation, offering greater resilience against outliers.

Question 3

A machine learning engineer evaluates a set of five student exam scores:

  1. Compute the normalized scores under standard Min-Max normalization to the interval .
  2. Rescale the median score () to the target interval using the generalized min-max formulation.

Min-Max Normalization to : Identify extrema:

Apply :

  • :
  • :
  • :
  • :
  • :

Generalized Min-Max Rescaling to for score : The generalized formula is:

Here and , so the range multiplier is :

Answer:
  • Normalized to : .
  • Rescaled to : the score maps to .
Question 4

A dataset of graduate student ages is collected:

  1. Compute the population mean and standard deviation .
  2. Calculate the -score standardized values for the youngest student () and oldest student ().
  3. Normalize the oldest student's age () using decimal scaling.

Compute mean and population standard deviation:

Sum of squared deviations:

Population variance and standard deviation ():

(Using sample standard deviation )

Compute -scores:

  • Youngest student ():

(With sample : )

  • Oldest student ():

(With sample : )

Decimal scaling for : Maximum absolute value is . Find :

Answer:

and . The -score for age is (or with ); for age it is (or with ). Under decimal scaling (), age normalizes to .

Question 5

Compare Ordinal (Label) Encoding with One-Hot Encoding. Explain why applying label encoding to nominal features can mislead linear algorithms, and define the dummy variable trap.

Comparison of Encoding Mechanisms:

  • Ordinal (Label) Encoding: Maps distinct categories to sequential integers (). It assumes a meaningful natural ordering among categories.
  • One-Hot Encoding: Expands a category into separate binary indicator columns (), where each column signifies the presence of one category. It assumes no ordering.

Why Label Encoding misleads linear algorithms on nominal data: If nominal categories without natural ordering (e.g., Country: Nepal , Japan , Germany ) are assigned integers, a linear model enforces an arithmetic relationship:

The model assumes countries are equally spaced along a single numerical spectrum, which introduces false ordering and arbitrary geometric constraints.

The Dummy Variable Trap: When a categorical feature with categories is converted to binary columns in a linear regression model that includes an intercept term , the sum of the columns equals the intercept column:

This creates perfect multicollinearity, making the feature matrix linearly dependent. The Gram matrix becomes singular and non-invertible. To avoid this trap, one binary column is omitted ( columns used), treating the omitted category as the reference baseline.

Answer:

Label encoding assigns integers to categories and is valid only for ordered attributes; applying it to nominal attributes introduces artificial ordering. One-hot encoding creates binary columns without assuming order. The dummy variable trap occurs when all columns are included in a regression model with an intercept, causing perfect multicollinearity. It is resolved by dropping one column ( retained).

Question 6

A financial dataset contains four heavily skewed customer annual income values:

  1. Calculate the natural logarithm for the minimum and maximum income values.
  2. Evaluate how the logarithmic transformation compresses the dynamic range between the extremes.

Compute natural logarithms:

  • Minimum income ():
  • Maximum income ():

Quantify dynamic range compression:

  • Original ratio:

The highest earner has the income of the lowest earner.

  • Log-transformed difference:

The multiplicative factor of is compressed into an additive difference of only units. This stabilizes the variance and reduces the leverage of extreme income earners in regression models.

Answer:

and . The logarithmic transformation compresses a -fold multiplicative difference in raw scale into an additive difference of only , dampening skewness and reducing outlier influence.

Question 7

A data pipeline needs to discretize a continuous performance score attribute using Equal-Width Binning with bins. The observed scores are:

  1. Calculate the bin width.
  2. Define the interval boundaries for the three bins.
  3. Assign each score to its respective bin.

Calculate bin width: Identify the minimum and maximum scores:

Define interval boundaries: Construct contiguous intervals:

  • Bin 1:
  • Bin 2:
  • Bin 3: (closed at upper boundary to include )

Assign scores to bins:

  • Bin 1 (): (Count = 4)
  • Bin 2 (): (Count = 3)
  • Bin 3 (): (Count = 3)
Answer:

Bin width is . The intervals are with 4 observations, with 3 observations, and with 3 observations.

Past exam questions for Foundation of Data Science

← Cleaning, Wrangling, Enrichment, Validation & PublishingDimensionality Reduction, Linear Factor Model & PCA →