Data Transformation, Encoding & Normalization
Open in the Padhaiko app3.7 Data Transformation and Normalization
Numerical datasets gathered across multiple domains typically exhibit wide disparities in scale, unit of measurement, and statistical distribution. For instance, in an employee analytics dataset, salary may range from to , while employee age ranges from to years, and experience from to years. If fed directly into learning algorithms, high-magnitude features disproportionately dominate the model's loss function, leading to distorted parameter estimation and slow gradient convergence.
Data Normalization and Scaling
Data Normalization is the process of mathematically transforming numerical feature values onto a standardized, common numerical scale (typically , , or a standard normal distribution with and ) without altering the relative ordering or information content of the data.
Normalization is indispensable in data preprocessing for three primary reasons:
- Equalizing Feature Influence: Distance-based algorithms (such as -nearest neighbors, -means clustering, support vector machines, and principal component analysis) calculate Euclidean distance:
Without scaling, features with large absolute ranges completely overpower smaller features.
- Accelerating Gradient Descent Convergence: In parameterized models (linear regression, logistic regression, artificial neural networks), unscaled features produce highly eccentric, elongated elliptical cost surfaces. Gradient vectors oscillate back and forth perpendicular to the gradient path, demanding tiny learning rates. Normalized features transform the cost surface into near-spherical contours, enabling direct and rapid convergence toward the global minimum.
- Preventing Numerical Instability: Prevents arithmetic overflow or underflow when computing exponential operations in activation functions (e.g., sigmoid or softmax).
Mathematical Normalization Techniques
1. Min-Max Normalization (Feature Rescaling)
Min-Max normalization linearly maps the original feature values from their native range into a target interval (most commonly ):
When mapping to , the formula simplifies to:
- Strengths: Bounded range; preserves zero-entries if mapped appropriately; retains exact relative distances between points.
- Limitations: Highly sensitive to extreme outliers; if a single extreme maximum enters the dataset, all standard values compress into a narrow cluster near .
2. -Score Standardization (Zero-Mean Scaling)
-score standardization transforms feature values based on the sample mean ( or ) and standard deviation ( or ):
The resulting standardized variable has mean and standard deviation :
- Strengths: Resilient against moderate outliers; does not bind data to a fixed interval; directly satisfies the assumption of many linear models (ridge regression, linear discriminant analysis).
- Limitations: Does not produce a bounded interval; not ideal when bounded positive features are strictly required (e.g., image pixels).
3. Decimal Scaling Normalization
Decimal scaling normalizes feature values by shifting the decimal point according to the maximum absolute value observed:
where is the smallest integer such that , formally computed as:
| Aspect | Min-Max Normalization | Z-Score Standardization |
|---|---|---|
| Mathematical Formula | ||
| Resulting Range | Fixed bounded range (typically ) | Unbounded (), centered at |
| Distribution Parameters | Range is standardized; mean and variance vary | Mean is strictly ; standard deviation is strictly |
| Outlier Sensitivity | Very high: outliers compress standard data | Moderate: outliers still have large -scores but do not compress the bulk |
| Ideal Model Context | Neural networks, image processing, -NN | Linear regression, logistic regression, SVM, PCA |
Non-Linear Data Transformations
When feature distributions exhibit extreme right-skewness or heteroscedasticity (non-constant variance), linear rescaling alone is insufficient. Data scientists employ non-linear transformations:
- Logarithmic Transformation:
Compresses large values while expanding smaller values, transforming exponential or heavy-tailed distributions into approximately Gaussian distributions.
- Power and Box-Cox Transformations: Parametric family of power transformations that systematically stabilizes variance:
Categorical Feature Encoding
Machine learning models require numeric tensor inputs. Categorical variables must be systematically encoded into numeric representations:
- Ordinal Encoding (Label Encoding): Appropriate for categorical features with an intrinsic order or rank (e.g., Education Level:
"High School","Bachelor","Master","PhD"). - One-Hot Encoding (OHE): Appropriate for nominal features with no intrinsic order (e.g., City:
"Kathmandu","Pokhara","Biratnagar"). For a categorical variable with distinct categories, OHE generates binary indicator columns ().- The Dummy Variable Trap: In linear regression models containing an intercept term (), including all binary columns creates perfect multicollinearity (). To avoid matrix singularity, exactly one category must be omitted ( columns retained as the reference baseline).
Data Discretization and Binning
Discretization converts continuous numerical features into categorical intervals (bins):
- Equal-Width Binning: Divides the range into intervals of identical width:
- Equal-Frequency (Quantile) Binning: Divides ordered data into intervals such that each bin contains approximately the same number of observations ().
Min-Max and Z-Score Normalization of Salary and Age
Perform step-by-step min-max normalization and -score standardization for both Salary and Age attributes across all five employee records.
Min-Max Normalization of Salary: Identify feature extremes:
Apply :
- For :
- For :
- For :
- For :
- For :
Min-Max Normalization of Age: Identify feature extremes:
Apply :
- For :
- For :
- For :
- For :
- For :
Min-Max normalized dataset summary:
| Feature | Emp 1 | Emp 2 | Emp 3 | Emp 4 | Emp 5 |
|---|---|---|---|---|---|
| Salary (Min-Max) | |||||
| Age (Min-Max) |
-Score Standardization of Salary: Compute the population mean :
Compute squared deviations from the mean:
Calculate population variance and standard deviation ():
Compute :
- For :
- For :
- For :
- For :
- For :
-Score Standardization of Age: Compute mean :
Compute squared deviations:
Calculate population variance and standard deviation:
(Using sample correction : )
Compute :
- For :
- For :
- For :
- For :
- For :
Standardized (-score) dataset summary:
| Feature | Emp 1 | Emp 2 | Emp 3 | Emp 4 | Emp 5 |
|---|---|---|---|---|---|
| Salary (-score) | |||||
| Age (-score) |
- Min-Max Normalized ():
- -Score Standardized ():
One-Hot Categorical Encoding and Decimal Scaling
Compute the decimal scaling exponent and scaled features for HoursWorked, and construct the one-hot encoded indicator matrix for Degree.
Determine decimal scaling parameter : Find the maximum absolute value in the feature:
Determine the smallest integer such that :
Divide each value by :
All scaled values lie strictly in the range .
Construct one-hot encoding for categorical attribute: The attribute has unique nominal categories:
- Category 1:
"Bachelor" - Category 2:
"Master" - Category 3:
"PhD"
Mitigate dummy variable trap: In linear regression with an intercept , the sum of the three dummy variables equals for all observations:
This introduces exact collinearity with the constant intercept vector , making the Gram matrix singular and non-invertible. To prevent this, one category is designated as the reference baseline (e.g., "Bachelor") and dropped, retaining indicator columns:
"Bachelor":"Master":"PhD":
Decimal scaling uses (), yielding . One-hot encoding creates binary columns, but to prevent the dummy variable trap in regression with an intercept, columns () are retained with "Bachelor" as the reference base.
Exercises: Transformation, Normalization & Encoding
Explain the concept of normalization and why it is used in data preprocessing. Provide an illustrative example showing how unnormalized features bias machine learning models.
Definition of Normalization: Normalization is the process of rescaling continuous numerical features into a standardized target range (such as or standard normal with ). It prevents attributes with large numerical scales from dominating those with smaller numerical ranges.
Reasons for Normalization in Preprocessing:
- Algorithm Fairness in Distance Calculations: Classifiers such as -nearest neighbors and -means compute geometric distances. Unscaled features with large ranges overwhelm the distance metric.
- Gradient Descent Optimization Stability: Neural networks and linear models optimize weights via gradient updates (). Unscaled features distort the loss surface into steep valleys, causing oscillations and slow convergence.
- Regularization Consistency: Penalties such as Ridge () and Lasso () penalize weights uniformly (). Without scaling, weights for large-magnitude features are artificially small and evade penalization, while weights for small-scale features are excessively penalized.
Illustrative Real-World Example: Consider predicting apartment rental prices using ( to ) and ( to ). In an unnormalized model, a difference of generates a squared difference of , whereas a -bedroom difference generates only . The model ignores bedroom count entirely unless normalized.
Normalization scales numerical features onto a common scale. It is required to equalize feature contributions in distance-based models, accelerate gradient descent convergence, and ensure consistent regularization penalties. An example is housing prediction, where floor area () completely dominates bedroom count () unless both are normalized.
Differentiate between Min-Max Normalization and -Score Standardization across at least four distinct comparative criteria.
Construct comparative criteria:
- Mathematical Formulation:
- Min-Max:
- -Score:
- Output Range:
- Min-Max: Strictly bounded (typically or ).
- -Score: Unbounded (), with mean centered at .
- Sensitivity to Outliers:
- Min-Max: Extremely sensitive; extreme outliers compress normal data into a narrow, uninformative band.
- -Score: Moderate sensitivity; preserves the outlier's position without compressing the remaining sample.
- Preservation of Statistical Properties:
- Min-Max: Preserves the exact original distribution shape and relative differences, but mean and variance vary.
- -Score: Forces mean to exactly and standard deviation to exactly .
Min-Max scales data into a bounded range based on extremes () and is highly sensitive to outliers. -Score standardization maps data to an unbounded scale with and based on sample mean and standard deviation, offering greater resilience against outliers.
A machine learning engineer evaluates a set of five student exam scores:
- Compute the normalized scores under standard Min-Max normalization to the interval .
- Rescale the median score () to the target interval using the generalized min-max formulation.
Min-Max Normalization to : Identify extrema:
Apply :
- :
- :
- :
- :
- :
Generalized Min-Max Rescaling to for score : The generalized formula is:
Here and , so the range multiplier is :
- Normalized to : .
- Rescaled to : the score maps to .
A dataset of graduate student ages is collected:
- Compute the population mean and standard deviation .
- Calculate the -score standardized values for the youngest student () and oldest student ().
- Normalize the oldest student's age () using decimal scaling.
Compute mean and population standard deviation:
Sum of squared deviations:
Population variance and standard deviation ():
(Using sample standard deviation )
Compute -scores:
- Youngest student ():
(With sample : )
- Oldest student ():
(With sample : )
Decimal scaling for : Maximum absolute value is . Find :
and . The -score for age is (or with ); for age it is (or with ). Under decimal scaling (), age normalizes to .
Compare Ordinal (Label) Encoding with One-Hot Encoding. Explain why applying label encoding to nominal features can mislead linear algorithms, and define the dummy variable trap.
Comparison of Encoding Mechanisms:
- Ordinal (Label) Encoding: Maps distinct categories to sequential integers (). It assumes a meaningful natural ordering among categories.
- One-Hot Encoding: Expands a category into separate binary indicator columns (), where each column signifies the presence of one category. It assumes no ordering.
Why Label Encoding misleads linear algorithms on nominal data: If nominal categories without natural ordering (e.g., Country: Nepal , Japan , Germany ) are assigned integers, a linear model enforces an arithmetic relationship:
The model assumes countries are equally spaced along a single numerical spectrum, which introduces false ordering and arbitrary geometric constraints.
The Dummy Variable Trap: When a categorical feature with categories is converted to binary columns in a linear regression model that includes an intercept term , the sum of the columns equals the intercept column:
This creates perfect multicollinearity, making the feature matrix linearly dependent. The Gram matrix becomes singular and non-invertible. To avoid this trap, one binary column is omitted ( columns used), treating the omitted category as the reference baseline.
Label encoding assigns integers to categories and is valid only for ordered attributes; applying it to nominal attributes introduces artificial ordering. One-hot encoding creates binary columns without assuming order. The dummy variable trap occurs when all columns are included in a regression model with an intercept, causing perfect multicollinearity. It is resolved by dropping one column ( retained).
A financial dataset contains four heavily skewed customer annual income values:
- Calculate the natural logarithm for the minimum and maximum income values.
- Evaluate how the logarithmic transformation compresses the dynamic range between the extremes.
Compute natural logarithms:
- Minimum income ():
- Maximum income ():
Quantify dynamic range compression:
- Original ratio:
The highest earner has the income of the lowest earner.
- Log-transformed difference:
The multiplicative factor of is compressed into an additive difference of only units. This stabilizes the variance and reduces the leverage of extreme income earners in regression models.
and . The logarithmic transformation compresses a -fold multiplicative difference in raw scale into an additive difference of only , dampening skewness and reducing outlier influence.
A data pipeline needs to discretize a continuous performance score attribute using Equal-Width Binning with bins. The observed scores are:
- Calculate the bin width.
- Define the interval boundaries for the three bins.
- Assign each score to its respective bin.
Calculate bin width: Identify the minimum and maximum scores:
Define interval boundaries: Construct contiguous intervals:
- Bin 1:
- Bin 2:
- Bin 3: (closed at upper boundary to include )
Assign scores to bins:
- Bin 1 (): (Count = 4)
- Bin 2 (): (Count = 3)
- Bin 3 (): (Count = 3)
Bin width is . The intervals are with 4 observations, with 3 observations, and with 3 observations.