Past exam question: 2081 Chaitra (Regular), Q4(b)
Open in the Padhaiko appHow do you handle missing data during data preprocessing? Briefly explain two methods illustrating with a small sample data example.
Solution
Overview of Missing Data in Preprocessing: Real-world datasets frequently contain missing values (represented as NaN, null, or blank cells) caused by equipment sensor failure, survey non-responses, or transmission drops. Because most machine learning algorithms cannot natively process missing numbers, handling them during preprocessing is critical to prevent pipeline failure and biased estimations.
The two most fundamental strategies are:
- Listwise (Row) Deletion: Discarding any sample record containing one or more missing feature values.
- Mean / Median Imputation: Replacing missing values with a central tendency statistic computed from the available observed values of that feature.
Sample Dataset Illustration: Consider a small academic dataset recording student study hours and exam scores, where Student has a missing exam score:
| Student ID | Study Hours () | Exam Score () |
|---|---|---|
| 1 | 2 | 55 |
| 2 | 4 | 70 |
| 3 | 5 | NaN |
| 4 | 7 | 85 |
| 5 | 8 | 90 |
Method 1: Listwise (Row) Deletion:
- Mechanism: The preprocessor scans each row in the dataset and removes entire records where any field is missing.
- Application to Sample: Student contains a
NaNin Exam Score, so the entire row corresponding to Student is discarded. - Resulting Dataset:
Student ID Study Hours () Exam Score () 1 2 55 2 4 70 4 7 85 5 8 90 - Trade-offs: Simple to implement and unbiased if data is Missing Completely at Random (MCAR). However, it permanently discards valid observed data in other attributes (such as Student 's study hours) and significantly reduces sample size when missingness is widespread.
Method 2: Mean Imputation:
- Mechanism: The missing attribute value is replaced with the arithmetic mean of all available observed values for that feature.
- Application to Sample: Calculate the mean of the observed exam scores :
The missing score for Student is imputed with .
- Resulting Dataset:
Student ID Study Hours () Exam Score () 1 2 55 2 4 70 3 5 75 4 7 85 5 8 90 - Trade-offs: Retains all sample records and preserves the overall sample mean. However, it artificially shrinks the feature's variance and distorts correlations with other variables.
Missing data is primarily handled by:
- Listwise Deletion: Removing any row containing missing values. In the sample, Student (
NaN) is eliminated, leaving 4 complete records. - Mean Imputation: Replacing the missing entry with the average of observed entries. In the sample, the observed mean score is , imputing Student 's score as while retaining all 5 records.