Past exam question: 2081 Chaitra (Regular), Q2(b)(ii)
Open in the Padhaiko appConsidering the test data size as 50,000, a typical classification model experimentation results are as shown on the confusion matrix.
| PREDICTED CLASS: Yes | PREDICTED CLASS: No | |
|---|---|---|
| ACTUAL CLASS: Yes | 1000 | 1000 |
| ACTUAL CLASS: No | 1000 | 47000 |
Explain the difference on resulted accuracy and -score value that you have calculated.
Solution
The Accuracy Paradox Under Severe Class Imbalance: The confusion matrix exhibits extreme class imbalance:
- Total actual negative instances ("No"): ( of dataset).
- Total actual positive instances ("Yes"): ( of dataset).
Because the majority class overwhelmingly dominates the sample, the high accuracy of is almost entirely driven by the True Negatives. A trivial, baseline model that blindly predicts "No" for all inputs would achieve an identical accuracy (), while providing zero predictive utility for detecting the positive class. Thus, accuracy creates a false impression of superior performance.
How F1-Score Reflects Actual Minority-Class Performance: Unlike accuracy, the -score completely disregards True Negatives () and focuses strictly on the positive class ("Yes"):
- The model fails to detect of actual positive cases (, as ).
- Half of all positive alerts generated by the model are false alarms (, as ).
The -score penalizes extreme imbalances between precision and recall via their harmonic mean, producing . This accurately exposes that the model is only moderately effective () on the minority class of interest, demonstrating why -score is far more informative than accuracy in imbalanced classification settings.
The large discrepancy (Accuracy vs. ) is caused by severe class imbalance ( negative vs. positive). Accuracy is dominated by the large number of True Negatives (), masking poor positive-class detection, whereas the -score ignores True Negatives and evaluates precision and recall on the positive minority class, correctly revealing moderate performance ().