Past exam question: 2081 Chaitra (Regular), Q7(ii)
Open in the Padhaiko appWrite brief notes on: (Any Three)
(ii) Feature selection
Solution
Definition and Objectives of Feature Selection: Feature selection is the process of identifying and selecting a subset of the most relevant, informative features (variables, predictors) from the original feature space to use in constructing a machine learning model.
Unlike feature extraction (e.g., PCA), which projects data into transformed mathematical dimensions, feature selection retains the original features in their native representation, preserving interpretability.
Key objectives:
- Mitigating the Curse of Dimensionality: Reducing feature count prevents data sparsity in high-dimensional spaces.
- Preventing Overfitting: Eliminating noisy or irrelevant features improves model generalization on unseen test data.
- Improving Computational Efficiency: Drastically speeds up training and inference times while reducing memory footprints.
- Enhancing Model Interpretability: Fewer, semantically meaningful features make model behavior transparent to domain experts.
Three Primary Methodological Paradigms: Feature selection techniques are broadly categorized into three core families:
- Filter Methods:
- Approach: Evaluates the intrinsic statistical properties of each feature independently of any machine learning model. Features are ranked by statistical significance, and those exceeding a threshold are kept.
- Common Metrics: Pearson correlation coefficient (for continuous linear relationships), Spearman rank correlation, Chi-Square () test of independence (for categorical variables), ANOVA -test, and Mutual Information.
- Trade-off: Computationally fast and scalable to massive datasets, but ignores complex multivariate feature interactions.
- Wrapper Methods:
- Approach: Treats a specific machine learning model as an evaluation harness, iteratively testing various subsets of features and scoring them on cross-validation performance.
- Algorithms: Forward Feature Selection (starting empty and greedily adding features), Backward Feature Elimination (starting with all features and removing the least impactful), and Recursive Feature Elimination (RFE).
- Trade-off: Optimizes performance specifically for the chosen model and captures feature interactions, but is computationally expensive and prone to overfitting on small datasets.
- Embedded (Intrinsic) Methods:
- Approach: Integrates feature selection directly into the mathematical optimization routine during model training.
- Algorithms: LASSO Regression ( regularization, which forces irrelevant coefficient weights to exactly zero via ), Ridge regression with thresholding, and Tree-based algorithms (Decision Trees, Random Forests) that rank features using Gini Impurity reduction or Mean Decrease in Impurity (MDI).
- Trade-off: Strikes an optimal balance between the low computational cost of filters and the high accuracy of wrappers.
Feature selection is the technique of isolating the most relevant subset of original features to combat overfitting, decrease training latency, and enhance interpretability without transforming the feature space. Its three main methodological paradigms are Filter methods (model-agnostic statistical ranking such as ANOVA or mutual information), Wrapper methods (iterative model-based searches such as RFE), and Embedded methods (built-in algorithmic selection such as LASSO regularization and tree-based importance).