Types of Analytics & Exploratory Data Analysis
Open in the Padhaiko app4.1 Data Analytics: Descriptive, Diagnostic, Predictive & Prescriptive Analytics
Data analytics is the discipline of examining, transforming, and modeling raw data to discover actionable insights, identify hidden patterns, and draw conclusions that guide decision-making. In modern organizations, analytics transforms decision-making from subjective intuition or "gut feeling" into empirical, evidence-based strategy.
Data Analytics & Strategic Importance
Data Analytics is the science and engineering of analyzing raw datasets to deduce systematic conclusions, optimize operational efficiency, mitigate financial risks, and uncover strategic opportunities.
Transitioning from intuition-driven to evidence-based decision-making yields three critical organizational advantages:
- Performance Optimization: Quantifies operational workflows to eliminate bottlenecks and reduce waste.
- Cost Reduction: Pinpoints expenditure inefficiencies and automates resource allocation.
- Strategic Opportunity Discovery: Identifies unmet market demands and shifts customer acquisition models from reactive to proactive.
The Four Paradigms of Analytics
Analytics problems are classified into four distinct maturity levels based on their temporal orientation (past vs. future) and objective (observation vs. action).
The Four Types of Analytics
- Descriptive Analytics ("What happened?"):
- Synthesizes historical data to summarize past performance and present state.
- Leverages descriptive summary statistics, data aggregation, KPIs, and tabular reporting dashboards.
- Example: Generating a monthly sales report showing a drop in regional revenue.
- Diagnostic Analytics ("Why did it happen?"):
- Investigates root causes of historical trends, anomalies, or performance shifts discovered during descriptive analysis.
- Employs drill-down queries, correlation analysis, data mining, and causal discovery techniques.
- Example: Investigating customer churn to determine that revenue dropped because of frequent mobile app crashes during checkout.
- Predictive Analytics ("What might happen?"):
- Projects future probabilities, trends, and behavioral patterns by fitting statistical models and machine learning algorithms on historical records.
- Leverages regression models, time-series forecasting, decision trees, and probabilistic classifiers.
- Example: Forecasting inventory demand for the upcoming quarter based on seasonal purchasing patterns and macroeconomic indicators.
- Prescriptive Analytics ("What should we do?"):
- Recommends optimal action plans and decision policies to maximize desirable outcomes or mitigate forecasted risks.
- Utilizes mathematical optimization (linear programming), simulation models, multi-criteria decision analysis, and reinforcement learning.
- Example: Automatically adjusting dynamic pricing algorithms and offering targeted discount coupons to high-risk churn customers.
| Paradigm | Core Question | Primary Objective | Key Techniques | Real-World Example |
|---|---|---|---|---|
| Descriptive | What happened? | Summarize historical facts | Aggregation, KPIs, pivot tables, counts | Monthly regional sales summary report |
| Diagnostic | Why did it happen? | Identify root causes & anomalies | Drill-down queries, correlation, clustering | Analyzing why sales dropped in a specific region |
| Predictive | What might happen? | Forecast future likelihoods | Linear regression, ARIMA, classification trees | Estimating loan default probabilities for applicants |
| Prescriptive | What should we do? | Determine the optimal action | Linear programming, simulation, heuristic rules | Recommending credit limits and interest rates |
4.2 Exploratory Data Analysis (EDA)
Exploratory Data Analysis (EDA) is an investigative philosophy and rigorous methodology pioneered by the statistician John Tukey. It emphasizes summarizing the primary empirical characteristics of a dataset, uncovering underlying structures, testing foundational assumptions, and exposing anomalies before committing to formal statistical hypothesis testing or complex machine learning modeling.
Exploratory Data Analysis (EDA)
Exploratory Data Analysis (EDA) is the critical initial phase of data analysis that combines descriptive statistics and graphical visualization techniques to understand the distribution, internal structure, and validity of a dataset without imposing rigid model assumptions.
Core Objectives of EDA:
- Summarize Core Characteristics: Calculate measures of central tendency () and dispersion ().
- Uncover Patterns & Correlations: Detect linear and non-linear relationships, cluster structures, and feature dependencies.
- Detect Anomalies & Outliers: Identify recording errors, sensor malfunctions, or genuine extreme observations that distort estimators.
- Test Statistical Assumptions: Verify whether data satisfies distributional prerequisites (such as normality, homoscedasticity, or absence of multicollinearity) assumed by predictive algorithms.
- Direct Data Preprocessing: Guide decisions regarding imputation of missing entries, categorical encoding, feature scaling, and transformations.
Quantitative Measures & The Five-Number Summary
While visual plots expose global shape, quantitative summary metrics provide exact mathematical benchmarks for distributions.
The Five-Number Summary
The Five-Number Summary is a robust non-parametric set of order statistics that summarizes the location, spread, and range of a continuous distribution without assuming normality:
Given a sample of observations arranged in non-decreasing order:
- Minimum (): The smallest observed value, .
- First Quartile ( / 25th Percentile): The median of the lower of the data, separating the lowest of observations from the upper .
- Second Quartile / Median ( / 50th Percentile): The central value separating the upper half from the lower half:
- Third Quartile ( / 75th Percentile): The median of the upper of the data, separating the lower of observations from the top .
- Maximum (): The largest observed value, .
Interquartile Range (IQR) & John Tukey's Outlier Fences
In exploratory data analysis, identifying outliers is essential because extreme values can severely bias ordinary least squares regression, inflate variance, and distort machine learning models.
Tukey's 1.5 × IQR Outlier Rule
Let and be the first and third quartiles of a dataset. The Interquartile Range () is defined as:
Tukey's inner fences define the boundaries for regular observations:
An observation is classified as a statistical outlier (or anomaly) if:
When outliers are detected, the whiskers of a standard box-plot extend only to the most extreme data points that fall strictly within , while outliers beyond the fences are plotted individually as isolated dots or asterisks.
Anatomy of a Box-and-Whisker Plot
A box-plot (box-and-whisker plot) graphically renders the Five-Number Summary and outlier boundaries in a single compact visual display:
- Central Box: Spans from to , containing the central of the data. Its width is exactly .
- Median Line: A solid line drawn across the interior of the box at . If the median line is not centered within the box, the data exhibits skewness.
- Whiskers: Straight lines extending outwards from the box edges to the smallest and largest observed data points that remain within the fences .
- Outlier Points: Individual markers plotted beyond the whisker tips for values or .
| Distribution Shape | Relative Positions | Box-Plot Visual Characteristic |
|---|---|---|
| Symmetric | Median line bisects the box evenly; left and right whiskers have equal length. | |
| Positively Skewed (Right-Skewed) | Median line sits near the bottom (); right/upper whisker is substantially longer; outliers occur at the upper end. | |
| Negatively Skewed (Left-Skewed) | Median line sits near the top (); left/lower whisker is substantially longer; outliers occur at the lower end. |
Worked Examples
Five-Number Summary and Outlier Detection (Even Sample Size)
A cricket team scored the following runs across 12 consecutive league matches:
Compute the Five-Number Summary, evaluate the Interquartile Range (), establish Tukey's outlier fences, and determine whether any match score is an outlier. Describe the box-plot.
Arranging observations in ascending order: Sorting the match scores in non-decreasing order:
Computing the Median (): Because is an even integer, the median is the arithmetic mean of the two central order statistics ( and ):
Evaluating Quartiles and : Dividing the ordered data into two equal halves of 6 observations each:
- Lower Half: The first quartile is the median of this lower subset (average of the 3rd and 4th values):
- Upper Half: The third quartile is the median of this upper subset (average of the 9th and 10th overall values):
Calculating Interquartile Range and Outlier Fences:
Applying Tukey's formula:
Identifying Outliers and Whiskers: Comparing all observations against the valid interval :
- Smallest observation is (no lower outlier).
- Largest observation is . Therefore, the score of is a high outlier.
- The largest non-outlier observation is .
Hence, the box-plot whiskers extend from the minimum non-outlier value () to the maximum non-outlier value (), with the point plotted as an isolated outlier.
The box spans from to with the median at . The lower whisker ends at , the upper whisker ends at , and is plotted as an individual outlier.
Five-Number Summary and Outlier Analysis (Odd Sample Size)
A cloud engineer monitors API response latency across 11 network requests, recording the following times (in milliseconds):
Compute the Five-Number Summary using the standard method of excluding the median for an odd sample size. Determine the outlier boundaries and identify any anomalous latency values.
Sorting the observations: Arranging the latency readings in ascending order:
Finding the Median (): Since is odd, the median is the unique middle observation at position :
Finding and (excluding the median):
- Lower Subset ( values strictly below median): The median of these numbers is the 3rd value:
- Upper Subset ( values strictly above median): The median of these numbers is the middle (9th overall) value:
Calculating IQR and Tukey's Fences:
Fences:
Outlier Identification: Evaluating all 11 observations against :
- Smallest value is (within fence).
- The highest latency of satisfies . Hence, is a high latency outlier.
- The largest non-outlier observation is .
Whiskers extend to on the lower end and on the upper end, with flagged as an outlier.
The box spans from to with the median line at . Whiskers extend to and , with plotted as an outlier.
Exercises
Exercises on Types of Analytics & Exploratory Data Analysis
Define data analytics and explain its importance in organizational decision-making. How does evidence-based decision-making differ from intuition-based strategy? [Assignment-4, Q1]
Definition of Data Analytics: Data analytics is the science, systematic process, and technology of inspecting, cleaning, transforming, and modeling raw data to draw meaningful conclusions, identify patterns, and support informed operational and strategic decision-making.
Importance in Decision-Making:
- Shifting from Intuition to Empirical Evidence: Replaces unverified human guesses, subjective biases, and anecdotal assumptions with statistically grounded facts.
- Operational Efficiency & Cost Minimization: Tracks internal workflows, exposes resource waste, and automates repetitive operational functions.
- Risk Mitigation: Analyzes historical loss indicators to proactively forecast credit defaults, fraudulent transactions, or system failures.
- Customer Centricity: Uncovers nuanced user behavioral patterns to tailor product recommendations and optimize retention.
Evidence-Based vs Intuition-Based Strategy:
- Intuition-based strategy relies on an executive's personal experience, gut feeling, and qualitative instincts. While fast, it is prone to cognitive biases (e.g., confirmation bias, overconfidence), cannot scale over complex multivariate data, and provides no mathematical risk boundaries.
- Evidence-based strategy relies on verified data pipelines, statistical hypothesis testing, and machine learning models. It provides quantifiable confidence intervals, allows objective A/B testing of corporate policies, and establishes repeatable, measurable outcomes.
Data analytics is the discipline of extracting actionable conclusions from raw data. It replaces subjective, bias-prone intuition with reproducible empirical evidence, enabling organizations to optimize operational efficiency, minimize financial risks, and make objective, data-backed decisions.
Differentiate between descriptive, diagnostic, predictive, and prescriptive analytics. Provide a concrete real-world engineering or business example for each category. [Assignment-4, Q2]
Defining the Four Analytics Paradigms:
- Descriptive Analytics: Analyzes past historical data to answer "What happened?" through summary metrics, counts, and reporting dashboards.
- Diagnostic Analytics: Drills down into historical anomalies to answer "Why did it happen?" by uncovering underlying causal factors and relationships.
- Predictive Analytics: Fits statistical and machine learning models to answer "What might happen?" by estimating future probabilities and trajectory forecasts.
- Prescriptive Analytics: Applies mathematical optimization and decision simulation to answer "What should we do?" by recommending specific optimal policies.
Concrete Industry Examples (Fleet Management & Smart Transportation):
- Descriptive: An engineering dashboard reports that of the municipal bus fleet experienced engine overheating incidents during June.
- Diagnostic: Telemetry data mining reveals that the overheated buses shared a specific coolant pump model that clogged whenever ambient operating temperatures exceeded .
- Predictive: A gradient boosting model analyzes current weather forecasts, route elevations, and vehicle telemetry to predict which 15 specific buses are likely to overheat next week.
- Prescriptive: An optimization algorithm automatically reschedules maintenance for those 15 buses before next Monday and reassigns cooler evening routes to vehicles awaiting servicing.
Descriptive analytics reports past facts (What happened?), diagnostic analytics isolates root causes (Why did it happen?), predictive analytics forecasts future probabilities (What might happen?), and prescriptive analytics dictates optimal action policies (What should we do?).
Define exploratory data analysis (EDA) and explain its primary objectives. How does EDA help detect outliers and uncover underlying data distributions? [Assignment-4, Q3]
Definition and Philosophy of EDA: Exploratory Data Analysis (EDA) is the foundational data science process of investigating datasets through descriptive statistics and graphical summaries. It aims to reveal internal structure, detect anomalies, check validity, and formulate hypotheses without imposing rigid parametric model constraints.
Primary Objectives of EDA:
- Uncovering data structure, distribution shape, and scale.
- Detecting errors, missing values, and statistical outliers.
- Testing assumptions required for subsequent modeling (e.g., normality, homoscedasticity).
- Identifying relationships, multicollinearity, and candidate predictors for feature engineering.
Detecting Outliers and Distributions:
- Distribution Discovery: Using histograms, kernel density estimation (KDE), and probability plots (Q-Q plots), EDA reveals whether a feature is unimodal, bimodal, symmetric, or heavily skewed.
- Outlier Detection: Through box-plots, scatter plots, and Tukey's fences, EDA identifies observations located far beyond expected probabilistic boundaries. This distinguishes data entry errors (e.g., an age of 250) from legitimate, high-leverage extreme cases.
EDA is the initial exploration of data through summary statistics and visual graphics to discover patterns and test assumptions. It identifies distribution shapes (skewness, modality) using histograms and exposes anomalies using Tukey's IQR fences and box-plots.
The following data represents the number of daily customer transactions recorded at a retail store across 12 consecutive operating days:
Derive the Five-Number Summary, calculate the Interquartile Range (), establish Tukey's outlier fences, and verify whether any day contains an outlying transaction count. Describe the properly labeled box-plot. [Assignment-4, Notebook p093 Q7b]
Sorting the 12 observations: Arranging the transaction counts in ascending order:
Total observations: .
Calculating Median (): With (even), the median is the average of the 6th and 7th values:
Calculating Quartiles and : Dividing the 12 sorted items into lower and upper halves of 6 numbers each:
- Lower Half: The median of the lower half is the average of its 3rd and 4th values:
- Upper Half: The median of the upper half is the average of its 3rd and 4th values (9th and 10th overall):
Evaluating IQR and Outlier Fences:
Fences:
Outlier Verification & Box-Plot Construction: The sample minimum is and the sample maximum is . Because all 12 observations lie strictly within , there are no outliers in the dataset.
The box-plot consists of:
- A lower whisker extending from down to .
- A central box spanning from to , with the internal median line at .
- An upper whisker extending from up to .
The box-plot shows a box from to with the median line at , and whiskers extending to the sample extremes at and .
Here is a sample of 10 graduate student ages from a university cohort:
- Women: 35, 29, 32, 23, 36
- Men: 28, 25, 29, 24, 23
Using this data, calculate the Five-Number Summary, evaluate the outlier fences, and construct the labeled box-plot showing all calculations. [Past exam 2081 Chaitra, Q5b]
Ordering the combined dataset: Combining all student ages and arranging them in ascending order:
Calculating the Five-Number Summary:
- Minimum ():
- Median (): Average of the 5th and 6th values:
- First Quartile (): Median of the lower 5 values :
- Third Quartile (): Median of the upper 5 values :
- Maximum ():
Evaluating IQR and Outlier Fences:
Tukey's fences:
Since all 10 ages fall in the interval , there are no outliers.
Box-Plot Representation:
- Left whisker extends from to .
- Central box spans from to with the median divider at .
- Right whisker extends from to .
The box-plot displays a box spanning from to with a vertical line at the median , and whiskers reaching and .
Explain how the relationship between the mean, median, and mode reflects distribution skewness. How is skewness visually diagnosed in a box-plot?
Mathematical Relationship between Mean, Median, and Mode:
- Symmetric Distribution:
Observations are evenly balanced around the central peak.
- Positively Skewed (Right-Skewed) Distribution:
A long tail of large positive values pulls the sensitive arithmetic mean to the right, whereas the median remains robust and resistant.
- Negatively Skewed (Left-Skewed) Distribution:
A long tail of unusually small negative values drags the mean to the left below the median.
Visual Diagnosis in a Box-Plot:
- Position of the Median Line: If the median line inside the box is closer to (lower quartile), the distribution is positively skewed; if closer to (upper quartile), it is negatively skewed.
- Whisker Lengths: A significantly longer upper whisker indicates positive skewness; a significantly longer lower whisker indicates negative skewness.
- Outlier Orientation: Outlier dots clustered solely beyond the upper whisker indicate strong positive skewness.
In symmetric distributions, . In right-skewed distributions, , reflected in a box-plot by a median line closer to and a longer upper whisker. In left-skewed distributions, , shown by a median line closer to and a longer lower whisker.