Applied Statistics and Exploratory Analysis
Open in the Padhaiko app4.1 Statistical measures: Correlation, covariance, skewness, kurtosis
Descriptive statistics summarize center, spread, linear association, and distributional shape before fitting predictive models.
Covariance and Correlation
Covariance vs. Correlation
- Sample Covariance (
cov(X, Y)): Measures the joint linear co-movement of two variablesXandYin their original units:cov(X, Y) = sum((x_i - x_bar) * (y_i - y_bar)) / (n - 1)Positive covariance meansXandYrise together; negative covariance means one tends to fall as the other rises; magnitude depends on variable scale. - Pearson Correlation Coefficient (
r): Scale-free, dimensionless linear association bounded to[-1, +1]:r = cov(X, Y) / (s_x * s_y)wheres_xands_yare sample standard deviations.r = +1is perfect positive linear fit,r = -1is perfect negative linear fit, andr = 0indicates no linear correlation (though non-linear association can still exist). - Spearman Rank Correlation (
rho): Applies Pearson's formula to rank-transformed values, capturing monotonic non-linear relationships and resisting outliers.
| Aspect | Pearson Correlation (r) | Spearman Rank Correlation (rho) |
|---|---|---|
| Relationship Measured | Strictly linear association on raw values | Monotonic association on sorted ranks |
| Scale & Distribution | Sensitive to extreme outliers; assumes interval/ratio scale | Robust to outliers and skewed monotonic curves |
| Data Type Support | Continuous numeric columns | Numeric or ordinal ranked data |
Shape Measures: Skewness and Kurtosis
- Skewness (Asymmetry):
skewness = 0: Symmetric distribution (e.g., Normal distribution;mean = median = mode).skewness > 0(Positive / Right-skewed): Long right tail pullingmean > median(e.g., household income, session duration).skewness < 0(Negative / Left-skewed): Long left tail pullingmean < median(e.g., easy exam scores).
- Kurtosis (Tail Heaviness / Peakedness):
- Standard normal distribution has Pearson kurtosis =
3.0and excess kurtosis =kurtosis - 3 = 0.0(mesokurtic). - Leptokurtic (
excess kurtosis > 0): Heavy tails and sharper peak (more extreme outliers, e.g., financial returns). - Platykurtic (
excess kurtosis < 0): Light tails and flatter peak (e.g., uniform distribution, excess kurtosis-1.2).
- Standard normal distribution has Pearson kurtosis =
import pandas as pd
from scipy import stats
s = pd.Series([10, 12, 14, 15, 18, 45])
print("Skewness:", round(s.skew(), 3)) # positive right skew
print("Excess kurtosis:", round(s.kurt(), 3)) # pandas .kurt() returns Fisher excess kurtosis4.2 Probability review, sampling, and hypothesis testing
Inferential statistics uses a finite sample to reason rigorously about an underlying population parameter under uncertainty.
Probability Essentials and Sampling
- Random Variables & Distributions: Discrete (Binomial, Poisson) and continuous (Normal
N(mu, sigma^2), Student'st, Chi-square) models describe expected data-generating behavior. - Sampling Techniques:
- Simple Random Sampling: Every member has equal selection probability (
df.sample(n=100, random_state=42)). - Stratified Sampling: Preserves class/subgroup proportions (
train_test_split(..., stratify=y)), essential for imbalanced classification targets. - Systematic & Cluster Sampling: Selecting every
k-th record or sampling whole geographic/organizational clusters.
- Simple Random Sampling: Every member has equal selection probability (
- Central Limit Theorem (CLT): For sample size
n >= 30, the sampling distribution of the sample meanx_barapproaches a Normal distributionN(mu, sigma^2 / n)with Standard ErrorSE = s / sqrt(n), even if the raw population is skewed.
Hypothesis Testing Framework
Null Hypothesis, p-value, and Decision Errors
- Hypotheses: State the Null Hypothesis (
H0) (no effect / no difference) and Alternative Hypothesis (H1orHa) (two-sided!=or one-sided>,<). - Test Statistic & p-value: Compute a test statistic (
t,z,F, orchi^2) from the sample. The p-value is the probability of observing a test statistic at least as extreme as the sample's assumingH0is true. - Decision Rule: Choose significance level
alpha(commonly0.05). Ifp-value <= alpha, rejectH0(statistically significant); otherwise, fail to rejectH0. - Error Types:
- Type I Error (False Positive): Rejecting
H0whenH0is actually true (probability =alpha). - Type II Error (False Negative): Failing to reject
H0whenH1is true (probability =beta; statistical power =1 - beta).
- Type I Error (False Positive): Rejecting
| Test | SciPy Function | Purpose / Data Requirement |
|---|---|---|
| One-Sample / Independent t-test | stats.ttest_1samp, stats.ttest_ind | Compare means of 1 or 2 continuous samples (equal_var=False for Welch's t-test) |
| Paired t-test | stats.ttest_rel(before, after) | Compare before/after measurements on the same subjects |
| One-Way ANOVA | stats.f_oneway(g1, g2, g3) | Test whether means differ across 3+ independent groups |
| Chi-Square Independence Test | stats.chi2_contingency(table) | Test association between two categorical variables in a contingency table |
| Mann–Whitney U Test | stats.mannwhitneyu(x, y) | Non-parametric alternative to independent t-test for skewed/ordinal data |
4.3 Regression and trend analysis using stats models
While scikit-learn emphasizes predictive accuracy, Python's statsmodels library provides classical statistical inference—coefficient standard errors, t-statistics, p-values, confidence intervals, and diagnostic tests.
Ordinary Least Squares (OLS) with statsmodels
For linear model y = beta_0 + beta_1 * x_1 + ... + beta_k * x_k + epsilon:
- Always call
X = sm.add_constant(X)when using matrix APIsm.OLS(y, X), becausestatsmodelsdoes not add an intercept automatically (unlike formula APIsmf.ols("y ~ x1 + x2", data=df)). - Calling
model = sm.OLS(y, X).fit()andmodel.summary()outputs key diagnostics:
| Metric | Interpretation | What to Watch For |
|---|---|---|
| R-squared / Adj. R-squared | Fraction of variance in y explained by predictors | Adjusted R-squared penalizes adding useless features |
| F-statistic (Prob F-stat) | Joint test that all slope coefficients equal zero | p < 0.05 indicates the overall model beats a constant mean |
| Coef and its p-value | Estimated slope and individual significance test (H0: beta_j = 0) | Predictors with p > 0.05 may not be statistically significant |
| Durbin–Watson | Autocorrelation in residuals (0 to 4; ~2.0 = none) | Values far below 1.5 or above 2.5 signal serial correlation |
| Condition Number | Detects multicollinearity among predictors | Values > 30 suggest correlated features inflating standard errors |
import pandas as pd
import statsmodels.api as sm
df = pd.DataFrame({"ad_spend": [1, 2, 3, 4, 5], "sales": [3, 5, 7, 9, 11]})
X = sm.add_constant(df[["ad_spend"]])
res = sm.OLS(df["sales"], X).fit()
print(res.params) # const=1.0, ad_spend=2.0
print("R2:", res.rsquared)Trend Analysis and Time-Series Decomposition
Trend analysis isolates long-term direction (y_t = Trend + Seasonal + Residual) using:
- Moving averages (
df["sales"].rolling(12).mean()), - Linear/polynomial time-index regression (
OLSont = 0, 1, 2, ...), or statsmodels.tsa.seasonal.seasonal_decompose(series, model="additive", period=12)to inspect underlying trend and seasonality before forecasting.
4.4 Exploratory data analysis (EDA) using descriptive and inferential methods
Exploratory Data Analysis (EDA) systematically inspects data quality, distributions, anomalies, and relationships before choosing transformations or models.
| Aspect | Descriptive EDA | Inferential EDA |
|---|---|---|
| Goal | Summarize what is in the sample (shape, center, spread, missingness) | Test whether observed patterns generalize beyond random sampling noise |
| Methods | df.describe(), quantiles, IQR, skewness, correlation matrix, group tables, histograms, box plots | Confidence intervals (95% CI), t-tests, ANOVA, Chi-square tests, OLS p-values |
| Key Pitfall | Mistaking sample noise or Simpson's paradox for real effects | Over-interpreting tiny p-values in huge datasets without checking effect size |
Systematic EDA Checklist
- Structural Audit: Inspect
.shape,.dtypes,.head(),.isna().sum(), and.duplicated().sum(). - Univariate Analysis: Compute mean, median, std, IQR (
Q3 - Q1), min/max, and outlier bounds via Tukey's 1.5 * IQR rule ([Q1 - 1.5*IQR, Q3 + 1.5*IQR]) for numeric features;.value_counts(normalize=True)for categoricals. - Bivariate & Multivariate Analysis: Cross-tabulations (
pd.crosstab), grouped aggregations (df.groupby("target").mean()), Pearson/Spearman heatmaps, and inferential tests (t-test/ANOVA for numeric vs. category, Chi-square for category vs. category). - Target Leakage & Domain Sanity Checks: Flag impossible values (negative age, future dates, post-outcome columns).
4.5 Automation of EDA workflows using Python
Writing repetitive describe(), isna(), skewness, and outlier loops manually across dozens of projects wastes time. Automating EDA builds repeatable, consistent diagnostic reports in seconds.
Automated EDA Approaches in Python
- Custom Reusable Python EDA Function / Module: Combine Pandas,
scipy.stats, and summary dictionaries to return structured audit tables (dtype, nulls,% missing, unique count, mean, median, skew, outlier count) that integrate directly into CI/CD or notebooks. - Dedicated Auto-EDA Libraries:
ydata-profiling(formerlypandas-profiling): Generates rich standalone HTML reports (ProfileReport(df).to_file("eda.html")) covering interactions, correlations, missingness matrices, and duplicate alerts.sweetviz/dtale: Compare train vs. test feature drift side by side or launch interactive browser grid explorers.
- Automated Guardrails in Pipelines: Pair automated EDA summaries with threshold checks (e.g., abort training if missingness exceeds 20% or target class ratio shifts sharply).
import pandas as pd
def auto_eda_summary(df: pd.DataFrame) -> pd.DataFrame:
rows = []
n = len(df)
for col in df.columns:
s = df[col]
missing = int(s.isna().sum())
rec = {
"column": col,
"dtype": str(s.dtype),
"missing": missing,
"missing_pct": round((missing / n) * 100, 2) if n else 0.0,
"n_unique": int(s.nunique(dropna=True)),
}
if pd.api.types.is_numeric_dtype(s) and not pd.api.types.is_bool_dtype(s):
rec["mean"] = round(float(s.mean()), 3)
rec["median"] = round(float(s.median()), 3)
rec["skew"] = round(float(s.skew()), 3)
rows.append(rec)
return pd.DataFrame(rows)Worked Examples
4.1
Given paired observations X = [1, 2, 3, 4, 5] and Y = [2, 4, 5, 9, 10], calculate manually (and verify with Python logic):
- Sample means
x_barandy_bar, - Sample covariance
cov(X, Y), - Sample standard deviations
s_xands_y, and Pearson correlationr.
Sample Means:
x_bar = (1 + 2 + 3 + 4 + 5) / 5 = 15 / 5 = 3.0y_bar = (2 + 4 + 5 + 9 + 10) / 5 = 30 / 5 = 6.0
Deviations, Sample Covariance, and Standard Deviations (n - 1 = 4):
x_i | y_i | dx = x_i - 3 | dy = y_i - 6 | dx * dy | dx^2 | dy^2 |
|---|---|---|---|---|---|---|
| 1 | 2 | -2 | -4 | 8 | 4 | 16 |
| 2 | 4 | -1 | -2 | 2 | 1 | 4 |
| 3 | 5 | 0 | -1 | 0 | 0 | 1 |
| 4 | 9 | 1 | 3 | 3 | 1 | 9 |
| 5 | 10 | 2 | 4 | 8 | 4 | 16 |
sum(dx * dy) = 8 + 2 + 0 + 3 + 8 = 21→cov(X, Y) = 21 / 4 = 5.25.sum(dx^2) = 4 + 1 + 0 + 1 + 4 = 10→s_x = sqrt(10 / 4) = sqrt(2.5) ≈ 1.5811.sum(dy^2) = 16 + 4 + 1 + 9 + 16 = 46→s_y = sqrt(46 / 4) = sqrt(11.5) ≈ 3.3912.
Pearson Correlation Coefficient r:
x_bar = 3.0, y_bar = 6.0, cov(X, Y) = 5.25, s_x ≈ 1.5811, s_y ≈ 3.3912, and Pearson correlation r ≈ 0.9791 (strong positive linear correlation).
4.2
During EDA on latency measurements (in ms), you observe Q1 = 25.0, median = 30.0, Q3 = 35.0, and a single spike at 62.0. Apply Tukey's 1.5 * IQR rule to find the lower and upper fences, determine whether 62.0 is an outlier, and explain how to test whether two server clusters differ significantly in mean latency.
Computing IQR and Tukey Fences:
IQR = Q3 - Q1 = 35.0 - 25.0 = 10.0 ms.- Lower fence =
Q1 - 1.5 * IQR = 25.0 - 1.5 * 10.0 = 25.0 - 15.0 = 10.0 ms. - Upper fence =
Q3 + 1.5 * IQR = 35.0 + 1.5 * 10.0 = 35.0 + 15.0 = 50.0 ms. - Because
62.0 > 50.0, the62.0 msspike is flagged as an upper outlier.
Testing Mean Difference Between Two Clusters: Run Welch's independent two-sample t-test (stats.ttest_ind(cluster_a, cluster_b, equal_var=False))—or stats.mannwhitneyu if heavy outliers violate normality—and reject H0 (equal means/distributions) only if p_value <= 0.05.
IQR = 10.0 ms, yielding fences [10.0, 50.0] ms. Since 62.0 > 50.0, 62.0 ms is an outlier. Compare two clusters using Welch's ttest_ind(..., equal_var=False) (or Mann–Whitney U if skewed) at alpha = 0.05.
4.3
How should a student answer an exam question asking: "How do you fit and interpret an OLS regression model in statsmodels, and how does EDA automation complement inferential testing"?
Explaining statsmodels OLS Setup and Key Output Fields: Show the 4-line pattern (sm.add_constant(X), sm.OLS(y, X).fit(), .summary()) and explain the five core diagnostic values: coefficients (coef), individual t-test P > |t|, Adjusted R-squared, F-statistic p-value, and Durbin–Watson / Condition Number (residual autocorrelation and multicollinearity).
Connecting Automated EDA to Inferential Testing: Explain that automated EDA scripts or ydata-profiling rapidly scan every column for missingness, skewness, extreme outliers, and collinear pairs first—ensuring assumptions (linearity, independence, outliers handled) are checked before trusting statsmodels p-values and confidence intervals.
Show sm.add_constant() + sm.OLS(y, X).fit().summary(), interpret coef, P > |t|, Adj. R-squared, F-stat, and multicollinearity diagnostics, and explain that automated EDA audits missingness, skew, and collinearity up front so downstream inferential tests remain valid.
Exercises
Practice Exercises — Applied Statistics and Exploratory Analysis
Differentiate between covariance and Pearson correlation. If two variables have cov(X, Y) = 6.0, s_x = 2.0, and s_y = 4.0, compute the Pearson correlation coefficient r and interpret its strength and direction.
Covariance vs. Pearson Correlation: Covariance measures linear co-variation in raw units (unit_X * unit_Y), making magnitudes incomparable across different feature scales. Pearson correlation normalizes covariance by s_x * s_y, producing a unitless value strictly in [-1, +1].
Computing r: r = cov(X, Y) / (s_x * s_y) = 6.0 / (2.0 * 4.0) = 6.0 / 8.0 = 0.75. Because r = +0.75 is close to +1, X and Y exhibit a strong positive linear relationship.
Covariance is scale-dependent while Pearson r is dimensionless in [-1, +1]. Here r = 6.0 / (2.0 * 4.0) = 0.75, indicating a strong positive linear association.
State the Central Limit Theorem (CLT) and define Standard Error (SE). For a population with sample standard deviation s = 2.4 and sample size n = 36, compute the Standard Error of the sample mean.
CLT and Standard Error: By the Central Limit Theorem, for sufficiently large random samples (n >= 30), the sampling distribution of the sample mean approaches a Normal distribution centered at the population mean mu, regardless of population shape. The Standard Error (SE = s / sqrt(n)) measures the standard deviation of that sample mean across repeated samples.
Calculation: SE = 2.4 / sqrt(36) = 2.4 / 6 = 0.4.
The CLT states that sample means follow an approximately Normal distribution for large n, with spread measured by Standard Error SE = s / sqrt(n). For s = 2.4 and n = 36, SE = 0.4.
Explain p-value, Type I error, and Type II error in hypothesis testing. Given an A/B test resulting in p = 0.018 at significance level alpha = 0.05, state the statistical decision and which error type could still occur.
Definitions and Decision:
- p-value: Probability of obtaining a test statistic at least as extreme as observed if
H0is true. - Type I error (
alpha): False positive (rejecting a trueH0). - Type II error (
beta): False negative (failing to reject a falseH0).
Since p = 0.018 <= alpha = 0.05, we reject H0.
Possible Remaining Error: Because we rejected H0, the only error possible under that decision is a Type I error (false positive, occurring with long-run rate alpha = 5% if H0 were actually true).
Reject H0 because 0.018 <= 0.05. Since H0 was rejected, a Type I error (false positive) is the error type that could occur if no real difference existed.
Why must sm.add_constant(X) be called when using sm.OLS(y, X) in statsmodels? Given a fitted OLS equation y_hat = 1.5 + 2.5 * x with P > |t| = 0.002 for the slope, predict y_hat when x = 4 and state whether the slope is significant at alpha = 0.05.
Need for sm.add_constant(X): sm.OLS(y, X) fits a line forced through the origin (y = beta_1 * x) unless a column of 1s is explicitly present in X. sm.add_constant(X) prepends that const column so an intercept beta_0 is estimated.
Prediction and Significance:
- At
x = 4:y_hat = 1.5 + 2.5 * 4 = 1.5 + 10.0 = 11.5. - Because
p = 0.002 <= 0.05, we rejectH0: beta_1 = 0; the slope is statistically significant.
sm.add_constant(X) adds the column of 1s required to estimate an intercept term. For x = 4, y_hat = 11.5, and since p = 0.002 <= 0.05, the slope is statistically significant.
Distinguish between descriptive EDA and inferential EDA, and explain how Python automates EDA workflows (auto_eda helpers and ydata-profiling). In a 2,000-row table where "income" has 50 NaN entries and right skewness 2.8, compute its missing percentage and suggest the appropriate imputation statistic.
Descriptive vs. Inferential EDA and Automation:
- Descriptive EDA summarizes observed sample patterns (counts, quantiles, skewness, correlations, visualizations). Inferential EDA uses hypothesis tests and confidence intervals (
ttest_ind,chi2_contingency, OLS p-values) to assess whether patterns reflect true population effects. - Automation uses reusable Python summary functions (
df.dtypes,.isna().sum(),.skew()) orydata-profilingHTML reports to audit every column consistently and enforce pipeline quality gates.
Missing Percentage and Imputation Choice:
- Missing percentage =
(50 / 2000) * 100 = 2.5%. - Because
"income"is strongly right-skewed (skew = 2.8 > 0), the mean is pulled upward by high earners; use the median for imputation (optionally alongside a log transformnp.log1p).
Descriptive EDA summarizes sample characteristics while inferential EDA tests generalization with statistical tests; automated Python scripts or ydata-profiling generate standardized audits across columns. Missingness is 2.5%, and median imputation should be used due to heavy right skew (2.8).