PadhaikoAll subjects

Applied Statistics and Exploratory Analysis

Advanced Python Programming for Data Science (ENCT 325)

Open in the Padhaiko app

4.1 Statistical measures: Correlation, covariance, skewness, kurtosis

Descriptive statistics summarize center, spread, linear association, and distributional shape before fitting predictive models.

Covariance and Correlation

Definition

Covariance vs. Correlation

AspectPearson Correlation (r)Spearman Rank Correlation (rho)
Relationship MeasuredStrictly linear association on raw valuesMonotonic association on sorted ranks
Scale & DistributionSensitive to extreme outliers; assumes interval/ratio scaleRobust to outliers and skewed monotonic curves
Data Type SupportContinuous numeric columnsNumeric or ordinal ranked data

Shape Measures: Skewness and Kurtosis

  1. Skewness (Asymmetry):
    • skewness = 0: Symmetric distribution (e.g., Normal distribution; mean = median = mode).
    • skewness > 0 (Positive / Right-skewed): Long right tail pulling mean > median (e.g., household income, session duration).
    • skewness < 0 (Negative / Left-skewed): Long left tail pulling mean < median (e.g., easy exam scores).
  2. Kurtosis (Tail Heaviness / Peakedness):
    • Standard normal distribution has Pearson kurtosis = 3.0 and excess kurtosis = kurtosis - 3 = 0.0 (mesokurtic).
    • Leptokurtic (excess kurtosis > 0): Heavy tails and sharper peak (more extreme outliers, e.g., financial returns).
    • Platykurtic (excess kurtosis < 0): Light tails and flatter peak (e.g., uniform distribution, excess kurtosis -1.2).
Python
import pandas as pd
from scipy import stats

s = pd.Series([10, 12, 14, 15, 18, 45])
print("Skewness:", round(s.skew(), 3))          # positive right skew
print("Excess kurtosis:", round(s.kurt(), 3))   # pandas .kurt() returns Fisher excess kurtosis

4.2 Probability review, sampling, and hypothesis testing

Inferential statistics uses a finite sample to reason rigorously about an underlying population parameter under uncertainty.

Probability Essentials and Sampling

Hypothesis Testing Framework

Definition

Null Hypothesis, p-value, and Decision Errors

  1. Hypotheses: State the Null Hypothesis (H0) (no effect / no difference) and Alternative Hypothesis (H1 or Ha) (two-sided != or one-sided >, <).
  2. Test Statistic & p-value: Compute a test statistic (t, z, F, or chi^2) from the sample. The p-value is the probability of observing a test statistic at least as extreme as the sample's assuming H0 is true.
  3. Decision Rule: Choose significance level alpha (commonly 0.05). If p-value <= alpha, reject H0 (statistically significant); otherwise, fail to reject H0.
  4. Error Types:
    • Type I Error (False Positive): Rejecting H0 when H0 is actually true (probability = alpha).
    • Type II Error (False Negative): Failing to reject H0 when H1 is true (probability = beta; statistical power = 1 - beta).
Common Hypothesis Tests in Python (scipy.stats)
TestSciPy FunctionPurpose / Data Requirement
One-Sample / Independent t-teststats.ttest_1samp, stats.ttest_indCompare means of 1 or 2 continuous samples (equal_var=False for Welch's t-test)
Paired t-teststats.ttest_rel(before, after)Compare before/after measurements on the same subjects
One-Way ANOVAstats.f_oneway(g1, g2, g3)Test whether means differ across 3+ independent groups
Chi-Square Independence Teststats.chi2_contingency(table)Test association between two categorical variables in a contingency table
Mann–Whitney U Teststats.mannwhitneyu(x, y)Non-parametric alternative to independent t-test for skewed/ordinal data

4.3 Regression and trend analysis using stats models

While scikit-learn emphasizes predictive accuracy, Python's statsmodels library provides classical statistical inference—coefficient standard errors, t-statistics, p-values, confidence intervals, and diagnostic tests.

Ordinary Least Squares (OLS) with statsmodels

For linear model y = beta_0 + beta_1 * x_1 + ... + beta_k * x_k + epsilon:

Interpreting statsmodels OLS Summary Metrics
MetricInterpretationWhat to Watch For
R-squared / Adj. R-squaredFraction of variance in y explained by predictorsAdjusted R-squared penalizes adding useless features
F-statistic (Prob F-stat)Joint test that all slope coefficients equal zerop < 0.05 indicates the overall model beats a constant mean
Coef and its p-valueEstimated slope and individual significance test (H0: beta_j = 0)Predictors with p > 0.05 may not be statistically significant
Durbin–WatsonAutocorrelation in residuals (0 to 4; ~2.0 = none)Values far below 1.5 or above 2.5 signal serial correlation
Condition NumberDetects multicollinearity among predictorsValues > 30 suggest correlated features inflating standard errors
Python
import pandas as pd
import statsmodels.api as sm

df = pd.DataFrame({"ad_spend": [1, 2, 3, 4, 5], "sales": [3, 5, 7, 9, 11]})
X = sm.add_constant(df[["ad_spend"]])
res = sm.OLS(df["sales"], X).fit()
print(res.params)     # const=1.0, ad_spend=2.0
print("R2:", res.rsquared)

Trend Analysis and Time-Series Decomposition

Trend analysis isolates long-term direction (y_t = Trend + Seasonal + Residual) using:


4.4 Exploratory data analysis (EDA) using descriptive and inferential methods

Exploratory Data Analysis (EDA) systematically inspects data quality, distributions, anomalies, and relationships before choosing transformations or models.

AspectDescriptive EDAInferential EDA
GoalSummarize what is in the sample (shape, center, spread, missingness)Test whether observed patterns generalize beyond random sampling noise
Methodsdf.describe(), quantiles, IQR, skewness, correlation matrix, group tables, histograms, box plotsConfidence intervals (95% CI), t-tests, ANOVA, Chi-square tests, OLS p-values
Key PitfallMistaking sample noise or Simpson's paradox for real effectsOver-interpreting tiny p-values in huge datasets without checking effect size

Systematic EDA Checklist

  1. Structural Audit: Inspect .shape, .dtypes, .head(), .isna().sum(), and .duplicated().sum().
  2. Univariate Analysis: Compute mean, median, std, IQR (Q3 - Q1), min/max, and outlier bounds via Tukey's 1.5 * IQR rule ([Q1 - 1.5*IQR, Q3 + 1.5*IQR]) for numeric features; .value_counts(normalize=True) for categoricals.
  3. Bivariate & Multivariate Analysis: Cross-tabulations (pd.crosstab), grouped aggregations (df.groupby("target").mean()), Pearson/Spearman heatmaps, and inferential tests (t-test/ANOVA for numeric vs. category, Chi-square for category vs. category).
  4. Target Leakage & Domain Sanity Checks: Flag impossible values (negative age, future dates, post-outcome columns).

4.5 Automation of EDA workflows using Python

Writing repetitive describe(), isna(), skewness, and outlier loops manually across dozens of projects wastes time. Automating EDA builds repeatable, consistent diagnostic reports in seconds.

Automated EDA Approaches in Python

  1. Custom Reusable Python EDA Function / Module: Combine Pandas, scipy.stats, and summary dictionaries to return structured audit tables (dtype, nulls, % missing, unique count, mean, median, skew, outlier count) that integrate directly into CI/CD or notebooks.
  2. Dedicated Auto-EDA Libraries:
    • ydata-profiling (formerly pandas-profiling): Generates rich standalone HTML reports (ProfileReport(df).to_file("eda.html")) covering interactions, correlations, missingness matrices, and duplicate alerts.
    • sweetviz / dtale: Compare train vs. test feature drift side by side or launch interactive browser grid explorers.
  3. Automated Guardrails in Pipelines: Pair automated EDA summaries with threshold checks (e.g., abort training if missingness exceeds 20% or target class ratio shifts sharply).
Python
import pandas as pd


def auto_eda_summary(df: pd.DataFrame) -> pd.DataFrame:
    rows = []
    n = len(df)
    for col in df.columns:
        s = df[col]
        missing = int(s.isna().sum())
        rec = {
            "column": col,
            "dtype": str(s.dtype),
            "missing": missing,
            "missing_pct": round((missing / n) * 100, 2) if n else 0.0,
            "n_unique": int(s.nunique(dropna=True)),
        }
        if pd.api.types.is_numeric_dtype(s) and not pd.api.types.is_bool_dtype(s):
            rec["mean"] = round(float(s.mean()), 3)
            rec["median"] = round(float(s.median()), 3)
            rec["skew"] = round(float(s.skew()), 3)
        rows.append(rec)
    return pd.DataFrame(rows)

Worked Examples

Example 4.1

4.1

Given paired observations X = [1, 2, 3, 4, 5] and Y = [2, 4, 5, 9, 10], calculate manually (and verify with Python logic):

  1. Sample means x_bar and y_bar,
  2. Sample covariance cov(X, Y),
  3. Sample standard deviations s_x and s_y, and Pearson correlation r.

Sample Means:

  • x_bar = (1 + 2 + 3 + 4 + 5) / 5 = 15 / 5 = 3.0
  • y_bar = (2 + 4 + 5 + 9 + 10) / 5 = 30 / 5 = 6.0

Deviations, Sample Covariance, and Standard Deviations (n - 1 = 4):

x_iy_idx = x_i - 3dy = y_i - 6dx * dydx^2dy^2
12-2-48416
24-1-2214
350-1001
4913319
510248416
  • sum(dx * dy) = 8 + 2 + 0 + 3 + 8 = 21 → cov(X, Y) = 21 / 4 = 5.25.
  • sum(dx^2) = 4 + 1 + 0 + 1 + 4 = 10 → s_x = sqrt(10 / 4) = sqrt(2.5) ≈ 1.5811.
  • sum(dy^2) = 16 + 4 + 1 + 9 + 16 = 46 → s_y = sqrt(46 / 4) = sqrt(11.5) ≈ 3.3912.

Pearson Correlation Coefficient r:

Answer:

x_bar = 3.0, y_bar = 6.0, cov(X, Y) = 5.25, s_x ≈ 1.5811, s_y ≈ 3.3912, and Pearson correlation r ≈ 0.9791 (strong positive linear correlation).

Example 4.2

4.2

During EDA on latency measurements (in ms), you observe Q1 = 25.0, median = 30.0, Q3 = 35.0, and a single spike at 62.0. Apply Tukey's 1.5 * IQR rule to find the lower and upper fences, determine whether 62.0 is an outlier, and explain how to test whether two server clusters differ significantly in mean latency.

Computing IQR and Tukey Fences:

  • IQR = Q3 - Q1 = 35.0 - 25.0 = 10.0 ms.
  • Lower fence = Q1 - 1.5 * IQR = 25.0 - 1.5 * 10.0 = 25.0 - 15.0 = 10.0 ms.
  • Upper fence = Q3 + 1.5 * IQR = 35.0 + 1.5 * 10.0 = 35.0 + 15.0 = 50.0 ms.
  • Because 62.0 > 50.0, the 62.0 ms spike is flagged as an upper outlier.

Testing Mean Difference Between Two Clusters: Run Welch's independent two-sample t-test (stats.ttest_ind(cluster_a, cluster_b, equal_var=False))—or stats.mannwhitneyu if heavy outliers violate normality—and reject H0 (equal means/distributions) only if p_value <= 0.05.

Answer:

IQR = 10.0 ms, yielding fences [10.0, 50.0] ms. Since 62.0 > 50.0, 62.0 ms is an outlier. Compare two clusters using Welch's ttest_ind(..., equal_var=False) (or Mann–Whitney U if skewed) at alpha = 0.05.

Example 4.3

4.3

How should a student answer an exam question asking: "How do you fit and interpret an OLS regression model in statsmodels, and how does EDA automation complement inferential testing"?

Explaining statsmodels OLS Setup and Key Output Fields: Show the 4-line pattern (sm.add_constant(X), sm.OLS(y, X).fit(), .summary()) and explain the five core diagnostic values: coefficients (coef), individual t-test P > |t|, Adjusted R-squared, F-statistic p-value, and Durbin–Watson / Condition Number (residual autocorrelation and multicollinearity).

Connecting Automated EDA to Inferential Testing: Explain that automated EDA scripts or ydata-profiling rapidly scan every column for missingness, skewness, extreme outliers, and collinear pairs first—ensuring assumptions (linearity, independence, outliers handled) are checked before trusting statsmodels p-values and confidence intervals.

Answer:

Show sm.add_constant() + sm.OLS(y, X).fit().summary(), interpret coef, P > |t|, Adj. R-squared, F-stat, and multicollinearity diagnostics, and explain that automated EDA audits missingness, skew, and collinearity up front so downstream inferential tests remain valid.


Exercises

Practice Exercises — Applied Statistics and Exploratory Analysis

Question 1

Differentiate between covariance and Pearson correlation. If two variables have cov(X, Y) = 6.0, s_x = 2.0, and s_y = 4.0, compute the Pearson correlation coefficient r and interpret its strength and direction.

Covariance vs. Pearson Correlation: Covariance measures linear co-variation in raw units (unit_X * unit_Y), making magnitudes incomparable across different feature scales. Pearson correlation normalizes covariance by s_x * s_y, producing a unitless value strictly in [-1, +1].

Computing r: r = cov(X, Y) / (s_x * s_y) = 6.0 / (2.0 * 4.0) = 6.0 / 8.0 = 0.75. Because r = +0.75 is close to +1, X and Y exhibit a strong positive linear relationship.

Answer:

Covariance is scale-dependent while Pearson r is dimensionless in [-1, +1]. Here r = 6.0 / (2.0 * 4.0) = 0.75, indicating a strong positive linear association.

Question 2

State the Central Limit Theorem (CLT) and define Standard Error (SE). For a population with sample standard deviation s = 2.4 and sample size n = 36, compute the Standard Error of the sample mean.

CLT and Standard Error: By the Central Limit Theorem, for sufficiently large random samples (n >= 30), the sampling distribution of the sample mean approaches a Normal distribution centered at the population mean mu, regardless of population shape. The Standard Error (SE = s / sqrt(n)) measures the standard deviation of that sample mean across repeated samples.

Calculation: SE = 2.4 / sqrt(36) = 2.4 / 6 = 0.4.

Answer:

The CLT states that sample means follow an approximately Normal distribution for large n, with spread measured by Standard Error SE = s / sqrt(n). For s = 2.4 and n = 36, SE = 0.4.

Question 3

Explain p-value, Type I error, and Type II error in hypothesis testing. Given an A/B test resulting in p = 0.018 at significance level alpha = 0.05, state the statistical decision and which error type could still occur.

Definitions and Decision:

  • p-value: Probability of obtaining a test statistic at least as extreme as observed if H0 is true.
  • Type I error (alpha): False positive (rejecting a true H0).
  • Type II error (beta): False negative (failing to reject a false H0).

Since p = 0.018 <= alpha = 0.05, we reject H0.

Possible Remaining Error: Because we rejected H0, the only error possible under that decision is a Type I error (false positive, occurring with long-run rate alpha = 5% if H0 were actually true).

Answer:

Reject H0 because 0.018 <= 0.05. Since H0 was rejected, a Type I error (false positive) is the error type that could occur if no real difference existed.

Question 4

Why must sm.add_constant(X) be called when using sm.OLS(y, X) in statsmodels? Given a fitted OLS equation y_hat = 1.5 + 2.5 * x with P > |t| = 0.002 for the slope, predict y_hat when x = 4 and state whether the slope is significant at alpha = 0.05.

Need for sm.add_constant(X): sm.OLS(y, X) fits a line forced through the origin (y = beta_1 * x) unless a column of 1s is explicitly present in X. sm.add_constant(X) prepends that const column so an intercept beta_0 is estimated.

Prediction and Significance:

  • At x = 4: y_hat = 1.5 + 2.5 * 4 = 1.5 + 10.0 = 11.5.
  • Because p = 0.002 <= 0.05, we reject H0: beta_1 = 0; the slope is statistically significant.
Answer:

sm.add_constant(X) adds the column of 1s required to estimate an intercept term. For x = 4, y_hat = 11.5, and since p = 0.002 <= 0.05, the slope is statistically significant.

Question 5

Distinguish between descriptive EDA and inferential EDA, and explain how Python automates EDA workflows (auto_eda helpers and ydata-profiling). In a 2,000-row table where "income" has 50 NaN entries and right skewness 2.8, compute its missing percentage and suggest the appropriate imputation statistic.

Descriptive vs. Inferential EDA and Automation:

  • Descriptive EDA summarizes observed sample patterns (counts, quantiles, skewness, correlations, visualizations). Inferential EDA uses hypothesis tests and confidence intervals (ttest_ind, chi2_contingency, OLS p-values) to assess whether patterns reflect true population effects.
  • Automation uses reusable Python summary functions (df.dtypes, .isna().sum(), .skew()) or ydata-profiling HTML reports to audit every column consistently and enforce pipeline quality gates.

Missing Percentage and Imputation Choice:

  • Missing percentage = (50 / 2000) * 100 = 2.5%.
  • Because "income" is strongly right-skewed (skew = 2.8 > 0), the mean is pulled upward by high earners; use the median for imputation (optionally alongside a log transform np.log1p).
Answer:

Descriptive EDA summarizes sample characteristics while inferential EDA tests generalization with statistical tests; automated Python scripts or ydata-profiling generate standardized audits across columns. Missingness is 2.5%, and median imputation should be used due to heavy right skew (2.8).

← Advanced Data Wrangling and Transformation (part 2: 3.4–3.6)Data Visualization and Storytelling →