Descriptive & Inferential Statistics, CLT & Sampling
Open in the Padhaiko app2.6 Descriptive and Inferential Statistics
In data science, statistics provides the mathematical framework for extracting meaning from raw data, evaluating patterns, and making sound decisions in the presence of uncertainty. Statistical methods are universally divided into two broad branches: descriptive statistics and inferential statistics.
Descriptive Statistics
Descriptive statistics involves organizing, summarizing, and presenting the main characteristics of a specific dataset without drawing conclusions or generalizations beyond the data observed.
Descriptive methods describe what is happening in the data at hand. The primary approaches include:
- Measures of Central Tendency:
- Mean ( or ): The arithmetic average of all observations:
- Median: The middle value when the data points are arranged in ascending order (robust to extreme outliers).
- Mode: The most frequently occurring value in the dataset.
- Measures of Dispersion (Spread):
- Range: The difference between the maximum and minimum observations: .
- Sample Variance (): The average squared deviation from the sample mean:
- Sample Standard Deviation (): The square root of the sample variance, expressed in the original units of measurement: .
- Interquartile Range (IQR): , capturing the spread of the middle of the data.
- Frequency Distributions and Visualizations:
- Frequency tables, histograms, box plots, bar charts, and scatter plots that summarize data shape, skewness, and modality.
Inferential Statistics
Inferential statistics is the branch of statistics that uses sample data to make predictions, generalizations, estimations, or hypothesis tests about an entire underlying population.
Because measuring an entire population (every customer, every sensor reading, every citizen) is often physically impossible or prohibitively expensive, data scientists collect a representative sample and apply probability theory to draw conclusions about the population.
Key approaches in inferential statistics include:
- Parameter Estimation: Approximating unknown population metrics through point estimates (e.g., using sample mean to estimate population mean ) and interval estimates (constructing confidence intervals).
- Hypothesis Testing: Formulating competing claims ( vs ) and assessing whether sample evidence justifies rejecting the default assumption.
- Regression Analysis: Modeling relationships between dependent and independent variables to forecast future values.
Population Parameters vs Sample Statistics
A clear distinction between the target group and the collected subset is fundamental to all statistical modeling:
- A population is the entire collection of individuals, measurements, or objects under study. Characteristics of a population are fixed constants called parameters (denoted by Greek letters: size , mean , variance , standard deviation ).
- A sample is a subset of observations drawn from the population. Numerical summaries computed from sample data are called statistics (denoted by Latin letters: size , mean , variance , standard deviation ).
| Aspect | Descriptive Statistics | Inferential Statistics |
|---|---|---|
| Primary Purpose | Summarize and describe observed data | Draw conclusions and make predictions about an unobserved population |
| Scope of Analysis | Limited strictly to the dataset at hand | Generalizes from a sample to the broader population |
| Basis of Validity | Direct mathematical aggregation | Probability theory and sampling distributions |
| Core Tools | Mean, median, variance, standard deviation, histograms | Hypothesis testing (, , ANOVA), confidence intervals, regression |
| Treatment of Uncertainty | Zero uncertainty (describes exact collected records) | Quantified uncertainty (margins of error, significance levels , -values) |
| Practical Example | Calculating the average response time of recorded API requests | Estimating whether a new server cache significantly reduces latency across all future user sessions |
2.7 Central Limit Theorem and Sample Distribution Concepts
In machine learning and data science, algorithms frequently rely on sample aggregates. Understanding how a sample summary behaves when sampling is repeated is the foundation of inferential confidence.
Concept of Sampling Distributions
A sampling distribution is the theoretical probability distribution of a sample statistic (such as the sample mean , sample proportion , or sample variance ) obtained by drawing all possible independent random samples of the same size from a population.
When we draw a sample of size , the observations are random variables. Consequently, any statistic computed from them—such as the sample mean —is itself a random variable with its own probability distribution, expected value, and standard deviation.
Properties of the sampling distribution of the sample mean :
- Expected Value (Unbiasedness): The mean of the sampling distribution of , denoted , is always equal to the true population mean :
- Standard Error (): The standard deviation of the sampling distribution of , denoted , is called the standard error of the mean:
where is the population standard deviation and is the sample size. The standard error quantifies how much the sample mean fluctuates from sample to sample; it decreases at the rate of as sample size grows.
Central Limit Theorem (CLT)
Let be a sequence of independent and identically distributed (i.i.d.) random variables drawn from any arbitrary population distribution having a well-defined population mean and a finite variance .
As the sample size becomes sufficiently large (empirically ), the sampling distribution of the sample mean approaches a normal distribution with mean and variance , regardless of the shape of the original parent population:
Equivalently, the standardized variable converges to the standard normal distribution:
| Requirement | Condition | What Happens if Violated |
|---|---|---|
| Independence | Observations must be randomly sampled and independent | Positive correlation among observations deflates the true variance, causing overconfident inferences |
| Finite Variance | For heavy-tailed distributions with infinite variance (e.g., Cauchy or Pareto with ), the sample mean never converges to a Gaussian bell curve | |
| Sample Size | (general rule of thumb) | For heavily skewed or multimodal parent distributions, larger samples ( or ) are needed before the normal approximation becomes accurate |
| Application Area | How the Central Limit Theorem Powers It |
|---|---|
| Parametric Hypothesis Testing | Allows data scientists to run -tests, -tests, and ANOVA on sample averages without requiring the raw data itself to be Gaussian |
| Confidence Interval Construction | Enables formulating margins of error across diverse business and engineering metrics |
| A/B Testing in Digital Products | Justifies comparing average revenue per user (ARPU), click-through rates, and session durations between treatment and control cohorts |
| Machine Learning Noise Modeling | Explains why measurement noise and composite residuals in linear regression are effectively modeled as Gaussian , since composite noise is the aggregate sum of numerous tiny, independent disturbances |
Normal Approximation to Discrete Distributions
A direct practical consequence of the Central Limit Theorem is the normal approximation to discrete probability distributions, particularly the Binomial and Poisson distributions.
- Approximation to the Binomial Distribution: Let count the number of successes in independent Bernoulli trials. Since is a sum of i.i.d. random variables with and :
- Applicability Condition: The approximation is valid when both and (or more conservatively, and ).
- Continuity Correction: Because a continuous distribution is used to approximate a discrete step function, the discrete integer value corresponds to the continuous interval :
Worked Out Examples
2.1
A finite population consists of four values: .
- Calculate the population mean and population standard deviation .
- List all possible random samples of size drawn with replacement (all samples) and calculate their sample means.
- Construct the probability distribution of the sample mean and verify numerically that and .
Computing Population Mean and Population Standard Deviation: The population values are numbers: .
The deviations and squared deviations are:
- For :
- For :
- For :
- For :
The population variance and standard deviation are:
Listing All 16 Possible Samples of Size 2 Drawn with Replacement: Each sample pair has mean :
| Sample | Mean | Sample | Mean | Sample | Mean | Sample | Mean |
|---|---|---|---|---|---|---|---|
Constructing Probability Distribution and Verifying Mean and Standard Error: Since all samples are equally likely, each carries probability . Grouping by distinct values of :
| Freq | ||||
|---|---|---|---|---|
| Total | 16 | 1.0 | 2528/16 = 158 | 399584/16 = 24974 |
Calculating the mean of the sampling distribution:
Calculating the variance and standard error of :
Comparing with the theoretical formula:
Both theoretical relationships and are exactly satisfied.
The population has and . The sampling distribution of across all samples has mean and standard deviation , validating and .
2.2
A large population of automotive parts has a mean lifetime of and a standard deviation of . A random sample of size is drawn from this population.
- State whether the Central Limit Theorem applies and specify the sampling distribution of the sample mean .
- Calculate the probability that the sample mean lifetime falls between and , i.e., .
Evaluating Applicability of CLT and Finding Standard Error: The sample size is . Since , the Central Limit Theorem applies regardless of whether the original population distribution is normal, skewed, or uniform.
The sampling distribution of the sample mean is approximately normal:
Evaluating the mean and standard error:
Standardizing the Bounds and Computing Probability: To find , we convert both boundary values into standard normal -scores:
Using the standard normal cumulative distribution function :
Because , the CLT applies with and standard error . The probability that the sample mean falls between and is approximately (or ).
2.3
A server experiences independent background jobs where each job succeeds with probability . In a batch of jobs, let denote the number of successful jobs ().
- Check the conditions for normal approximation.
- Using the continuity-corrected normal approximation, find the probability that between and jobs succeed inclusive, i.e., .
Checking Applicability Conditions and Finding Distribution Parameters: For a Binomial random variable with and :
Both conditions are well above , confirming that the normal distribution provides a reliable approximation.
Parameters of the approximating normal variable :
Applying Continuity Correction and Standardizing: Because the discrete integer range is , applying the continuity correction broadens the boundaries by on both sides:
Standardizing the endpoints into standard normal -scores:
Using the standard normal cumulative distribution function :
The normal approximation conditions are satisfied (). With continuity correction, (or ).
Exercises
Exercises
Define inferential statistics and provide a concrete practical example of its application in modern data science.
Formal Definition: Inferential statistics is the branch of mathematical statistics that deduces properties, estimates parameters, and tests hypotheses regarding an underlying population based on the empirical observations collected in a representative sample.
Practical Application in Data Science: Consider an e-commerce platform with active users. The engineering team redesigns the checkout page to reduce friction. Surveying or testing all users simultaneously is impractical and risks platform-wide revenue loss.
Instead, the team exposes a randomly sampled treatment cohort of users to the new interface and keeps users on the existing interface (A/B testing). Using inferential statistics, the team calculates the conversion rates, constructs confidence intervals for the lift in conversion, and performs a two-sample hypothesis test to confirm whether the observed improvement reflects a genuine population-wide trend rather than random sample variation.
Inferential statistics uses sample data to make generalized predictions and test hypotheses about an unobserved population. A classic data science application is A/B testing, where metrics from a sample cohort are analyzed to infer whether a new feature genuinely improves engagement across the entire user base.
Contrast population parameters and sample statistics. Present the standard mathematical notation for population size, mean, variance, and standard deviation alongside their sample counterparts.
Conceptual Distinction: A population parameter is a fixed numerical characteristic describing the complete population. Because populations are typically vast, parameters are usually unknown constants.
A sample statistic is a numerical value computed directly from the observed sample data. Because samples vary randomly, statistics are random variables that serve as estimators of the corresponding population parameters.
Comparison of Notation:
| Metric | Population Parameter (Greek) | Sample Statistic (Latin) |
|---|---|---|
| Total Size / Count | ||
| Mean | ||
| Variance | ||
| Standard Deviation | ||
| Proportion | or |
Population parameters () describe the complete population and are constant but generally unknown. Sample statistics () describe a collected subset, vary between samples, and serve as estimators for population parameters.
Explain the concept of estimation in inferential statistics, distinguish between point estimation and interval estimation, and define confidence level.
Point Estimation vs Interval Estimation: Estimation is the process of using sample statistics to approximate unknown population parameters:
- Point Estimation: A point estimator calculates a single numerical value to approximate the parameter. For example, using the sample mean as the single estimate for the population mean height . While concise, a point estimate gives no indication of uncertainty or margin of error.
- Interval Estimation: An interval estimator constructs a range of plausible values within which the parameter is expected to lie with a specified degree of confidence:
For instance, estimating that the population mean height is between and .
Confidence Level: The confidence level, denoted , is the long-run proportion of constructed intervals that will contain the true population parameter if independent random sampling were repeated indefinitely under identical conditions. A confidence level means that in out of independent random samples, the resulting interval will successfully enclose the true parameter .
Point estimation provides a single best-guess value ( for ), whereas interval estimation provides a range that captures uncertainty. The confidence level quantifies the long-run probability that repeated independent sample intervals enclose the true population parameter.
Prepare a technical note on the Central Limit Theorem (CLT). State the formal theorem, list its necessary assumptions, and explain why it is considered fundamental in data analysis.
Formal Theorem Statement: Let be independent and identically distributed (i.i.d.) random variables drawn from an arbitrary population distribution with mean and finite variance . As , the distribution of the sample mean converges in distribution to a normal distribution:
Necessary Assumptions:
- Independence: Sampling must be random and independent (or sample size if sampling without replacement).
- Finite Variance: The second moment must be finite (). Distributions with heavy power-law tails and undefined variance (such as the Cauchy distribution) fail the CLT.
- Adequate Sample Size: For moderately symmetric distributions, is sufficient; for severely skewed distributions, or is required.
Why it is Fundamental in Data Analysis:
- It permits parametric inferences (-tests, -tests, confidence intervals) on real-world datasets without requiring normality in the underlying population.
- It underpins hypothesis testing in A/B experimentation and controlled machine learning model comparisons.
- It validates treating aggregate measurement errors and regression residuals as normally distributed noise .
The Central Limit Theorem asserts that the sample mean of i.i.d. observations with finite variance converges to a normal distribution regardless of parent population shape. It is vital because it enables parametric hypothesis testing, confidence interval derivation, and error modeling on non-Gaussian data.
The assessed tax values of all registered vehicles in a state have a population mean of and a population standard deviation of . Suppose random samples of size vehicles are drawn repeatedly.
- Determine the mean and the standard error of the sample mean .
- State the approximate distribution of and justify your answer.
Calculating Mean and Standard Error of the Sampling Distribution: By the unbiasedness of the sample mean:
The standard error of the sample mean is:
Determining the Distribution: Even though vehicle values in the real world are positively skewed (with many inexpensive cars and a few high-value luxury cars), the sample size is , which significantly exceeds the threshold of .
By the Central Limit Theorem, the sampling distribution of is approximately normal:
The mean of the sample mean is and the standard error is . By the Central Limit Theorem (), follows an approximately normal distribution .
An electronic hardware manufacturing facility produces precision resistors with a nominal mean resistance of and standard deviation . The resistance values are known to follow a normal distribution. If a quality control engineer inspects a random sample of resistors, what is the probability that the sample mean resistance is less than ?
Evaluating Sampling Distribution Properties: Because the parent population is normally distributed, the sampling distribution of is exactly normal for any sample size :
Mean and standard error:
Standardizing and Computing Probability: To find , we compute the standard normal -score:
Looking up in the standard normal distribution table:
The standard error of the sample mean is , giving a -score of . The probability of finding a sample mean resistance less than is approximately (or ).
A balanced coin is tossed times independently. Let represent the number of heads obtained (). Using the normal approximation with continuity correction, determine the probability of obtaining between and heads inclusive, i.e., .
Verifying Conditions and Deriving Parameters: For and :
The conditions for the normal approximation are amply satisfied.
Parameters of the approximating normal distribution:
Applying Continuity Correction and Calculating Probability: Converting the discrete range into continuous bounds:
Standardizing the continuous endpoints:
Using the cumulative normal distribution:
Using the normal approximation () with continuity correction, the interval is (), yielding a probability of approximately (or ).
Define the term "Sampling distribution". [Assignment 2, Q17]
Formal Definition: A sampling distribution is the probability distribution of a sample statistic (such as the sample mean , sample variance , or sample proportion ) computed across all possible independent random samples of the same fixed size drawn from a given population.
Key Properties for the Sample Mean : When random samples of size are drawn from a population with mean and standard deviation :
- Unbiasedness (Expected Value): The mean of the sampling distribution of equals the population mean:
- Standard Error of the Mean: The standard deviation of the sampling distribution measures sample-to-sample variability and decreases with :
- Asymptotic Normality: By the Central Limit Theorem, as increases (), the sampling distribution of approaches regardless of the population's shape.
A sampling distribution is the probability distribution of a statistic (such as the mean or variance ) calculated from all possible samples of the same size drawn from a population. For the sample mean , its mean is and its standard error is .
Differentiate between Descriptive Statistics and Inferential Statistics with illustrative examples. [Assignment, notebook p236]
Conceptual Overview:
- Descriptive Statistics focuses on organizing, summarizing, and presenting the observed characteristics of a collected dataset without making generalizations beyond it.
- Inferential Statistics uses sample data and probability theory to make generalizations, estimations, predictions, or decisions about a larger population.
Point-by-Point Comparison:
| Basis of Comparison | Descriptive Statistics | Inferential Statistics |
|---|---|---|
| Core Function | Describes and summarizes collected data | Makes predictions, estimations, or conclusions |
| Objective | To understand the given dataset at hand | To generalize findings about a larger population |
| Data Used | Uses the complete observed dataset | Uses random sample data () drawn from a population () |
| Nature of Output | Reports exact facts (mean , median, charts) with no guessing | Provides estimates (confidence intervals) or hypothesis decisions with quantified uncertainty |
| Illustrative Example | The average marks of a class of students is | Inferring from a sample that many students across the university will score above |
Descriptive statistics summarizes and describes the exact dataset collected (e.g., reporting that the average mark in a class is ), whereas inferential statistics uses sample data to generalize, estimate parameters (), and make predictions or decisions about a larger population.