PadhaikoAll subjects

Descriptive & Inferential Statistics, CLT & Sampling

Foundation of Data Science

Open in the Padhaiko app

2.6 Descriptive and Inferential Statistics

In data science, statistics provides the mathematical framework for extracting meaning from raw data, evaluating patterns, and making sound decisions in the presence of uncertainty. Statistical methods are universally divided into two broad branches: descriptive statistics and inferential statistics.

Definition

Descriptive Statistics

Descriptive statistics involves organizing, summarizing, and presenting the main characteristics of a specific dataset without drawing conclusions or generalizations beyond the data observed.

Descriptive methods describe what is happening in the data at hand. The primary approaches include:

  1. Measures of Central Tendency:
    • Mean ( or ): The arithmetic average of all observations:
  1. Measures of Dispersion (Spread):
    • Range: The difference between the maximum and minimum observations: .
    • Sample Variance (): The average squared deviation from the sample mean:
  1. Frequency Distributions and Visualizations:
    • Frequency tables, histograms, box plots, bar charts, and scatter plots that summarize data shape, skewness, and modality.
Definition

Inferential Statistics

Inferential statistics is the branch of statistics that uses sample data to make predictions, generalizations, estimations, or hypothesis tests about an entire underlying population.

Because measuring an entire population (every customer, every sensor reading, every citizen) is often physically impossible or prohibitively expensive, data scientists collect a representative sample and apply probability theory to draw conclusions about the population.

Key approaches in inferential statistics include:

  1. Parameter Estimation: Approximating unknown population metrics through point estimates (e.g., using sample mean to estimate population mean ) and interval estimates (constructing confidence intervals).
  2. Hypothesis Testing: Formulating competing claims ( vs ) and assessing whether sample evidence justifies rejecting the default assumption.
  3. Regression Analysis: Modeling relationships between dependent and independent variables to forecast future values.
Definition

Population Parameters vs Sample Statistics

A clear distinction between the target group and the collected subset is fundamental to all statistical modeling:

AspectDescriptive StatisticsInferential Statistics
Primary PurposeSummarize and describe observed dataDraw conclusions and make predictions about an unobserved population
Scope of AnalysisLimited strictly to the dataset at handGeneralizes from a sample to the broader population
Basis of ValidityDirect mathematical aggregationProbability theory and sampling distributions
Core ToolsMean, median, variance, standard deviation, histogramsHypothesis testing (, , ANOVA), confidence intervals, regression
Treatment of UncertaintyZero uncertainty (describes exact collected records)Quantified uncertainty (margins of error, significance levels , -values)
Practical ExampleCalculating the average response time of recorded API requestsEstimating whether a new server cache significantly reduces latency across all future user sessions

2.7 Central Limit Theorem and Sample Distribution Concepts

In machine learning and data science, algorithms frequently rely on sample aggregates. Understanding how a sample summary behaves when sampling is repeated is the foundation of inferential confidence.

Definition

Concept of Sampling Distributions

A sampling distribution is the theoretical probability distribution of a sample statistic (such as the sample mean , sample proportion , or sample variance ) obtained by drawing all possible independent random samples of the same size from a population.

When we draw a sample of size , the observations are random variables. Consequently, any statistic computed from them—such as the sample mean —is itself a random variable with its own probability distribution, expected value, and standard deviation.

Properties of the sampling distribution of the sample mean :

  1. Expected Value (Unbiasedness): The mean of the sampling distribution of , denoted , is always equal to the true population mean :
  1. Standard Error (): The standard deviation of the sampling distribution of , denoted , is called the standard error of the mean:

where is the population standard deviation and is the sample size. The standard error quantifies how much the sample mean fluctuates from sample to sample; it decreases at the rate of as sample size grows.

Theorem

Central Limit Theorem (CLT)

Let be a sequence of independent and identically distributed (i.i.d.) random variables drawn from any arbitrary population distribution having a well-defined population mean and a finite variance .

As the sample size becomes sufficiently large (empirically ), the sampling distribution of the sample mean approaches a normal distribution with mean and variance , regardless of the shape of the original parent population:

Equivalently, the standardized variable converges to the standard normal distribution:

Core Assumptions and Conditions for the Central Limit Theorem
RequirementConditionWhat Happens if Violated
IndependenceObservations must be randomly sampled and independentPositive correlation among observations deflates the true variance, causing overconfident inferences
Finite VarianceFor heavy-tailed distributions with infinite variance (e.g., Cauchy or Pareto with ), the sample mean never converges to a Gaussian bell curve
Sample Size (general rule of thumb)For heavily skewed or multimodal parent distributions, larger samples ( or ) are needed before the normal approximation becomes accurate
Why the CLT is Indispensable in Data Science
Application AreaHow the Central Limit Theorem Powers It
Parametric Hypothesis TestingAllows data scientists to run -tests, -tests, and ANOVA on sample averages without requiring the raw data itself to be Gaussian
Confidence Interval ConstructionEnables formulating margins of error across diverse business and engineering metrics
A/B Testing in Digital ProductsJustifies comparing average revenue per user (ARPU), click-through rates, and session durations between treatment and control cohorts
Machine Learning Noise ModelingExplains why measurement noise and composite residuals in linear regression are effectively modeled as Gaussian , since composite noise is the aggregate sum of numerous tiny, independent disturbances
Definition

Normal Approximation to Discrete Distributions

A direct practical consequence of the Central Limit Theorem is the normal approximation to discrete probability distributions, particularly the Binomial and Poisson distributions.

  1. Approximation to the Binomial Distribution: Let count the number of successes in independent Bernoulli trials. Since is a sum of i.i.d. random variables with and :
  1. Continuity Correction: Because a continuous distribution is used to approximate a discrete step function, the discrete integer value corresponds to the continuous interval :

Worked Out Examples

Example 2.1

2.1

A finite population consists of four values: .

  1. Calculate the population mean and population standard deviation .
  2. List all possible random samples of size drawn with replacement (all samples) and calculate their sample means.
  3. Construct the probability distribution of the sample mean and verify numerically that and .

Computing Population Mean and Population Standard Deviation: The population values are numbers: .

The deviations and squared deviations are:

  • For :
  • For :
  • For :
  • For :

The population variance and standard deviation are:

Listing All 16 Possible Samples of Size 2 Drawn with Replacement: Each sample pair has mean :

SampleMean SampleMean SampleMean SampleMean

Constructing Probability Distribution and Verifying Mean and Standard Error: Since all samples are equally likely, each carries probability . Grouping by distinct values of :

Freq
Total161.02528/16 = 158399584/16 = 24974

Calculating the mean of the sampling distribution:

Calculating the variance and standard error of :

Comparing with the theoretical formula:

Both theoretical relationships and are exactly satisfied.

Answer:

The population has and . The sampling distribution of across all samples has mean and standard deviation , validating and .

Example 2.2

2.2

A large population of automotive parts has a mean lifetime of and a standard deviation of . A random sample of size is drawn from this population.

  1. State whether the Central Limit Theorem applies and specify the sampling distribution of the sample mean .
  2. Calculate the probability that the sample mean lifetime falls between and , i.e., .

Evaluating Applicability of CLT and Finding Standard Error: The sample size is . Since , the Central Limit Theorem applies regardless of whether the original population distribution is normal, skewed, or uniform.

The sampling distribution of the sample mean is approximately normal:

Evaluating the mean and standard error:

Standardizing the Bounds and Computing Probability: To find , we convert both boundary values into standard normal -scores:

Using the standard normal cumulative distribution function :

Answer:

Because , the CLT applies with and standard error . The probability that the sample mean falls between and is approximately (or ).

Example 2.3

2.3

A server experiences independent background jobs where each job succeeds with probability . In a batch of jobs, let denote the number of successful jobs ().

  1. Check the conditions for normal approximation.
  2. Using the continuity-corrected normal approximation, find the probability that between and jobs succeed inclusive, i.e., .

Checking Applicability Conditions and Finding Distribution Parameters: For a Binomial random variable with and :

Both conditions are well above , confirming that the normal distribution provides a reliable approximation.

Parameters of the approximating normal variable :

Applying Continuity Correction and Standardizing: Because the discrete integer range is , applying the continuity correction broadens the boundaries by on both sides:

Standardizing the endpoints into standard normal -scores:

Using the standard normal cumulative distribution function :

Answer:

The normal approximation conditions are satisfied (). With continuity correction, (or ).

Exercises

Exercises

Question 1

Define inferential statistics and provide a concrete practical example of its application in modern data science.

Formal Definition: Inferential statistics is the branch of mathematical statistics that deduces properties, estimates parameters, and tests hypotheses regarding an underlying population based on the empirical observations collected in a representative sample.

Practical Application in Data Science: Consider an e-commerce platform with active users. The engineering team redesigns the checkout page to reduce friction. Surveying or testing all users simultaneously is impractical and risks platform-wide revenue loss.

Instead, the team exposes a randomly sampled treatment cohort of users to the new interface and keeps users on the existing interface (A/B testing). Using inferential statistics, the team calculates the conversion rates, constructs confidence intervals for the lift in conversion, and performs a two-sample hypothesis test to confirm whether the observed improvement reflects a genuine population-wide trend rather than random sample variation.

Answer:

Inferential statistics uses sample data to make generalized predictions and test hypotheses about an unobserved population. A classic data science application is A/B testing, where metrics from a sample cohort are analyzed to infer whether a new feature genuinely improves engagement across the entire user base.

Question 2

Contrast population parameters and sample statistics. Present the standard mathematical notation for population size, mean, variance, and standard deviation alongside their sample counterparts.

Conceptual Distinction: A population parameter is a fixed numerical characteristic describing the complete population. Because populations are typically vast, parameters are usually unknown constants.

A sample statistic is a numerical value computed directly from the observed sample data. Because samples vary randomly, statistics are random variables that serve as estimators of the corresponding population parameters.

Comparison of Notation:

MetricPopulation Parameter (Greek)Sample Statistic (Latin)
Total Size / Count
Mean
Variance
Standard Deviation
Proportion or
Answer:

Population parameters () describe the complete population and are constant but generally unknown. Sample statistics () describe a collected subset, vary between samples, and serve as estimators for population parameters.

Question 3

Explain the concept of estimation in inferential statistics, distinguish between point estimation and interval estimation, and define confidence level.

Point Estimation vs Interval Estimation: Estimation is the process of using sample statistics to approximate unknown population parameters:

  1. Point Estimation: A point estimator calculates a single numerical value to approximate the parameter. For example, using the sample mean as the single estimate for the population mean height . While concise, a point estimate gives no indication of uncertainty or margin of error.
  2. Interval Estimation: An interval estimator constructs a range of plausible values within which the parameter is expected to lie with a specified degree of confidence:

For instance, estimating that the population mean height is between and .

Confidence Level: The confidence level, denoted , is the long-run proportion of constructed intervals that will contain the true population parameter if independent random sampling were repeated indefinitely under identical conditions. A confidence level means that in out of independent random samples, the resulting interval will successfully enclose the true parameter .

Answer:

Point estimation provides a single best-guess value ( for ), whereas interval estimation provides a range that captures uncertainty. The confidence level quantifies the long-run probability that repeated independent sample intervals enclose the true population parameter.

Question 4

Prepare a technical note on the Central Limit Theorem (CLT). State the formal theorem, list its necessary assumptions, and explain why it is considered fundamental in data analysis.

Formal Theorem Statement: Let be independent and identically distributed (i.i.d.) random variables drawn from an arbitrary population distribution with mean and finite variance . As , the distribution of the sample mean converges in distribution to a normal distribution:

Necessary Assumptions:

  1. Independence: Sampling must be random and independent (or sample size if sampling without replacement).
  2. Finite Variance: The second moment must be finite (). Distributions with heavy power-law tails and undefined variance (such as the Cauchy distribution) fail the CLT.
  3. Adequate Sample Size: For moderately symmetric distributions, is sufficient; for severely skewed distributions, or is required.

Why it is Fundamental in Data Analysis:

  1. It permits parametric inferences (-tests, -tests, confidence intervals) on real-world datasets without requiring normality in the underlying population.
  2. It underpins hypothesis testing in A/B experimentation and controlled machine learning model comparisons.
  3. It validates treating aggregate measurement errors and regression residuals as normally distributed noise .
Answer:

The Central Limit Theorem asserts that the sample mean of i.i.d. observations with finite variance converges to a normal distribution regardless of parent population shape. It is vital because it enables parametric hypothesis testing, confidence interval derivation, and error modeling on non-Gaussian data.

Question 5

The assessed tax values of all registered vehicles in a state have a population mean of and a population standard deviation of . Suppose random samples of size vehicles are drawn repeatedly.

  1. Determine the mean and the standard error of the sample mean .
  2. State the approximate distribution of and justify your answer.

Calculating Mean and Standard Error of the Sampling Distribution: By the unbiasedness of the sample mean:

The standard error of the sample mean is:

Determining the Distribution: Even though vehicle values in the real world are positively skewed (with many inexpensive cars and a few high-value luxury cars), the sample size is , which significantly exceeds the threshold of .

By the Central Limit Theorem, the sampling distribution of is approximately normal:

Answer:

The mean of the sample mean is and the standard error is . By the Central Limit Theorem (), follows an approximately normal distribution .

Question 6

An electronic hardware manufacturing facility produces precision resistors with a nominal mean resistance of and standard deviation . The resistance values are known to follow a normal distribution. If a quality control engineer inspects a random sample of resistors, what is the probability that the sample mean resistance is less than ?

Evaluating Sampling Distribution Properties: Because the parent population is normally distributed, the sampling distribution of is exactly normal for any sample size :

Mean and standard error:

Standardizing and Computing Probability: To find , we compute the standard normal -score:

Looking up in the standard normal distribution table:

Answer:

The standard error of the sample mean is , giving a -score of . The probability of finding a sample mean resistance less than is approximately (or ).

Question 7

A balanced coin is tossed times independently. Let represent the number of heads obtained (). Using the normal approximation with continuity correction, determine the probability of obtaining between and heads inclusive, i.e., .

Verifying Conditions and Deriving Parameters: For and :

The conditions for the normal approximation are amply satisfied.

Parameters of the approximating normal distribution:

Applying Continuity Correction and Calculating Probability: Converting the discrete range into continuous bounds:

Standardizing the continuous endpoints:

Using the cumulative normal distribution:

Answer:

Using the normal approximation () with continuity correction, the interval is (), yielding a probability of approximately (or ).

Question 8

Define the term "Sampling distribution". [Assignment 2, Q17]

Formal Definition: A sampling distribution is the probability distribution of a sample statistic (such as the sample mean , sample variance , or sample proportion ) computed across all possible independent random samples of the same fixed size drawn from a given population.

Key Properties for the Sample Mean : When random samples of size are drawn from a population with mean and standard deviation :

  1. Unbiasedness (Expected Value): The mean of the sampling distribution of equals the population mean:
  1. Standard Error of the Mean: The standard deviation of the sampling distribution measures sample-to-sample variability and decreases with :
  1. Asymptotic Normality: By the Central Limit Theorem, as increases (), the sampling distribution of approaches regardless of the population's shape.
Answer:

A sampling distribution is the probability distribution of a statistic (such as the mean or variance ) calculated from all possible samples of the same size drawn from a population. For the sample mean , its mean is and its standard error is .

Question 9

Differentiate between Descriptive Statistics and Inferential Statistics with illustrative examples. [Assignment, notebook p236]

Conceptual Overview:

  • Descriptive Statistics focuses on organizing, summarizing, and presenting the observed characteristics of a collected dataset without making generalizations beyond it.
  • Inferential Statistics uses sample data and probability theory to make generalizations, estimations, predictions, or decisions about a larger population.

Point-by-Point Comparison:

Basis of ComparisonDescriptive StatisticsInferential Statistics
Core FunctionDescribes and summarizes collected dataMakes predictions, estimations, or conclusions
ObjectiveTo understand the given dataset at handTo generalize findings about a larger population
Data UsedUses the complete observed datasetUses random sample data () drawn from a population ()
Nature of OutputReports exact facts (mean , median, charts) with no guessingProvides estimates (confidence intervals) or hypothesis decisions with quantified uncertainty
Illustrative ExampleThe average marks of a class of students is Inferring from a sample that many students across the university will score above
Answer:

Descriptive statistics summarizes and describes the exact dataset collected (e.g., reporting that the average mark in a class is ), whereas inferential statistics uses sample data to generalize, estimate parameters (), and make predictions or decisions about a larger population.

Past exam questions for Foundation of Data Science

← Probability, Random Variables & DistributionsHypothesis Testing, z/t-Tests & Confidence Intervals →