PadhaikoAll subjects

Hypothesis Testing, z/t-Tests & Confidence Intervals

Foundation of Data Science

Open in the Padhaiko app

2.8 Confidence Interval and Hypothesis Testing

In scientific investigation and data-driven engineering, researchers rarely accept assertions without quantitative verification. Hypothesis testing is a formal statistical framework for determining whether empirical evidence from a sample supports a specific conjecture about a population parameter, or whether the observed effect could simply be an artifact of random sampling chance.

Definition

Null and Alternative Hypotheses

Every statistical test evaluates two mutually exclusive and collectively exhaustive hypotheses:

  1. Null Hypothesis (): The baseline assumption that no real difference, effect, or relationship exists. It represents the status quo and is presumed true until sample evidence decisively contradicts it.
    • Example for a population mean: (e.g., "The mean weight of candy bars is ").
  2. Alternative Hypothesis ( or ): The research hypothesis stating that a meaningful difference, effect, or departure from the baseline exists. Depending on the direction of interest, the alternative hypothesis takes one of three forms:
    • Two-Tailed Test (Non-Directional):

Detects any significant difference in either direction (larger or smaller).

Detects whether the population parameter is strictly greater than the hypothesized baseline.

Detects whether the population parameter is strictly less than the hypothesized baseline.

Definition

Type I and Type II Errors

Because decisions are based on sample statistics rather than exhaustive census enumeration, errors can occur:

  1. Type I Error (): Rejecting the null hypothesis when it is actually true. The probability of committing a Type I error is the significance level (commonly set by convention to , , or ).
  1. Type II Error (): Failing to reject the null hypothesis when is false (i.e., the alternative is true).
  1. Power of a Test (): The probability of correctly rejecting a false null hypothesis. High power is desirable in machine learning experiments and A/B testing to ensure genuine improvements are reliably detected.
Decision Matrix: True State of Nature vs Test Decision
Statistical DecisionNull Hypothesis is Actually TrueNull Hypothesis is Actually False
Fail to Reject Correct Decision (Confidence )Type II Error (Probability )
Reject Type I Error (Significance level )Correct Decision (Statistical Power )
Definition

p-Value and Decision Rules

The -value is the probability, assuming that the null hypothesis is strictly true, of observing a test statistic at least as extreme as the one computed from the sample data:

The -value quantifies the strength of empirical evidence against :

AspectZ-TestStudent's t-Test
Population Variance Known (or by CLT)Unknown (estimated via sample variance )
Sample SizeLarge ()Typically small (), though valid for any
Assumed Parent DistributionAny distribution with finite variance (CLT applies)Must be approximately normal for small
Test Statistic Formula
Reference DistributionStandard Normal distribution Student's -distribution with degrees of freedom
Tail BehaviorFixed tailsHeavier tails that widen as degrees of freedom decrease

2.9 Hypothesis Testing for Differences Between Two Populations Means, The Two-Sample t-Test, Confidence Interval for Mean of Normal

Statistical inference extends naturally to constructing interval estimates and comparing two distinct populations, which is the mathematical foundation of A/B testing and comparative model evaluation.

Definition

Confidence Interval for Mean of Normal Population

A confidence interval for the unknown population mean provides an estimated range derived from sample data that is expected to contain with confidence level :

  1. Large Sample or Known Population Variance (-Interval): When the population standard deviation is known, or when the sample size allowing :

Standard critical values from the normal distribution:

  1. Small Sample with Unknown Population Variance (-Interval): When , is unknown, and the population is approximately normal:

where is the critical value from the Student's -distribution with degrees of freedom.

Definition

Single Population Mean Tests: Z-Test and t-Test

To test whether a single population mean equals a hypothesized baseline :

  1. One-Sample -Test ( or known ):
  1. One-Sample Student's -Test (, unknown , normal population):
Definition

Hypothesis Testing for Differences Between Two Population Means

In comparative studies, we evaluate whether two independent populations have identical means:

  1. Large Independent Samples ( and , -Test): By the Central Limit Theorem, the difference between sample means is approximately normal:

Reject at level if .

  1. Small Independent Samples with Equal Variances (Two-Sample Pooled -Test): When , populations are normal, and we can reasonably assume equal variances (), we combine both sample variances into a single pooled sample variance :

The two-sample pooled -statistic is:

Reject if .

Worked Out Examples

Example 2.4

2.4

A random sample of employees of an industrial concern yields a sample mean monthly income of . The standard deviation of the population is known to be .

  1. Compute the standard error of the sample mean.
  2. At the significance level, test whether the sample mean differs significantly from the claimed population mean of .

Computing the Standard Error: The sample size is and the population standard deviation is :

Formulating Hypotheses: We test whether the mean income differs from Rs. 2850 (a non-directional, two-tailed test):

  • Null Hypothesis: (mean employee income does not differ from Rs. 2850)
  • Alternative Hypothesis: (mean employee income differs from Rs. 2850)

Calculating the Test Statistic: Because is known and , we use the one-sample -test:

Determining the Critical Value and Drawing Conclusion: For a two-tailed test at significance level :

Rejection condition: Reject if . Here, .

Decision: Reject . There is overwhelming statistical evidence that the true mean monthly income of the employees differs significantly from Rs. 2850.

Answer:

The standard error is . The test statistic is . Because at , the null hypothesis is decisively rejected; the mean income differs significantly from Rs. 2850.

Example 2.5

2.5

The average nightly cost of a hotel room in New York City is advertised as . A travel analytics team surveys a random sample of hotels and records a sample mean cost of with a sample standard deviation of . Assuming hotel rates follow an approximately normal distribution, test at the significance level whether the true mean cost differs from .

Formulating Hypotheses: We conduct a two-tailed test:

  • Null Hypothesis:
  • Alternative Hypothesis:

Selecting the Test and Calculating the Test Statistic: The sample size , the population standard deviation is unknown, and the population is approximately normal. Therefore, a one-sample Student's -test is appropriate.

Degrees of freedom:

Standard error of the sample mean:

Test statistic:

Critical Value and Decision: For a two-tailed -test with at :

Rejection condition: Reject if . Here, .

Decision: Fail to reject . There is insufficient evidence at the level to conclude that the true mean nightly hotel room cost differs from .

Answer:

With degrees of freedom , the computed test statistic is and the two-tailed critical threshold is . Because , we fail to reject ; the sample provides insufficient evidence to conclude the true mean room cost differs from .

Example 2.6

2.6

Two independent groups of students, Group A and Group B, complete a comprehensive data science assessment:

Assuming population variances are unknown, test at the significance level whether there is a statistically significant difference between the population mean scores of the two groups ( versus ).

Formulating Hypotheses:

  • Null Hypothesis: (no difference in mean scores)
  • Alternative Hypothesis: (significant difference in mean scores)

Selecting the Test and Calculating the Test Statistic: Both sample sizes are large ( and ). By the Central Limit Theorem, the sampling distribution of is approximately normal. We apply the two-sample large-sample -test.

The standard error of the difference between two sample means:

Calculating the -statistic:

Determining the Critical Value and Reaching a Decision: For a two-tailed test at significance level :

Rejection region: . Here, (the test statistic lies in the critical rejection region).

Decision: Reject . There is statistically significant evidence at the level that the population mean scores of Group A and Group B are different (Group B scored significantly higher).

Answer:

The standard error of the difference in means is , resulting in a test statistic of . Since , we reject ; there is a statistically significant difference between the two student cohorts.

Example 2.7

2.7

Construct confidence intervals for the following two independent scenarios:

  1. An IT enterprise surveys software developers and records an average daily programming time of with a standard deviation of . Construct a confidence interval for the true population mean daily programming time.
  2. A machine learning class evaluates a random sample of student submissions, finding a mean score of with sample standard deviation . Construct a confidence interval for the true population mean score, given .

Scenario 1: Large Sample () Confidence Interval: Because , the sampling distribution of is normal by the CLT. For a confidence level (), the critical two-tailed value is .

Margin of error:

Confidence interval:

With confidence, the true mean coding time for all developers is between and hours per day.

Scenario 2: Small Sample () Confidence Interval: Because and is unknown, we use the Student's -distribution with . The critical value at confidence is .

Margin of error:

Confidence interval:

With confidence, the true population mean exam score is between and .

Answer:

For the large sample of developers (), the confidence interval is . For the small sample of students (), the confidence interval using is .

Exercises

Exercises

Question 1

Explain the meaning of null and alternative hypotheses for a test about a population mean. Distinguish between a two-tailed test and a one-tailed test, giving an illustrative practical scenario for each.

Null vs Alternative Hypotheses:

  • The null hypothesis () represents the default status quo statement that no change, effect, or deviation exists: . It is assumed true unless empirical sample evidence warrants rejection.
  • The alternative hypothesis () represents the research claim that a meaningful difference or directional change exists.

Two-Tailed vs One-Tailed Tests:

  1. One-Tailed (Directional) Test: Used when researchers are specifically interested in an effect in only one direction (increase or decrease):
    • Right-tailed: versus .
    • Left-tailed: versus .
    • Scenario: A manufacturing engineer introduces a new lubricant to extend bearing lifespan. The test only cares if lifespan increases ().
  2. Two-Tailed (Non-Directional) Test: Used when any deviation from the baseline in either direction is consequential:
    • Hypotheses: versus .
    • Scenario: A pharmaceutical lab fills ampoules with of medication. Overfilling wastes expensive drug product, while underfilling leads to under-dosing. Both deviations must be detected ().
Answer:

The null hypothesis () posits no effect (), while the alternative () posits an effect. A one-tailed test detects deviations in a single direction ( or , e.g., testing if an optimization speeds up execution), whereas a two-tailed test detects any difference in either direction (, e.g., quality control checking vial volume).

Question 2

Interpret the -value and explain its significance in statistical hypothesis testing. Why does failing to reject the null hypothesis not prove that the null hypothesis is true?

Interpretation of the -Value: The -value is the probability, calculated under the assumption that is true, of observing a test statistic as extreme as, or more extreme than, the value obtained from the sample data.

  • A small -value () indicates that the observed sample outcome is very improbable under , leading to the rejection of .
  • A large -value () indicates that the observed data is readily compatible with random variation under .

Why "Failing to Reject" Does Not Prove Truth: Failing to reject simply means that the empirical data does not supply sufficient contradictory evidence at the chosen significance level . It may be that a genuine effect exists, but the sample size was too small to detect it (low statistical power).

In legal terms, an acquittal verdict states "not guilty" (insufficient evidence to convict beyond a reasonable doubt), never "proven innocent". In the same manner, statistical testing never claims that has been proven true.

Answer:

The -value measures the probability of obtaining the observed data assuming holds; if , is rejected. Failing to reject does not prove is true, because low sample size or high variance may conceal a real effect; it merely reflects absence of sufficient contradictory evidence.

Question 3

Tabulate the primary differences between a -test and Student's -test for testing hypotheses about a population mean.

Tabulating the Differences:

Feature-TestStudent's -Test
Population Variance ()Known, or sample size is large ()Unknown and estimated by sample variance
Sample Size RequirementLarge ()Typically small (), but valid for large
Distributional AssumptionNo strict parent normality needed if (CLT)Parent population must be approximately normal
Test Statistic
Critical ValuesConstant critical thresholds ( at )Depend on degrees of freedom
Tail ThicknessThin Gaussian tailsThicker tails accounting for extra variance from estimating
Answer:

A -test is used when population standard deviation is known or sample size , comparing against the standard normal curve. A -test is used when is unknown and estimated by with , comparing against Student's -distribution with .

Question 4

A trucking logistics firm is suspicious of a supplier's claim that the average lifetime of a commercial tire model is at least . To evaluate this claim, the firm tests a random sample of tires and obtains a sample mean lifetime of with a sample standard deviation of . What should the firm conclude if the maximum allowable probability of Type I error is ?

Formulating Hypotheses: The firm's suspicion is that the mean lifetime is strictly less than (a left-tailed test):

  • Null Hypothesis: (supplier's claim holds)
  • Alternative Hypothesis: (tires wear out faster than claimed)

Computing the Test Statistic: Since , the Central Limit Theorem applies and we use the one-sample -test with :

Critical Value and Decision: For a left-tailed test at significance level :

Rejection rule: Reject if . Because , the test statistic falls deep within the critical rejection region.

Decision: Reject at the significance level. The firm's suspicion is statistically confirmed: the average tire lifetime is significantly less than .

Answer:

The left-tailed test yields standard error and test statistic . Because at , the null hypothesis is rejected; the trucking firm's suspicion is justified.

Question 5

The breaking strength of industrial cables manufactured by a traditional process has a known mean of and a population standard deviation of . A newly developed metallurgical treatment is claimed to increase cable breaking strength. A test sample of treated cables yields a sample mean breaking strength of . Does the empirical evidence support the manufacturer's claim at the level of significance?

Formulating Hypotheses: The manufacturer claims that the new process increases breaking strength (a right-tailed test):

  • Null Hypothesis:
  • Alternative Hypothesis:

Computing the Test Statistic: The population standard deviation is known () and . We apply the one-sample -test:

Determining Critical Value and Conclusion: For a right-tailed test at significance level :

Rejection rule: Reject if . Because , the test statistic falls well into the upper rejection region.

Decision: Reject . There is decisive evidence at the significance level supporting the claim that the new metallurgical process significantly increases cable breaking strength.

Answer:

The right-tailed test gives standard error and test statistic . Because at , is rejected; the empirical evidence supports the claim that the new process increases breaking strength.

Question 6

Two search ranking algorithms, Algorithm A and Algorithm B, are evaluated on latency across small independent benchmark query workloads:

Assuming both parent populations are approximately normally distributed with equal variances, perform a two-sample pooled Student's -test at the significance level to test against , given .

Formulating Hypotheses and Degrees of Freedom:

  • Null Hypothesis:
  • Alternative Hypothesis:

Degrees of freedom:

Calculating Pooled Variance and Standard Error: The pooled sample variance combines both sample variances weighted by their degrees of freedom:

Standard error of the difference in sample means:

Calculating Test Statistic and Reaching a Decision:

Critical value for two-tailed test at with :

Rejection rule: Reject if . Here, .

Decision: Reject . There is a statistically significant difference between the mean latencies of Algorithm A and Algorithm B at the significance level.

Answer:

The pooled standard deviation is , standard error is , and the test statistic is . Since , we reject ; Algorithm A has significantly lower latency than Algorithm B.

Question 7

A quality assurance test on IoT sensor batteries records an average continuous operating lifespan of with a sample standard deviation of .

  1. Construct a confidence interval for the true population mean operating lifespan ().
  2. Construct a confidence interval for the same metric ().
  3. State why the confidence interval is wider than the interval.

Computing the Standard Error: With , the CLT ensures that the sample mean distribution is approximately normal:

Constructing 90% and 99% Confidence Intervals:

  1. Confidence Interval: Margin of error: .
  1. Confidence Interval: Margin of error: .

Explanation of Interval Width vs Confidence Level: A higher confidence level requires a wider interval because greater certainty requires a broader net to capture the true parameter. To be sure of enclosing , the critical multiplier increases from to , increasing the margin of error from to .

Answer:

The confidence interval is and the confidence interval is . The interval is wider because higher certainty requires a larger margin of error to minimize the risk of missing the true parameter.

Question 8

Define Type I error and Type II error in hypothesis testing, and state their probabilities and practical interpretations. [Assignment, notebook p236]

Definitions of Type I and Type II Errors: In statistical hypothesis testing, two types of decision errors can occur when evaluating a null hypothesis :

  1. Type I Error (False Positive): A Type I error occurs when we reject a true null hypothesis ( is actually true, but the test rejects it).
    • Its probability is denoted by (the level of significance, commonly or ):
  1. Type II Error (False Negative): A Type II error occurs when we fail to reject a false null hypothesis ( is actually false and is true, but the test fails to detect the effect).
    • Its probability is denoted by :
  • The complement is the statistical power of the test—the probability of correctly rejecting a false null hypothesis.

Summary Decision Table:

Test Decision is True is False
Reject Type I Error (probability )Correct Decision (Power )
Fail to Reject Correct Decision (probability )Type II Error (probability )
Answer:

A Type I error () occurs when a true null hypothesis is rejected (false positive). A Type II error () occurs when a false null hypothesis is not rejected (false negative).

Past exam questions for Foundation of Data Science

← Descriptive & Inferential Statistics, CLT & SamplingAnalysis of Variance (ANOVA) →