Hypothesis Testing, z/t-Tests & Confidence Intervals
Open in the Padhaiko app2.8 Confidence Interval and Hypothesis Testing
In scientific investigation and data-driven engineering, researchers rarely accept assertions without quantitative verification. Hypothesis testing is a formal statistical framework for determining whether empirical evidence from a sample supports a specific conjecture about a population parameter, or whether the observed effect could simply be an artifact of random sampling chance.
Null and Alternative Hypotheses
Every statistical test evaluates two mutually exclusive and collectively exhaustive hypotheses:
- Null Hypothesis (): The baseline assumption that no real difference, effect, or relationship exists. It represents the status quo and is presumed true until sample evidence decisively contradicts it.
- Example for a population mean: (e.g., "The mean weight of candy bars is ").
- Alternative Hypothesis ( or ): The research hypothesis stating that a meaningful difference, effect, or departure from the baseline exists. Depending on the direction of interest, the alternative hypothesis takes one of three forms:
- Two-Tailed Test (Non-Directional):
Detects any significant difference in either direction (larger or smaller).
- Right-Tailed Test (Directional / Upper-Tailed):
Detects whether the population parameter is strictly greater than the hypothesized baseline.
- Left-Tailed Test (Directional / Lower-Tailed):
Detects whether the population parameter is strictly less than the hypothesized baseline.
Type I and Type II Errors
Because decisions are based on sample statistics rather than exhaustive census enumeration, errors can occur:
- Type I Error (): Rejecting the null hypothesis when it is actually true. The probability of committing a Type I error is the significance level (commonly set by convention to , , or ).
- Legal Analogy: Convicting an innocent defendant (a false positive).
- Type II Error (): Failing to reject the null hypothesis when is false (i.e., the alternative is true).
- Legal Analogy: Acquitting a guilty defendant (a false negative).
- Power of a Test (): The probability of correctly rejecting a false null hypothesis. High power is desirable in machine learning experiments and A/B testing to ensure genuine improvements are reliably detected.
| Statistical Decision | Null Hypothesis is Actually True | Null Hypothesis is Actually False |
|---|---|---|
| Fail to Reject | Correct Decision (Confidence ) | Type II Error (Probability ) |
| Reject | Type I Error (Significance level ) | Correct Decision (Statistical Power ) |
p-Value and Decision Rules
The -value is the probability, assuming that the null hypothesis is strictly true, of observing a test statistic at least as extreme as the one computed from the sample data:
The -value quantifies the strength of empirical evidence against :
- Decision Rule:
- If : Reject in favor of . The result is statistically significant.
- If : Fail to reject . There is insufficient evidence to conclude the alternative.
- Key Caveat: Failing to reject does not prove that is true; it only indicates that the data does not provide sufficient contrary evidence. For this reason, statisticians never state that they "accept" the null hypothesis.
| Aspect | Z-Test | Student's t-Test |
|---|---|---|
| Population Variance | Known (or by CLT) | Unknown (estimated via sample variance ) |
| Sample Size | Large () | Typically small (), though valid for any |
| Assumed Parent Distribution | Any distribution with finite variance (CLT applies) | Must be approximately normal for small |
| Test Statistic Formula | ||
| Reference Distribution | Standard Normal distribution | Student's -distribution with degrees of freedom |
| Tail Behavior | Fixed tails | Heavier tails that widen as degrees of freedom decrease |
2.9 Hypothesis Testing for Differences Between Two Populations Means, The Two-Sample t-Test, Confidence Interval for Mean of Normal
Statistical inference extends naturally to constructing interval estimates and comparing two distinct populations, which is the mathematical foundation of A/B testing and comparative model evaluation.
Confidence Interval for Mean of Normal Population
A confidence interval for the unknown population mean provides an estimated range derived from sample data that is expected to contain with confidence level :
- Large Sample or Known Population Variance (-Interval): When the population standard deviation is known, or when the sample size allowing :
Standard critical values from the normal distribution:
- Confidence ():
- Confidence ():
- Confidence ():
- Small Sample with Unknown Population Variance (-Interval): When , is unknown, and the population is approximately normal:
where is the critical value from the Student's -distribution with degrees of freedom.
Single Population Mean Tests: Z-Test and t-Test
To test whether a single population mean equals a hypothesized baseline :
- One-Sample -Test ( or known ):
- Rejection Rule for Two-Tailed Test at level : Reject if .
- Rejection Rule for Right-Tailed Test: Reject if .
- Rejection Rule for Left-Tailed Test: Reject if .
- One-Sample Student's -Test (, unknown , normal population):
- Rejection Rule for Two-Tailed Test: Reject if .
Hypothesis Testing for Differences Between Two Population Means
In comparative studies, we evaluate whether two independent populations have identical means:
- Large Independent Samples ( and , -Test): By the Central Limit Theorem, the difference between sample means is approximately normal:
Reject at level if .
- Small Independent Samples with Equal Variances (Two-Sample Pooled -Test): When , populations are normal, and we can reasonably assume equal variances (), we combine both sample variances into a single pooled sample variance :
The two-sample pooled -statistic is:
Reject if .
Worked Out Examples
2.4
A random sample of employees of an industrial concern yields a sample mean monthly income of . The standard deviation of the population is known to be .
- Compute the standard error of the sample mean.
- At the significance level, test whether the sample mean differs significantly from the claimed population mean of .
Computing the Standard Error: The sample size is and the population standard deviation is :
Formulating Hypotheses: We test whether the mean income differs from Rs. 2850 (a non-directional, two-tailed test):
- Null Hypothesis: (mean employee income does not differ from Rs. 2850)
- Alternative Hypothesis: (mean employee income differs from Rs. 2850)
Calculating the Test Statistic: Because is known and , we use the one-sample -test:
Determining the Critical Value and Drawing Conclusion: For a two-tailed test at significance level :
Rejection condition: Reject if . Here, .
Decision: Reject . There is overwhelming statistical evidence that the true mean monthly income of the employees differs significantly from Rs. 2850.
The standard error is . The test statistic is . Because at , the null hypothesis is decisively rejected; the mean income differs significantly from Rs. 2850.
2.5
The average nightly cost of a hotel room in New York City is advertised as . A travel analytics team surveys a random sample of hotels and records a sample mean cost of with a sample standard deviation of . Assuming hotel rates follow an approximately normal distribution, test at the significance level whether the true mean cost differs from .
Formulating Hypotheses: We conduct a two-tailed test:
- Null Hypothesis:
- Alternative Hypothesis:
Selecting the Test and Calculating the Test Statistic: The sample size , the population standard deviation is unknown, and the population is approximately normal. Therefore, a one-sample Student's -test is appropriate.
Degrees of freedom:
Standard error of the sample mean:
Test statistic:
Critical Value and Decision: For a two-tailed -test with at :
Rejection condition: Reject if . Here, .
Decision: Fail to reject . There is insufficient evidence at the level to conclude that the true mean nightly hotel room cost differs from .
With degrees of freedom , the computed test statistic is and the two-tailed critical threshold is . Because , we fail to reject ; the sample provides insufficient evidence to conclude the true mean room cost differs from .
2.6
Two independent groups of students, Group A and Group B, complete a comprehensive data science assessment:
- Group A: , sample mean , sample standard deviation .
- Group B: , sample mean , sample standard deviation .
Assuming population variances are unknown, test at the significance level whether there is a statistically significant difference between the population mean scores of the two groups ( versus ).
Formulating Hypotheses:
- Null Hypothesis: (no difference in mean scores)
- Alternative Hypothesis: (significant difference in mean scores)
Selecting the Test and Calculating the Test Statistic: Both sample sizes are large ( and ). By the Central Limit Theorem, the sampling distribution of is approximately normal. We apply the two-sample large-sample -test.
The standard error of the difference between two sample means:
Calculating the -statistic:
Determining the Critical Value and Reaching a Decision: For a two-tailed test at significance level :
Rejection region: . Here, (the test statistic lies in the critical rejection region).
Decision: Reject . There is statistically significant evidence at the level that the population mean scores of Group A and Group B are different (Group B scored significantly higher).
The standard error of the difference in means is , resulting in a test statistic of . Since , we reject ; there is a statistically significant difference between the two student cohorts.
2.7
Construct confidence intervals for the following two independent scenarios:
- An IT enterprise surveys software developers and records an average daily programming time of with a standard deviation of . Construct a confidence interval for the true population mean daily programming time.
- A machine learning class evaluates a random sample of student submissions, finding a mean score of with sample standard deviation . Construct a confidence interval for the true population mean score, given .
Scenario 1: Large Sample () Confidence Interval: Because , the sampling distribution of is normal by the CLT. For a confidence level (), the critical two-tailed value is .
Margin of error:
Confidence interval:
With confidence, the true mean coding time for all developers is between and hours per day.
Scenario 2: Small Sample () Confidence Interval: Because and is unknown, we use the Student's -distribution with . The critical value at confidence is .
Margin of error:
Confidence interval:
With confidence, the true population mean exam score is between and .
For the large sample of developers (), the confidence interval is . For the small sample of students (), the confidence interval using is .
Exercises
Exercises
Explain the meaning of null and alternative hypotheses for a test about a population mean. Distinguish between a two-tailed test and a one-tailed test, giving an illustrative practical scenario for each.
Null vs Alternative Hypotheses:
- The null hypothesis () represents the default status quo statement that no change, effect, or deviation exists: . It is assumed true unless empirical sample evidence warrants rejection.
- The alternative hypothesis () represents the research claim that a meaningful difference or directional change exists.
Two-Tailed vs One-Tailed Tests:
- One-Tailed (Directional) Test: Used when researchers are specifically interested in an effect in only one direction (increase or decrease):
- Right-tailed: versus .
- Left-tailed: versus .
- Scenario: A manufacturing engineer introduces a new lubricant to extend bearing lifespan. The test only cares if lifespan increases ().
- Two-Tailed (Non-Directional) Test: Used when any deviation from the baseline in either direction is consequential:
- Hypotheses: versus .
- Scenario: A pharmaceutical lab fills ampoules with of medication. Overfilling wastes expensive drug product, while underfilling leads to under-dosing. Both deviations must be detected ().
The null hypothesis () posits no effect (), while the alternative () posits an effect. A one-tailed test detects deviations in a single direction ( or , e.g., testing if an optimization speeds up execution), whereas a two-tailed test detects any difference in either direction (, e.g., quality control checking vial volume).
Interpret the -value and explain its significance in statistical hypothesis testing. Why does failing to reject the null hypothesis not prove that the null hypothesis is true?
Interpretation of the -Value: The -value is the probability, calculated under the assumption that is true, of observing a test statistic as extreme as, or more extreme than, the value obtained from the sample data.
- A small -value () indicates that the observed sample outcome is very improbable under , leading to the rejection of .
- A large -value () indicates that the observed data is readily compatible with random variation under .
Why "Failing to Reject" Does Not Prove Truth: Failing to reject simply means that the empirical data does not supply sufficient contradictory evidence at the chosen significance level . It may be that a genuine effect exists, but the sample size was too small to detect it (low statistical power).
In legal terms, an acquittal verdict states "not guilty" (insufficient evidence to convict beyond a reasonable doubt), never "proven innocent". In the same manner, statistical testing never claims that has been proven true.
The -value measures the probability of obtaining the observed data assuming holds; if , is rejected. Failing to reject does not prove is true, because low sample size or high variance may conceal a real effect; it merely reflects absence of sufficient contradictory evidence.
Tabulate the primary differences between a -test and Student's -test for testing hypotheses about a population mean.
Tabulating the Differences:
| Feature | -Test | Student's -Test |
|---|---|---|
| Population Variance () | Known, or sample size is large () | Unknown and estimated by sample variance |
| Sample Size Requirement | Large () | Typically small (), but valid for large |
| Distributional Assumption | No strict parent normality needed if (CLT) | Parent population must be approximately normal |
| Test Statistic | ||
| Critical Values | Constant critical thresholds ( at ) | Depend on degrees of freedom |
| Tail Thickness | Thin Gaussian tails | Thicker tails accounting for extra variance from estimating |
A -test is used when population standard deviation is known or sample size , comparing against the standard normal curve. A -test is used when is unknown and estimated by with , comparing against Student's -distribution with .
A trucking logistics firm is suspicious of a supplier's claim that the average lifetime of a commercial tire model is at least . To evaluate this claim, the firm tests a random sample of tires and obtains a sample mean lifetime of with a sample standard deviation of . What should the firm conclude if the maximum allowable probability of Type I error is ?
Formulating Hypotheses: The firm's suspicion is that the mean lifetime is strictly less than (a left-tailed test):
- Null Hypothesis: (supplier's claim holds)
- Alternative Hypothesis: (tires wear out faster than claimed)
Computing the Test Statistic: Since , the Central Limit Theorem applies and we use the one-sample -test with :
Critical Value and Decision: For a left-tailed test at significance level :
Rejection rule: Reject if . Because , the test statistic falls deep within the critical rejection region.
Decision: Reject at the significance level. The firm's suspicion is statistically confirmed: the average tire lifetime is significantly less than .
The left-tailed test yields standard error and test statistic . Because at , the null hypothesis is rejected; the trucking firm's suspicion is justified.
The breaking strength of industrial cables manufactured by a traditional process has a known mean of and a population standard deviation of . A newly developed metallurgical treatment is claimed to increase cable breaking strength. A test sample of treated cables yields a sample mean breaking strength of . Does the empirical evidence support the manufacturer's claim at the level of significance?
Formulating Hypotheses: The manufacturer claims that the new process increases breaking strength (a right-tailed test):
- Null Hypothesis:
- Alternative Hypothesis:
Computing the Test Statistic: The population standard deviation is known () and . We apply the one-sample -test:
Determining Critical Value and Conclusion: For a right-tailed test at significance level :
Rejection rule: Reject if . Because , the test statistic falls well into the upper rejection region.
Decision: Reject . There is decisive evidence at the significance level supporting the claim that the new metallurgical process significantly increases cable breaking strength.
The right-tailed test gives standard error and test statistic . Because at , is rejected; the empirical evidence supports the claim that the new process increases breaking strength.
Two search ranking algorithms, Algorithm A and Algorithm B, are evaluated on latency across small independent benchmark query workloads:
- Algorithm A: , sample mean latency , sample standard deviation .
- Algorithm B: , sample mean latency , sample standard deviation .
Assuming both parent populations are approximately normally distributed with equal variances, perform a two-sample pooled Student's -test at the significance level to test against , given .
Formulating Hypotheses and Degrees of Freedom:
- Null Hypothesis:
- Alternative Hypothesis:
Degrees of freedom:
Calculating Pooled Variance and Standard Error: The pooled sample variance combines both sample variances weighted by their degrees of freedom:
Standard error of the difference in sample means:
Calculating Test Statistic and Reaching a Decision:
Critical value for two-tailed test at with :
Rejection rule: Reject if . Here, .
Decision: Reject . There is a statistically significant difference between the mean latencies of Algorithm A and Algorithm B at the significance level.
The pooled standard deviation is , standard error is , and the test statistic is . Since , we reject ; Algorithm A has significantly lower latency than Algorithm B.
A quality assurance test on IoT sensor batteries records an average continuous operating lifespan of with a sample standard deviation of .
- Construct a confidence interval for the true population mean operating lifespan ().
- Construct a confidence interval for the same metric ().
- State why the confidence interval is wider than the interval.
Computing the Standard Error: With , the CLT ensures that the sample mean distribution is approximately normal:
Constructing 90% and 99% Confidence Intervals:
- Confidence Interval: Margin of error: .
- Confidence Interval: Margin of error: .
Explanation of Interval Width vs Confidence Level: A higher confidence level requires a wider interval because greater certainty requires a broader net to capture the true parameter. To be sure of enclosing , the critical multiplier increases from to , increasing the margin of error from to .
The confidence interval is and the confidence interval is . The interval is wider because higher certainty requires a larger margin of error to minimize the risk of missing the true parameter.
Define Type I error and Type II error in hypothesis testing, and state their probabilities and practical interpretations. [Assignment, notebook p236]
Definitions of Type I and Type II Errors: In statistical hypothesis testing, two types of decision errors can occur when evaluating a null hypothesis :
- Type I Error (False Positive): A Type I error occurs when we reject a true null hypothesis ( is actually true, but the test rejects it).
- Its probability is denoted by (the level of significance, commonly or ):
- Type II Error (False Negative): A Type II error occurs when we fail to reject a false null hypothesis ( is actually false and is true, but the test fails to detect the effect).
- Its probability is denoted by :
- The complement is the statistical power of the test—the probability of correctly rejecting a false null hypothesis.
Summary Decision Table:
| Test Decision | is True | is False |
|---|---|---|
| Reject | Type I Error (probability ) | Correct Decision (Power ) |
| Fail to Reject | Correct Decision (probability ) | Type II Error (probability ) |
A Type I error () occurs when a true null hypothesis is rejected (false positive). A Type II error () occurs when a false null hypothesis is not rejected (false negative).