PadhaikoAll subjects

Analysis of Variance (ANOVA)

Foundation of Data Science

Open in the Padhaiko app

2.10 Analysis of Variance (ANOVA)

When evaluating differences across experimental conditions or machine learning models, data scientists frequently need to compare more than two groups simultaneously—such as evaluating three search ranking algorithms, four website color palettes, or five hyperparameter configurations.

Definition

Purpose of ANOVA and Problem of Multiple Testing

Analysis of Variance (ANOVA) is a hypothesis-testing procedure developed by Ronald Fisher used to evaluate whether the means of three or more independent populations are statistically equal:

Why Not Use Repeated Pairwise Two-Sample -Tests? If an experiment contains groups, comparing every pair requires:

Each test carries a Type I error probability of . If the null hypothesis is true for all groups, the probability of at least one false positive—known as the family-wise error rate ()—balloons rapidly:

For groups ( tests), the false positive rate exceeds .

ANOVA solves this problem by performing an omnibus test that evaluates all group means simultaneously in a single unified calculation, strictly preserving the overall Type I error rate at the desired .

Definition

Partition of Total Variation: Between vs Within

The foundational insight of ANOVA is that the total variability among all observed data points can be partitioned into two independent, additive components:

Let be the number of groups, be the number of observations in group , be the total sample size across all groups, and denote the -th observation in group .

  1. Grand Mean () and Group Means ():
  1. Sum of Squares:
    • Between-Group Sum of Squares (, or ): Measures how much the group sample means deviate from the grand mean. This captures the treatment effect plus random sampling error:
  1. Degrees of Freedom ():

Note that .

  1. Mean Squares (): Dividing each sum of squares by its degrees of freedom produces variance estimates:
  1. The -Statistic: The test statistic is the ratio of between-group variance to within-group variance:
Structure of the Standard One-Way ANOVA Table
Source of VariationSum of Squares ()Degrees of Freedom ()Mean Square ()-RatioCritical Threshold
Between Groups (Treatments)
Within Groups (Error)——
Total———
Definition

Assumptions of One-Way ANOVA

For the -distribution to accurately model the test statistic, three core assumptions must be satisfied:

  1. Normality: The residuals (or response variable values) within each of the groups must follow an approximately normal distribution.
    • ANOVA is moderately robust to mild violations of normality when sample sizes are equal and .
  2. Homoscedasticity (Homogeneity of Variances): The population variances of all groups must be equal: .
    • This assumption is formally tested using Levene's test or Bartlett's test. If violated with unequal group sizes, Welch's ANOVA or non-parametric tests (Kruskal–Wallis) must be substituted.
  3. Independence of Observations: All observations must be randomly sampled and independent of one another, both within each group and across groups.
Definition

Post-Hoc Tests

The ANOVA -test is an omnibus test: when it rejects the null hypothesis , it reveals only that at least one group mean differs significantly from the others. It does not indicate which specific pairs of groups differ, nor how many pairs differ.

To identify exactly which group means differ without inflating the family-wise Type I error rate, researchers perform post-hoc tests (conducted only after a significant ANOVA result):

  1. Tukey's Honestly Significant Difference (HSD): The gold standard for pairwise comparisons when group sizes are equal. It calculates a minimum difference threshold between means:

Any pair of group means whose absolute difference exceeds is declared statistically significantly different at level .

  1. Bonferroni Correction: Divides the target by the number of comparisons: . Simple and conservative, but loses statistical power when is large.
  2. Scheffé's Method: Highly flexible method that controls error rate for all possible linear contrasts, though more conservative than Tukey's HSD.
Definition

Two-Way ANOVA: Main Effects and Interaction

While One-Way ANOVA tests a single independent factor across multiple levels, Two-Way ANOVA evaluates the simultaneous effects of two independent categorical factors (Factor A and Factor B) on a single continuous dependent variable:

Two-Way ANOVA tests three distinct null hypotheses:

  1. Main Effect of Factor A: There is no difference in the dependent variable across the levels of Factor A ().
  2. Main Effect of Factor B: There is no difference in the dependent variable across the levels of Factor B ().
  3. Interaction Effect (): The effect of Factor A does not depend on the level of Factor B ().
    • An interaction occurs when the effect of one factor changes depending on the level of another. For example, a machine learning optimization algorithm might drastically speed up training on GPUs but have negligible impact on CPUs.

Worked Out Examples

Example 2.8

2.8

A research scientist measures performance across three experimental cohorts () with subjects per cohort:

  1. State the null and alternative hypotheses.
  2. Calculate the group means and the grand mean.
  3. Compute the Sum of Squares Between Groups () and Within Groups ().
  4. Construct the complete One-Way ANOVA table.
  5. Given critical value , state whether the cohort means differ significantly at the level.

Formulating Hypotheses:

  • Null Hypothesis: (all cohort population means are equal)
  • Alternative Hypothesis:

Computing Group Means and Grand Mean: The total sample size is .

Group means:

Grand mean across all observations:

Calculating Between-Group and Within-Group Sum of Squares: Between-Group Sum of Squares ():

Within-Group Sum of Squares ():

  • Cohort A:
  • Cohort B:
  • Cohort C:

Total Sum of Squares:

Constructing the ANOVA Table and Reaching a Decision: Degrees of freedom:

Mean squares:

Test statistic:

Source of Variation-RatioCritical
Between Cohorts
Within Cohorts——
Total———

Decision: Since , we fail to reject . There is no statistically significant difference among the three cohort means at the level.

Answer:

The One-Way ANOVA yields () and (), resulting in . Because , we fail to reject ; there is no significant difference among the three cohort means.

Example 2.9

2.9

A university department evaluates three distinct teaching pedagogies (Method A, Method B, and Method C) on final exam test scores (out of 100). Three student groups of size each record the following scores:

Perform a One-Way ANOVA at the significance level to test whether the three teaching methods result in significantly different average exam scores. Critical value: .

Formulating Hypotheses and Calculating Group Totals:

  • Null Hypothesis:
  • Alternative Hypothesis:

Totals and means for each group ():

  • Method A:
  • Method B:
  • Method C:

Grand total and grand mean ():

Computing the Sum of Squares: Between-Group Sum of Squares ():

Within-Group Sum of Squares (): Sum of squares of all raw observations :

Total Sum of Squares:

Constructing ANOVA Table and Testing Significance: Degrees of freedom:

Mean squares:

-statistic:

Source of Variation-RatioCritical
Between Methods
Within Methods (Error)——
Total———

Decision: Since , we reject the null hypothesis at the significance level. There is a statistically significant difference in student exam scores among the three teaching methods.

Answer:

The One-Way ANOVA produces and , yielding an -statistic of . Because , is rejected; the three teaching pedagogies lead to significantly different student performance.

Example 2.10

2.10

A database systems researcher benchmarks the response latencies (in milliseconds) of three database query optimization techniques with unequal sample sizes:

Perform a One-Way ANOVA at the significance level to test whether the mean latencies of the three optimization techniques differ significantly. Given critical value .

Formulating Hypotheses and Calculating Group Metrics:

  • Null Hypothesis: (all query optimizers have equal mean latency)
  • Alternative Hypothesis:

Group totals and means:

  • Technique 1 ():
  • Technique 2 ():
  • Technique 3 ():

Grand total and grand mean ():

Computing Sum of Squares: Between-Group Sum of Squares ():

Within-Group Sum of Squares ():

  • Group 1:
  • Group 2:
  • Group 3:

Total Sum of Squares:

Constructing ANOVA Table and Testing Significance: Degrees of freedom:

Mean squares:

-statistic:

Source of Variation-RatioCritical
Between Optimizers
Within Optimizers (Error)——
Total———

Decision: Since , we reject . There is an exceptionally strong, statistically significant difference among the mean latencies of the three database query optimizers.

Answer:

The One-Way ANOVA with unequal group sizes gives and , yielding . Because , we reject ; query optimization techniques differ significantly in latency (Technique 3 is fastest at , Technique 2 is slowest at ).

Exercises

Exercises

Question 1

Explain the core purpose of ANOVA in comparing multiple group means. Why is it statistically unsound to replace ANOVA with multiple pairwise two-sample -tests?

Purpose of ANOVA: The purpose of Analysis of Variance (ANOVA) is to compare the means of populations simultaneously to determine whether at least one population mean differs from the others, while holding the global probability of a Type I error constant at a chosen significance level .

Flaw of Multiple Pairwise -Tests: When pairwise -tests are performed across groups, the total number of pairwise tests is . Each test carries an independent probability of not making a Type I error.

Assuming independence, the overall family-wise error rate across all tests is:

For groups (), with individual :

Conducting multiple -tests severely inflates the false positive rate, making it very likely to declare a difference significant purely by random chance. ANOVA avoids this by executing a single simultaneous test.

Answer:

ANOVA tests equality of multiple group means simultaneously. Replacing it with pairwise -tests causes severe Type I error inflation: across tests, the family-wise error rate grows to (reaching for groups), producing frequent spurious discoveries.

Question 2

State the three fundamental mathematical assumptions required for One-Way ANOVA. What practical diagnostic methods are used to inspect these assumptions, and how should an analyst respond if the equal variance assumption is violated?

The Three Core Assumptions:

  1. Normality: The response variable values (or residuals) within each population must be normally distributed: .
  2. Homoscedasticity (Homogeneity of Variances): All populations must share a common variance: .
  3. Independence: All observations must be mutually independent random samples.

Diagnostics and Remediation:

  • Diagnostics: Normality is assessed using Q-Q plots and the Shapiro–Wilk test. Homogeneity of variances is checked using Levene's test or Bartlett's test.
  • Handling Unequal Variances: If the homoscedasticity assumption is violated, the standard -test loses validity. The analyst should:
    1. Apply Welch's ANOVA, which does not assume equal variances and uses adjusted degrees of freedom.
    2. Use a variance-stabilizing data transformation (e.g., logarithmic or square-root transform).
    3. Adopt a non-parametric alternative, such as the Kruskal–Wallis test.
Answer:

The three assumptions are normality of residuals, homoscedasticity (), and independence of observations. Diagnostics include Q-Q plots and Levene's test. If variances are unequal, Welch's ANOVA or the non-parametric Kruskal–Wallis test should be used.

Question 3

An agricultural data scientist tests the effect of three distinct bio-fertilizers on plant yield (in kg per plot) using independent plots per treatment:

Construct the complete One-Way ANOVA table and determine whether the bio-fertilizer treatments produce significantly different yields at the level ().

Formulating Hypotheses and Calculating Group Totals:

  • Null Hypothesis:
  • Alternative Hypothesis:

Totals and means for each treatment ():

  • Treatment 1:
  • Treatment 2:
  • Treatment 3:

Grand total and grand mean ():

Calculating Between-Group and Within-Group Sum of Squares: Between-Group Sum of Squares ():

Within-Group Sum of Squares ():

  • Treatment 1:
  • Treatment 2:
  • Treatment 3:

ANOVA Table and Decision: Degrees of freedom:

Mean squares:

Source of Variation-RatioCritical
Between Treatments
Within Treatments (Error)——
Total———

Decision: Since , we reject . Bio-fertilizer treatments produce significantly different crop yields.

Answer:

The One-Way ANOVA yields and , giving . Because , we reject ; the bio-fertilizer treatments result in statistically significant yield differences.

Question 4

An incomplete One-Way ANOVA table from an experiment evaluating algorithm configurations across a total of trials has several missing values:

Source of VariationSum of Squares ()Degrees of Freedom ()Mean Square ()-Statistic
Between Groups??
Within Groups??—
Total??——
  1. Fill in all missing values in the table.
  2. Given critical value , state the test decision at .

Deducing the Missing Table Entries: Using the fundamental ANOVA relationships:

  1. Total Sum of Squares ():
  1. Within-Group Degrees of Freedom ():
  1. Total Degrees of Freedom ():
  1. Between-Group Mean Square ():
  1. Within-Group Mean Square ():
  1. -Statistic:

Completed Table and Decision Rule: The completed ANOVA table is:

Source of VariationSum of Squares ()Degrees of Freedom ()Mean Square ()-Statistic
Between Groups
Within Groups—
Total——

Decision Rule: At , the critical value is . Because computed , we reject the null hypothesis . There is statistically significant evidence that at least one algorithm configuration produces a different mean outcome.

Answer:

The deduced values are , , , , , and . Because , the null hypothesis is rejected at .

Question 5

Explain why post-hoc tests are required after an ANOVA test rejects the null hypothesis. Describe the mechanism of Tukey's Honestly Significant Difference (HSD) test.

Why Post-Hoc Tests are Necessary: A statistically significant ANOVA -test indicates only that the null hypothesis is false—meaning at least one group mean is different. However, the -statistic is non-directional and non-specific: it cannot identify which specific pair (e.g., whether Group 1 differs from Group 2, or Group 2 from Group 3) caused the rejection.

Running unadjusted -tests at this stage would cause Type I error inflation. Post-hoc tests evaluate all pairwise differences while mathematically enforcing control over the family-wise error rate .

Mechanism of Tukey's HSD Test: Tukey's HSD calculates a single critical difference threshold:

where:

  • is the studentized range statistic table value for groups with within-group degrees of freedom at level .
  • is the mean square within groups from the ANOVA table.
  • is the sample size per group (or harmonic mean for unequal samples).

For every pair of group sample means , the absolute difference is calculated:

  • If , the pair is declared statistically significantly different at level .
  • If , the difference is attributed to chance.
Answer:

ANOVA only establishes that at least one group mean differs; post-hoc tests identify the specific differing pairs while controlling the family-wise error rate. Tukey's HSD computes a critical difference ; any pairwise mean difference exceeding this threshold is declared statistically significant.

Question 6

Distinguish between One-Way ANOVA and Two-Way ANOVA. Define the concepts of "main effect" and "interaction effect", providing a concrete data science scenario illustrating an interaction.

Comparison of One-Way vs Two-Way ANOVA:

DimensionOne-Way ANOVATwo-Way ANOVA
Number of Independent VariablesOne categorical factor with levelsTwo categorical factors (Factor A with levels, Factor B with levels)
Hypotheses TestedOne (: all group means equal)Three (: main effect A, : main effect B, : interaction)
Error Partitioning
Analytical ScopeCompares isolated factor levelsEvaluates independent factor effects plus synergy/interference

Main Effect vs Interaction Effect:

  • Main Effect: The direct impact of a single independent factor averaged across all levels of the other factor.
  • Interaction Effect: Occurs when the impact of one factor depends on the specific level of the second factor.

Data Science Scenario Illustrating Interaction: Suppose a machine learning engineering team benchmarks inference throughput across:

  • Factor A: Hardware Architecture (GPU vs CPU).
  • Factor B: Model Quantization (Full FP32 vs 8-bit INT8).

If INT8 quantization yields an throughput speedup on GPU due to dedicated Tensor Cores, but produces only a speedup on CPU due to lack of vector INT8 instructions, the hardware and quantization factors interact. The benefit of quantization cannot be stated without specifying which hardware is utilized.

Answer:

One-Way ANOVA evaluates one independent factor; Two-Way ANOVA evaluates two independent factors simultaneously, testing for two main effects and their interaction. An interaction occurs when the effect of one factor depends on the other—such as INT8 quantization producing massive speedups on GPUs but minimal speedups on CPUs.

Past exam questions for Foundation of Data Science

← Hypothesis Testing, z/t-Tests & Confidence IntervalsTypes of Data, Preprocessing Needs & Data Collection →