Statistical Inference (part 1: 3.1–3.3)
Open in the Padhaiko app3.1 Point estimations and properties of estimators
Statistical inference uses sample data to draw conclusions about unknown population parameters (, such as , , or ). An estimator is a statistic (random variable) used to approximate ; a point estimate is the single numerical value obtained when is evaluated on an observed sample .
3.1.1 Properties of Good Point Estimators
Unbiasedness, Consistency, Efficiency, and Sufficiency
Let be an estimator of an unknown parameter based on a random sample of size :
- Unbiasedness: is an unbiased estimator of if its expected value equals the true parameter for all possible values of :
The bias is . The overall accuracy of any estimator (biased or unbiased) is measured by its Mean Squared Error ():
(Examples: Sample mean , sample proportion , and sample variance are unbiased; is biased by .)
- Consistency: A sequence of estimators is consistent for if converges in probability to as sample size increases ( for every ).
- Sufficient condition: If and (i.e., ), then is consistent.
- Efficiency: Between two unbiased estimators and , is more efficient if . The relative efficiency of with respect to is:
An unbiased estimator achieving the smallest possible variance among all unbiased estimators is a Minimum Variance Unbiased Estimator (MVUE) (attaining the Cramér–Rao Lower Bound under regularity conditions).
- Sufficiency: An estimator is sufficient for if it extracts all information in the sample relevant to —meaning the conditional distribution of the sample given does not depend on . By the Neyman–Fisher Factorization Theorem, is sufficient iff the joint likelihood factors as .
3.1.2 Common Methods of Point Estimation
- Method of Moments (MoM): Equate sample moments to population moments and solve for unknown parameters.
- Maximum Likelihood Estimation (MLE): Choose to maximize the likelihood function (or equivalently the log-likelihood ). MLEs are consistent, asymptotically normal, asymptotically efficient, and invariant under reparameterization.
3.1
Let be an i.i.d. random sample from a population with unknown mean and finite variance . Consider two candidate point estimators for :
- Check whether and are unbiased estimators of .
- Compute their variances and determine which estimator is more efficient (and its relative efficiency).
- Explain whether is consistent as .
Check unbiasedness using linearity of expectation ():
Both and are unbiased estimators of .
Compute variances using independence ():
Since , is more efficient, with relative efficiency:
Check consistency of : For any sample size , and . Therefore, is a consistent estimator of .
Both and are unbiased; is more efficient (, relative efficiency ); is consistent because .
3.2 Confidence intervals for mean and proportions
A point estimate alone gives no indication of its precision. An interval estimate or Confidence Interval (CI) takes the form:
where is the confidence level (commonly , , or , corresponding to and two-tailed ).
3.2.1 Confidence Interval Formulas
| Target Parameter | Conditions | Confidence Interval Formula | Standard Error () |
|---|---|---|---|
| Mean (known , or large ) | Normal population or CLT (); if unknown and , replace by | ||
| Mean (unknown , small ) | Normal population; degrees of freedom | ||
| Proportion (large sample) | and |
3.2
- (Mean, small sample) Tensile tests on welded joints give sample mean and sample standard deviation . Assuming normality, construct a confidence interval for the true mean tensile strength ().
- (Proportion) Out of tested PCB assemblies, fail automated optical inspection. Construct a confidence interval for the true defect proportion ().
- (Sample size) What minimum sample size is needed to estimate a proportion within at confidence () if no prior estimate of is available?
Part 1 — -interval for (, ):
Part 2 — -interval for (): Since and :
Part 3 — Conservative sample size for proportion ():
- CI for : ; 2. CI for : ; 3. Minimum sample size: .
3.3 Hypothesis testing, parametric and non-parametric tests, procedure of hypothesis
A statistical hypothesis is a testable claim about one or more population parameters or the shape of a probability distribution. Hypothesis testing evaluates whether sample evidence is strong enough to reject a baseline claim.
3.3.1 Null and Alternative Hypotheses, Errors, and -Value
Hypotheses, Type I and Type II Errors, Power, and p-Value
- Null Hypothesis (): The baseline status-quo statement of "no difference" or "no effect" (always contains equality: , , or ). Tested directly under the assumption that is true.
- Alternative Hypothesis ( or ): The research or engineering claim of change/difference ( for a two-tailed test; for right-tailed; for left-tailed).
- Decision Errors:
- Type I Error (, Producer's Risk): Rejecting when is actually true. is the level of significance.
- Type II Error (, Consumer's Risk): Failing to reject when is actually true. .
- Power of the Test (): Probability of correctly rejecting a false : .
- -Value: The smallest significance level at which would be rejected for the observed data—equivalently, the probability (assuming is true) of obtaining a test statistic at least as extreme as the observed value. Decision rule: Reject if .
| Test Decision | is Actually True | is Actually False ( True) |
|---|---|---|
| Fail to Reject | Correct decision () | Type II Error () |
| Reject | Type I Error () | Correct decision (Power ) |
3.3.2 General Procedure of Hypothesis Testing
Every formal hypothesis test follows six standard steps:
- State and clearly in terms of population parameters (and identify whether the test is two-tailed, right-tailed, or left-tailed).
- Choose the significance level (typically or ).
- Select the appropriate test statistic (e.g., , , , ) and verify its assumptions (normality, independence, sample size).
- Determine the critical (rejection) region from statistical tables (or set the -value threshold ).
- Compute the test statistic (and optionally the -value) from the sample data.
- Make a statistical decision and engineering conclusion: Reject if the test statistic falls in the critical region (or ); otherwise fail to reject . State the conclusion in the physical context of the problem.
3.3.3 Parametric vs. Non-Parametric Tests
| Aspect | Parametric Tests | Non-Parametric Tests |
|---|---|---|
| Distributional Assumption | Assumes specific underlying population distribution (usually Normal) and interval/ratio scale | Makes no strict assumption about population shape; works on nominal/ordinal/ranked data |
| Target Quantity | Parameters such as mean () or variance () | Medians, ranks, signs, empirical distributions (), or independence |
| Statistical Power | Higher power when assumptions hold | Slightly lower power on normal data, but more robust to outliers and heavy skew |
| Typical Examples | -test, -tests (1-sample, 2-sample, paired), one-way ANOVA (-test) | Sign test, Wilcoxon tests, Mann–Whitney , Kruskal–Wallis, Chi-square (), Kolmogorov–Smirnov |
3.3
A chemical process has standard yield mean with known standard deviation . To test against using a sample of size , an engineer uses the decision rule: reject if .
- Find the significance level (Type I error probability).
- Find the Type II error probability and the power of the test when the true mean yield has shifted to (, ).
Standard error of :
Part 1 — Type I Error () under :
Part 2 — Type II Error () and Power () under : The test fails to reject when :
Using :
- Significance level ; 2. Type II error and Power ().
Practice Exercises — Point Estimation, Confidence Intervals & Hypothesis Concepts
Define unbiasedness, consistency, efficiency, and sufficiency of a point estimator. Give the relationship between , , and bias, and state why the denominator is used in sample variance .
Four Properties of Estimators:
- Unbiasedness: for all .
- Consistency: as (implied if bias and ).
- Efficiency: Smallest variance among unbiased estimators; is more efficient than if .
- Sufficiency: captures all information in the sample about .
MSE Identity and Sample Variance:
Because , dividing by yields , making unbiased.
The four properties are unbiasedness (), consistency (), efficiency (minimum variance), and sufficiency (retains all sample information about ). , and divides by to achieve .
Given two independent observations from a population with mean and variance , compare the estimators and . Verify both are unbiased and compute the relative efficiency of with respect to .
Check unbiasedness:
Both estimators are unbiased.
Compare variances and relative efficiency:
Both are unbiased; has smaller variance () and is more efficient with relative efficiency .
A sample of steel cables has mean breaking load . Assume the population standard deviation is .
- Construct a confidence interval for the true mean breaking load ().
- How large a sample is required if the margin of error must not exceed ?
Part 1 — Confidence Interval:
Part 2 — Required sample size for :
- CI: ; 2. Minimum sample size: .
In a usability survey of engineering students, reported experiencing compiler timeout issues during lab exams. Construct a confidence interval for the true population proportion ().
Compute and :
Compute CI:
The confidence interval for is (or to ).
A machine fills resin cartridges to a target mean with . To test against with , the quality team rejects whenever .
- Compute the Type I error probability ().
- Compute the Type II error probability and the power if the true mean drops to ().
Compute under ():
Compute and Power under :
- ; 2. and Power ().