Analysis of simulation output
Open in the Padhaiko app8.1 Confidence intervals and hypothesis testing
Because stochastic simulations use random numbers, a single simulation run produces just one random observation of a performance metric (such as average delay or blocking probability ). Drawing valid engineering conclusions therefore requires interval estimation and hypothesis testing across independent observations.
Confidence Interval for Mean Performance Measure
Given independent and identically distributed (i.i.d.) replication averages with sample mean and sample variance , the confidence interval (CI) for the true mean is:
where is the half-width (precision) of the confidence interval and is the Student's -critical value with degrees of freedom.
Confidence Interval for Proportions and Required Sample Size
- Proportion / Probability Metric (): For independent Bernoulli trials with observed successes (, large ):
- Sample-Size (Replication Count) Formula: To shrink the half-width to at most a target tolerance (), we estimate from an initial pilot sample variance :
- Hypothesis Testing & Two-System Comparison: To compare two designs and , construct a CI for (via Welch's two-sample -test for independent runs, or a paired -test when Common Random Numbers are used). If the CI excludes , the difference is statistically significant at level .
8.2 Estimation methods
Output analysis distinguishes between point estimation (a single best guess) and interval estimation (a range quantifying sampling uncertainty), and uses specialized methods when observations within one run are autocorrelated.
| Estimation Method | Quantity Estimated | Key Formula / Mechanism | Best Suited For |
|---|---|---|---|
| Point Estimation | Mean , variance , quantile | , | Quick summary, baseline reporting |
| Independent Replications | with -CI | separate runs using non-overlapping RNG streams | Both terminating and steady-state simulations |
| Batch Means | Steady-state mean from one long run | Discard warm-up , split remaining observations into batches of size ( large enough that batch means are nearly uncorrelated) | Steady-state simulations with expensive warm-up |
| Regenerative Method | Steady-state ratio | Partition one long run at regeneration points (e.g., empty-and-idle system) into i.i.d. cycles | Markovian / queuing systems that return to often |
8.3 Simulation run statistics
Simulation studies fall into two fundamentally different regimes that dictate how run statistics are collected and analyzed:
| Aspect | Terminating Simulation | Non-Terminating (Steady-State) Simulation |
|---|---|---|
| Stopping Rule | Ends at a well-defined natural event (e.g., bank closes at 5:00 PM, jobs finished) | No natural end; studies long-run equilibrium as |
| Initial Condition | Part of the real problem (e.g., empty shop at 9:00 AM) | Artificial; introduces initial transient bias that must be removed |
| Primary Analysis | Method of Independent Replications ( runs) | Warm-up deletion + Replication/Deletion or Batch Means |
Discrete-Time vs. Continuous-Time Statistics
- Discrete-Time (Observation-Based) Statistics: Computed over sequential entity observations (e.g., customer waiting time ):
- Continuous-Time (Time-Persistent) Statistics: Computed by integrating a piecewise-constant state trajectory over simulation horizon (e.g., queue length or server busy status ):
8.4 Replication of runs
The method of independent replications executes the simulation model independent times under identical model parameters, changing only the random-number seeds (or assigning disjoint random-number streams) between runs.
- Each replication produces a single summary output (e.g., mean delay or throughput in run ).
- Because the streams are independent, are genuinely independent and identically distributed, making standard -confidence intervals mathematically exact (assuming approximate normality of run averages).
Variance Reduction via Common Random Numbers (CRN)
When comparing two competing system configurations and (e.g., adding a second server vs. speeding up one server), running both configurations with the exact same sequence of random arrivals and service draws in each replication creates positive correlation ():
Subtracting sharply reduces the variance of the paired difference , letting analysts detect real improvements with far fewer replications using a paired -confidence interval.
8.5 Elimination of initial bias
When a steady-state simulation starts in an artificial empty-and-idle state (), early observations are biased downward relative to long-run behavior. This distortion is called initialization bias (initial transient).
Warm-Up Period (Initial Data Deletion)
To eliminate initial bias in steady-state output analysis, the analyst discards observations collected during an initial warm-up period of length (or observations) and computes estimators only from the post-transient data:
Standard Methods to Eliminate or Mitigate Initial Bias
- Truncation / Warm-Up Deletion (Welch’s Method): Run pilot replications, average the trajectories across replications at each time index , smooth with a moving average, and visually pick the warm-up point (or ) where the curve flattens into steady state.
- Realistic Initial State: Initialize queues and machines close to their expected steady-state occupancy (e.g., starting with jobs in queue instead of ) rather than empty-and-idle.
- Sufficiently Long Run Length (): Because initial bias decays at rate , running a very long horizon dilutes any remaining transient distortion.
Worked Out Examples
1
Five independent replications () of a terminating bank simulation (9:00 AM–5:00 PM) produce mean customer waiting times (in minutes):
- Compute the point estimate , sample standard deviation , and confidence interval ().
- How many total replications are required to achieve a confidence interval half-width of at most (given , )?
Computing sample mean and sample standard deviation :
Constructing the confidence interval ():
Estimating required replications for :
- Normal approximation: .
- Refining with -distribution: trying ():
(Trying with gives , so is the smallest integer satisfying the bound.)
- Point estimate , , CI .
- replications are needed to guarantee a half-width .
2
In a single-server queue simulation starting empty at and ending at , the recorded queue length is:
- for
- for
- for
- for
- for
- Compute the continuous time-averaged queue length over the full horizon .
- To eliminate empty-start initial bias, discard a warm-up period of and compute the truncated steady-state estimate over .
Integrating over the full horizon :
Removing warm-up and integrating over (duration ): On , on (length ), on (length ), on (length ), and on (length ):
- Full-run time average: .
- Post-warm-up steady-state estimate over : (notice how discarding the empty start removes downward bias).
3
Two web-server scheduling policies (Policy A and Policy B) are compared across replications using Common Random Numbers (CRN), giving paired mean response times (in ms):
| Replication | Policy A () | Policy B () |
|---|---|---|
Construct a paired- confidence interval for () and determine whether Policy B is significantly faster.
Computing paired differences :
Computing paired standard deviation and CI:
The paired- confidence interval for is . Since the entire interval lies strictly above , Policy B is statistically significantly faster than Policy A.
Exercises
Practice Exercises — Analysis of Simulation Output
Distinguish between terminating and non-terminating (steady-state) simulations, and explain why applying the naive i.i.d. formula to raw observations from a single simulation run produces invalid confidence intervals.
Terminating vs. Non-Terminating Simulations:
- A terminating simulation has a natural starting state and well-defined stopping event (e.g., an 8-hour bank day); initial conditions are realistic and independent replications yield valid -CIs directly.
- A non-terminating simulation investigates long-run equilibrium () with no natural stop event, requiring warm-up deletion and batch means or long replications.
Why Fails Within One Run: Consecutive observations in a run (e.g., waiting times and in a queue) are positively autocorrelated. Because exceeds when , naive underestimates true variance and creates overly optimistic (too narrow) confidence intervals.
Terminating runs end at a natural event while steady-state runs analyze long-run equilibrium. Naive fails inside a single run because positive autocorrelation inflates the true variance of the sample mean.
Explain the method of batch means for steady-state output analysis and describe three techniques used to eliminate initial transient bias in non-terminating simulations.
Method of Batch Means: In one long steady-state run after discarding the warm-up period , the remaining observations are divided into contiguous batches of equal size . When is sufficiently large, the batch means become nearly uncorrelated and approximately normal, permitting a standard -CI with degrees of freedom.
Three Techniques to Eliminate Initial Bias:
- Warm-Up Deletion (Truncation / Welch’s Method): Discard the initial transient interval identified from pilot averaged curves.
- Realistic Initial State: Start queues/inventories near expected steady-state values rather than empty-and-idle.
- Long Run Horizon: Make total run length so transient error vanishes at .
Batch means splits a single post-warm-up run into large batches whose averages are nearly independent. Initial bias is removed via warm-up deletion (Welch's method), realistic initialization, and very long run lengths.
Four independent replications of a manufacturing cell produce daily throughput values parts/day. Compute the confidence interval for the mean daily throughput ().
Computing and ():
Constructing the CI:
The confidence interval for mean daily throughput is .
In independent simulated network connections, succeed within the latency deadline (). Compute a confidence interval for the true success probability ().
Computing standard error of :
Constructing the CI:
The confidence interval for the success probability is ( to ).
Five consecutive batch means from a steady-state queuing simulation are .
- Compute the mean over all batches.
- If Batch () is discarded as the warm-up batch, compute the steady-state point estimate from the remaining batches and state the percentage change.
Full-run mean vs. truncated mean:
Discarding Batch :
Percentage change after warm-up deletion:
- Full -batch mean: .
- Truncated steady-state mean: (an increase of after removing the low warm-up batch).