PadhaikoAll subjects

Analysis of simulation output

Simulation and Modeling (ENCT 353)

Open in the Padhaiko app

8.1 Confidence intervals and hypothesis testing

Because stochastic simulations use random numbers, a single simulation run produces just one random observation of a performance metric (such as average delay or blocking probability ). Drawing valid engineering conclusions therefore requires interval estimation and hypothesis testing across independent observations.

Definition

Confidence Interval for Mean Performance Measure

Given independent and identically distributed (i.i.d.) replication averages with sample mean and sample variance , the confidence interval (CI) for the true mean is:

where is the half-width (precision) of the confidence interval and is the Student's -critical value with degrees of freedom.

Confidence Interval for Proportions and Required Sample Size


8.2 Estimation methods

Output analysis distinguishes between point estimation (a single best guess) and interval estimation (a range quantifying sampling uncertainty), and uses specialized methods when observations within one run are autocorrelated.

Summary of Simulation Output Estimation Methods
Estimation MethodQuantity EstimatedKey Formula / MechanismBest Suited For
Point EstimationMean , variance , quantile, Quick summary, baseline reporting
Independent Replications with -CI separate runs using non-overlapping RNG streamsBoth terminating and steady-state simulations
Batch MeansSteady-state mean from one long runDiscard warm-up , split remaining observations into batches of size ( large enough that batch means are nearly uncorrelated)Steady-state simulations with expensive warm-up
Regenerative MethodSteady-state ratio Partition one long run at regeneration points (e.g., empty-and-idle system) into i.i.d. cyclesMarkovian / queuing systems that return to often

8.3 Simulation run statistics

Simulation studies fall into two fundamentally different regimes that dictate how run statistics are collected and analyzed:

AspectTerminating SimulationNon-Terminating (Steady-State) Simulation
Stopping RuleEnds at a well-defined natural event (e.g., bank closes at 5:00 PM, jobs finished)No natural end; studies long-run equilibrium as
Initial ConditionPart of the real problem (e.g., empty shop at 9:00 AM)Artificial; introduces initial transient bias that must be removed
Primary AnalysisMethod of Independent Replications ( runs)Warm-up deletion + Replication/Deletion or Batch Means

Discrete-Time vs. Continuous-Time Statistics

  1. Discrete-Time (Observation-Based) Statistics: Computed over sequential entity observations (e.g., customer waiting time ):
  1. Continuous-Time (Time-Persistent) Statistics: Computed by integrating a piecewise-constant state trajectory over simulation horizon (e.g., queue length or server busy status ):

8.4 Replication of runs

The method of independent replications executes the simulation model independent times under identical model parameters, changing only the random-number seeds (or assigning disjoint random-number streams) between runs.

Variance Reduction via Common Random Numbers (CRN)

When comparing two competing system configurations and (e.g., adding a second server vs. speeding up one server), running both configurations with the exact same sequence of random arrivals and service draws in each replication creates positive correlation ():

Subtracting sharply reduces the variance of the paired difference , letting analysts detect real improvements with far fewer replications using a paired -confidence interval.


8.5 Elimination of initial bias

When a steady-state simulation starts in an artificial empty-and-idle state (), early observations are biased downward relative to long-run behavior. This distortion is called initialization bias (initial transient).

Definition

Warm-Up Period (Initial Data Deletion)

To eliminate initial bias in steady-state output analysis, the analyst discards observations collected during an initial warm-up period of length (or observations) and computes estimators only from the post-transient data:

Standard Methods to Eliminate or Mitigate Initial Bias

  1. Truncation / Warm-Up Deletion (Welch’s Method): Run pilot replications, average the trajectories across replications at each time index , smooth with a moving average, and visually pick the warm-up point (or ) where the curve flattens into steady state.
  2. Realistic Initial State: Initialize queues and machines close to their expected steady-state occupancy (e.g., starting with jobs in queue instead of ) rather than empty-and-idle.
  3. Sufficiently Long Run Length (): Because initial bias decays at rate , running a very long horizon dilutes any remaining transient distortion.

Worked Out Examples

Example 1

1

Five independent replications () of a terminating bank simulation (9:00 AM–5:00 PM) produce mean customer waiting times (in minutes):

  1. Compute the point estimate , sample standard deviation , and confidence interval ().
  2. How many total replications are required to achieve a confidence interval half-width of at most (given , )?

Computing sample mean and sample standard deviation :

Constructing the confidence interval ():

Estimating required replications for :

  • Normal approximation: .
  • Refining with -distribution: trying ():

(Trying with gives , so is the smallest integer satisfying the bound.)

Answer:
  1. Point estimate , , CI .
  2. replications are needed to guarantee a half-width .
Example 2

2

In a single-server queue simulation starting empty at and ending at , the recorded queue length is:

  1. Compute the continuous time-averaged queue length over the full horizon .
  2. To eliminate empty-start initial bias, discard a warm-up period of and compute the truncated steady-state estimate over .

Integrating over the full horizon :

Removing warm-up and integrating over (duration ): On , on (length ), on (length ), on (length ), and on (length ):

Answer:
  1. Full-run time average: .
  2. Post-warm-up steady-state estimate over : (notice how discarding the empty start removes downward bias).
Example 3

3

Two web-server scheduling policies (Policy A and Policy B) are compared across replications using Common Random Numbers (CRN), giving paired mean response times (in ms):

Paired CRN Replications for Policy A vs. Policy B
Replication Policy A ()Policy B ()

Construct a paired- confidence interval for () and determine whether Policy B is significantly faster.

Computing paired differences :

Computing paired standard deviation and CI:

Answer:

The paired- confidence interval for is . Since the entire interval lies strictly above , Policy B is statistically significantly faster than Policy A.


Exercises

Practice Exercises — Analysis of Simulation Output

Question 1

Distinguish between terminating and non-terminating (steady-state) simulations, and explain why applying the naive i.i.d. formula to raw observations from a single simulation run produces invalid confidence intervals.

Terminating vs. Non-Terminating Simulations:

  • A terminating simulation has a natural starting state and well-defined stopping event (e.g., an 8-hour bank day); initial conditions are realistic and independent replications yield valid -CIs directly.
  • A non-terminating simulation investigates long-run equilibrium () with no natural stop event, requiring warm-up deletion and batch means or long replications.

Why Fails Within One Run: Consecutive observations in a run (e.g., waiting times and in a queue) are positively autocorrelated. Because exceeds when , naive underestimates true variance and creates overly optimistic (too narrow) confidence intervals.

Answer:

Terminating runs end at a natural event while steady-state runs analyze long-run equilibrium. Naive fails inside a single run because positive autocorrelation inflates the true variance of the sample mean.

Question 2

Explain the method of batch means for steady-state output analysis and describe three techniques used to eliminate initial transient bias in non-terminating simulations.

Method of Batch Means: In one long steady-state run after discarding the warm-up period , the remaining observations are divided into contiguous batches of equal size . When is sufficiently large, the batch means become nearly uncorrelated and approximately normal, permitting a standard -CI with degrees of freedom.

Three Techniques to Eliminate Initial Bias:

  1. Warm-Up Deletion (Truncation / Welch’s Method): Discard the initial transient interval identified from pilot averaged curves.
  2. Realistic Initial State: Start queues/inventories near expected steady-state values rather than empty-and-idle.
  3. Long Run Horizon: Make total run length so transient error vanishes at .
Answer:

Batch means splits a single post-warm-up run into large batches whose averages are nearly independent. Initial bias is removed via warm-up deletion (Welch's method), realistic initialization, and very long run lengths.

Question 3

Four independent replications of a manufacturing cell produce daily throughput values parts/day. Compute the confidence interval for the mean daily throughput ().

Computing and ():

Constructing the CI:

Answer:

The confidence interval for mean daily throughput is .

Question 4

In independent simulated network connections, succeed within the latency deadline (). Compute a confidence interval for the true success probability ().

Computing standard error of :

Constructing the CI:

Answer:

The confidence interval for the success probability is ( to ).

Question 5

Five consecutive batch means from a steady-state queuing simulation are .

  1. Compute the mean over all batches.
  2. If Batch () is discarded as the warm-up batch, compute the steady-state point estimate from the remaining batches and state the percentage change.

Full-run mean vs. truncated mean:

Discarding Batch :

Percentage change after warm-up deletion:

Answer:
  1. Full -batch mean: .
  2. Truncated steady-state mean: (an increase of after removing the low warm-up batch).
← Verification and Validation of Simulation ModelsSimulation software →