PadhaikoAll subjects

Verification and Validation of Simulation Models

Simulation and Modeling (ENCT 353)

Open in the Padhaiko app

7.1 Verification and validation

Even a mathematically elegant simulation fails if its code contains logic bugs or its assumptions do not match reality. Verification and validation establish the technical correctness and real-world credibility of a simulation study.

Definition

Verification vs. Validation

AspectVerificationValidation
Core QuestionDoes the program execute the conceptual model without bugs?Does the model reproduce real-world system behavior accurately?
Objects ComparedConceptual Model Computerized ProgramComputerized Model Real-World System
Primary MethodsCode walkthroughs, deterministic traces, unit tests, debugger runsExpert face validity, goodness-of-fit, -tests, Turing test
TimingDuring and immediately after model codingThroughout conceptualization, input analysis, and output testing

Following Sargent's modeling framework, four pillars support model credibility (stakeholder trust): (1) Conceptual Model Validation, (2) Computerized Model Verification, (3) Operational Validation, and (4) Data Validity.


7.2 Verification of simulation models

Because simulation programs contain concurrent event streams, queues, and random variates, standard software testing alone is insufficient. Banks, Carson, and Nelson recommend eight practical verification techniques:

  1. Peer Code Review / Structured Walk-Through: Have another analyst read the simulation code line by line to check branching logic, queue discipline, and unit consistency.
  2. Modular Top-Down Development and Unit Testing: Build and verify each subprogram (random variate generator, arrival routine, departure routine, statistics accumulator) independently before integration.
  3. Trace Debugging (Event-by-Event Trace): Print the simulation clock, event type, state variables, and Future Event List (FEL) at every event and compare line by line against a manual hand calculation.
  4. Replacing Random Variables with Constants (Deterministic Run): Substitute fixed constants (e.g., inter-arrival , service ) so exact waiting times and utilizations can be checked analytically.
  5. Checking Against Known Analytical Solutions: Run the stochastic model under simplified Markovian assumptions (, , Little's Law ) and verify simulated long-run averages match closed-form formulas.
  6. Extreme-Input and Boundary Testing: Feed zero arrivals, utilization (), or single-customer edge cases to verify the code never divides by zero, drops entities, or deadlocks.
  7. Graphical Animation: Animate entity movement, queues, and server state icons to spot impossible behaviors immediately (e.g., two customers occupying one single-server booth simultaneously).
  8. Interactive Symbolic Debugger: Use breakpoints and watch expressions to inspect state variables whenever counter-intuitive conditions arise.

7.3 Calibration and validation of models

Definition

Calibration vs. Validation

Practical Validation Methods


7.4 Naylor and finger validation process

Naylor and Finger (1967) established a classic three-step validation methodology widely tested in simulation courses:

Definition

Naylor and Finger’s Three-Step Validation Process

  1. Step A — Build a Model with High Face Validity: Develop a conceptual model that appears reasonable a priori to knowledgeable observers by combining:
    • Domain theory and established physical/queuing laws,
    • Direct observation of the real system, and
    • Structured interviews with operators, managers, and domain experts.
  2. Step B — Validate Model Assumptions: Test both structural assumptions (how components interact, queue disciplines, routing rules) and data assumptions (reliability of historical records, independence, and goodness-of-fit of input distributions via K–S or tests).
  3. Step C — Compare Model Input–Output Transformations with the Real System: Treat the simulation as an input–output black box. Feed historical or experimental inputs into the simulation and statistically compare simulated output measures against real-world output (or real-world sample ) using confidence intervals, one-sample or two-sample -tests, paired -tests, or Turing tests.

7.5 Validation: Errors

Because stochastic simulation validation relies on finite random samples, statistical tests are always exposed to two types of decision errors:

Type I and Type II Errors in Simulation Model Validation
Error TypeStatistical DefinitionSimulation MeaningStakeholder Risk
Type I Error ()Reject when is actually trueRejecting a valid model as inaccurateModel Builder’s Risk () — wastes effort rebuilding a good model
Type II Error ()Fail to reject when is actually falseAccepting an invalid model as validModel User’s Risk () — leads to costly bad real-world decisions

Controlling Validation Errors


Worked Out Examples

Example 1

1

In Step C of the Naylor and Finger validation process, a packaging line historically averages . Six independent simulation replications () produce throughputs:

Test model validity at significance level ().

Formulating hypotheses:

Computing sample mean and sample standard deviation :

Evaluating and deciding:

Since , we fail to reject .

Answer:

With (), there is no statistically significant difference between the simulation output () and the real system (); the model passes operational validation at .

Example 2

2

A real-world call center has known mean waiting time . A simulation analyst runs replications (known ) and tests at ().

  1. Find the acceptance interval for sample mean and state the Model Builder’s Risk.
  2. Suppose the simulation model actually contains a hidden bug causing its true mean to be . Compute the -bounds for the Type II error (, Model User’s Risk) (given ).

Finding the acceptance region and Model Builder’s Risk: Standard error of the mean is .

The Model Builder’s Risk is the Type I error probability ().

Computing Type II Error () when true model mean is :

Standardizing with and :

(The statistical power to catch this bug is .)

Answer:
  1. Acceptance interval: ; Model Builder's Risk ().
  2. Model User's Risk (Type II error) for is ().
Example 3

3

Exam-Style Design Question: Outline a complete verification and validation plan for a discrete-event simulation model of a university hospital Emergency Department (ED) before hospital management relies on it to hire new triage nurses.

Phase 1 — Verification ("Building the Model Right"):

  • Modular Unit Tests & Code Review: Peer-review patient triage priority rules and test the arrival/service variate generators separately.
  • Deterministic & Trace Run: Replace random patient arrivals and treatment times with fixed constants (e.g., arrivals, treatment) and trace the FEL event-by-event against hand calculations.
  • Little's Law & Animation Check: Run under steady-state parameters to verify , and inspect animation to confirm doctors never treat two patients simultaneously.

Phase 2 — Calibration & Validation via Naylor and Finger:

  • Face Validity: Review patient flowcharts (walk-in vs. ambulance priority, lab delays) with ED doctors and head nurses.
  • Assumptions Validation: Run K–S / goodness-of-fit tests on months of historical ED arrival timestamps and treatment logs to confirm fitted distributions.
  • Calibration vs. Holdout Validation: Calibrate shift-change delays using Months , then validate against holdout Months using a paired -test on average waiting time and a Turing test with ED managers.
Answer:

A sound ED V&V plan combines code reviews, deterministic FEL traces, analytical cross-checks, and animation for verification, followed by expert face validity, K–S/ input validation, out-of-sample calibration/testing, paired -tests, and a Turing test under Naylor and Finger's framework.


Exercises

Practice Exercises — Verification and Validation

Question 1

Clearly distinguish between verification, calibration, and validation of a simulation model, and describe five practical techniques used to verify simulation code.

Distinction:

  • Verification: Confirms the computer program matches the conceptual model ("Did we build the model right?").
  • Calibration: Tunes uncertain model parameters against historical data so model outputs match observed records.
  • Validation: Evaluates whether the calibrated model accurately predicts the real system on independent data ("Did we build the right model?").

Five Verification Techniques:

  1. Peer code walk-throughs; 2. Event-by-event FEL trace compared with hand calculation; 3. Deterministic constant-input runs; 4. Checking steady-state output against analytical or Little's Law; 5. Graphical animation and boundary-condition stress tests.
Answer:

Verification checks code against the conceptual model, calibration tunes parameters on historical data, and validation checks real-world predictive accuracy on independent data. Key verification methods include walkthroughs, FEL traces, deterministic runs, analytical checks, and animation.

Question 2

Explain the three-step validation process proposed by Naylor and Finger, and describe how the Turing test is conducted during operational validation.

Naylor and Finger’s Three Steps:

  1. High Face Validity: Ensure the model appears reasonable a priori to domain experts using theory, observation, and interviews.
  2. Validate Model Assumptions: Test structural rules (routing, queue discipline) and input data assumptions (independence, K–S or distribution fit).
  3. Compare Input–Output Transformations: Statistically compare simulated responses against real-system measurements via confidence intervals, -tests, or Turing tests.

Conducting the Turing Test: Unlabeled summary reports from both the real system and the simulation model (covering the same operating conditions) are shown to experienced system operators/managers. If the experts cannot reliably tell which report came from reality and which came from the simulator—and why—the model demonstrates operational validity.

Answer:

Naylor and Finger's process builds face validity, validates structural and input assumptions, and compares input–output behavior against the real system. In a Turing test, domain experts try to distinguish unlabeled real and simulated reports; inability to separate them supports validity.

Question 3

Explain Type I error () and Type II error () in simulation model validation. Why are they called Model Builder’s Risk and Model User’s Risk, and how can both be reduced simultaneously?

Type I vs. Type II Validation Errors:

  • Type I Error (, Model Builder’s Risk): Rejecting when the model is actually valid. It penalizes the model builder by forcing unnecessary rework.
  • Type II Error (, Model User’s Risk): Failing to reject when the model is actually invalid. It penalizes the decision-maker (user) who deploys a flawed model in practice.

Reducing Both Risks Simultaneously: For a fixed sample size , decreasing increases . The only way to decrease both and at the same time is to increase the number of independent simulation replications (which shrinks the standard error and increases statistical power ).

Answer:

Type I error (, Model Builder's Risk) rejects a valid model, while Type II error (, Model User's Risk) accepts an invalid model. Both risks can be reduced simultaneously only by increasing the number of replications .

Question 4

To validate an inventory simulation over historical weeks, an analyst pairs weekly real-system holding cost with simulated holding cost (using historical weekly demands) and obtains differences :

Perform a paired -test at () to test .

Computing sample mean and sample standard deviation ():

Evaluating the paired -statistic:

Since , do not reject .

Answer:

With (), the paired -test shows no significant difference between simulated and historical weekly costs at .

Question 5

A manufacturing assembly line has a known true cycle time of . Nine independent simulation runs yield with sample standard deviation .

  1. Compute the confidence interval for the model's mean cycle time ().
  2. Based on this interval (and ), should the model be accepted or rejected at ?

Constructing the confidence interval:

Testing against : Because lies inside the confidence interval (equivalently, ), we fail to reject .

Answer:
  1. confidence interval: .
  2. Since lies inside this interval (), the model is accepted (not rejected) at .
← Random Number (part 2: 6.4–6.6)Analysis of simulation output →