Verification and Validation of Simulation Models
Open in the Padhaiko app7.1 Verification and validation
Even a mathematically elegant simulation fails if its code contains logic bugs or its assumptions do not match reality. Verification and validation establish the technical correctness and real-world credibility of a simulation study.
Verification vs. Validation
- Verification ("Building the model right"): The process of confirming that the computerized model (program code, event scheduling, arithmetic, and data structures) faithfully implements the conceptual model and specifications.
- Validation ("Building the right model"): The substantive process of confirming that the computerized model, within its domain of intended applicability, represents the real-world system accurately enough for the study's decision objectives.
| Aspect | Verification | Validation |
|---|---|---|
| Core Question | Does the program execute the conceptual model without bugs? | Does the model reproduce real-world system behavior accurately? |
| Objects Compared | Conceptual Model Computerized Program | Computerized Model Real-World System |
| Primary Methods | Code walkthroughs, deterministic traces, unit tests, debugger runs | Expert face validity, goodness-of-fit, -tests, Turing test |
| Timing | During and immediately after model coding | Throughout conceptualization, input analysis, and output testing |
Following Sargent's modeling framework, four pillars support model credibility (stakeholder trust): (1) Conceptual Model Validation, (2) Computerized Model Verification, (3) Operational Validation, and (4) Data Validity.
7.2 Verification of simulation models
Because simulation programs contain concurrent event streams, queues, and random variates, standard software testing alone is insufficient. Banks, Carson, and Nelson recommend eight practical verification techniques:
- Peer Code Review / Structured Walk-Through: Have another analyst read the simulation code line by line to check branching logic, queue discipline, and unit consistency.
- Modular Top-Down Development and Unit Testing: Build and verify each subprogram (random variate generator, arrival routine, departure routine, statistics accumulator) independently before integration.
- Trace Debugging (Event-by-Event Trace): Print the simulation clock, event type, state variables, and Future Event List (FEL) at every event and compare line by line against a manual hand calculation.
- Replacing Random Variables with Constants (Deterministic Run): Substitute fixed constants (e.g., inter-arrival , service ) so exact waiting times and utilizations can be checked analytically.
- Checking Against Known Analytical Solutions: Run the stochastic model under simplified Markovian assumptions (, , Little's Law ) and verify simulated long-run averages match closed-form formulas.
- Extreme-Input and Boundary Testing: Feed zero arrivals, utilization (), or single-customer edge cases to verify the code never divides by zero, drops entities, or deadlocks.
- Graphical Animation: Animate entity movement, queues, and server state icons to spot impossible behaviors immediately (e.g., two customers occupying one single-server booth simultaneously).
- Interactive Symbolic Debugger: Use breakpoints and watch expressions to inspect state variables whenever counter-intuitive conditions arise.
7.3 Calibration and validation of models
Calibration vs. Validation
- Calibration: The iterative process of comparing preliminary model outputs against observed historical data and tuning unmeasured or uncertain model parameters (e.g., driver reaction time, machine breakdown rate, service efficiency) until discrepancies are minimized.
- Validation: Testing the calibrated model against a new, independent dataset (out-of-sample data not used during calibration) to verify genuine predictive accuracy rather than overfitting.
Practical Validation Methods
- Face Validity: Subject-matter experts and system operators inspect model structure and outputs to judge whether behaviors appear reasonable.
- Sensitivity / Parameter-Variability Analysis: Vary key input parameters (e.g., increasing arrival rate by ) and verify that simulated outputs move in the physically expected direction and magnitude (e.g., mean queue delay increases).
- Historical Data Comparison: Drive the model with actual recorded historical input traces (trace-driven simulation) and compare simulated throughput or delay against historical records.
- Turing Test: Present side-by-side reports of real system logs and simulated output logs (without labels) to domain experts; if experts cannot reliably distinguish real from simulated reports, the model achieves strong operational validity.
7.4 Naylor and finger validation process
Naylor and Finger (1967) established a classic three-step validation methodology widely tested in simulation courses:
Naylor and Finger’s Three-Step Validation Process
- Step A — Build a Model with High Face Validity: Develop a conceptual model that appears reasonable a priori to knowledgeable observers by combining:
- Domain theory and established physical/queuing laws,
- Direct observation of the real system, and
- Structured interviews with operators, managers, and domain experts.
- Step B — Validate Model Assumptions: Test both structural assumptions (how components interact, queue disciplines, routing rules) and data assumptions (reliability of historical records, independence, and goodness-of-fit of input distributions via K–S or tests).
- Step C — Compare Model Input–Output Transformations with the Real System: Treat the simulation as an input–output black box. Feed historical or experimental inputs into the simulation and statistically compare simulated output measures against real-world output (or real-world sample ) using confidence intervals, one-sample or two-sample -tests, paired -tests, or Turing tests.
7.5 Validation: Errors
Because stochastic simulation validation relies on finite random samples, statistical tests are always exposed to two types of decision errors:
| Error Type | Statistical Definition | Simulation Meaning | Stakeholder Risk |
|---|---|---|---|
| Type I Error () | Reject when is actually true | Rejecting a valid model as inaccurate | Model Builder’s Risk () — wastes effort rebuilding a good model |
| Type II Error () | Fail to reject when is actually false | Accepting an invalid model as valid | Model User’s Risk () — leads to costly bad real-world decisions |
Controlling Validation Errors
- Statistical Power (): The probability of correctly detecting and rejecting a truly invalid model whose mean differs from by an important practical difference .
- Tradeoff Between and : For a fixed number of replications , tightening (lowering Type I risk, e.g., from to ) widens the acceptance interval and increases (Type II user risk)!
- Simultaneous Reduction via Sample Size : The only way to reduce both Model Builder's Risk () and Model User's Risk () simultaneously is to increase the number of independent replications (lowering standard error ). In practice, is often more dangerous than because deploying a flawed model causes operational losses.
Worked Out Examples
1
In Step C of the Naylor and Finger validation process, a packaging line historically averages . Six independent simulation replications () produce throughputs:
Test model validity at significance level ().
Formulating hypotheses:
Computing sample mean and sample standard deviation :
Evaluating and deciding:
Since , we fail to reject .
With (), there is no statistically significant difference between the simulation output () and the real system (); the model passes operational validation at .
2
A real-world call center has known mean waiting time . A simulation analyst runs replications (known ) and tests at ().
- Find the acceptance interval for sample mean and state the Model Builder’s Risk.
- Suppose the simulation model actually contains a hidden bug causing its true mean to be . Compute the -bounds for the Type II error (, Model User’s Risk) (given ).
Finding the acceptance region and Model Builder’s Risk: Standard error of the mean is .
The Model Builder’s Risk is the Type I error probability ().
Computing Type II Error () when true model mean is :
Standardizing with and :
(The statistical power to catch this bug is .)
- Acceptance interval: ; Model Builder's Risk ().
- Model User's Risk (Type II error) for is ().
3
Exam-Style Design Question: Outline a complete verification and validation plan for a discrete-event simulation model of a university hospital Emergency Department (ED) before hospital management relies on it to hire new triage nurses.
Phase 1 — Verification ("Building the Model Right"):
- Modular Unit Tests & Code Review: Peer-review patient triage priority rules and test the arrival/service variate generators separately.
- Deterministic & Trace Run: Replace random patient arrivals and treatment times with fixed constants (e.g., arrivals, treatment) and trace the FEL event-by-event against hand calculations.
- Little's Law & Animation Check: Run under steady-state parameters to verify , and inspect animation to confirm doctors never treat two patients simultaneously.
Phase 2 — Calibration & Validation via Naylor and Finger:
- Face Validity: Review patient flowcharts (walk-in vs. ambulance priority, lab delays) with ED doctors and head nurses.
- Assumptions Validation: Run K–S / goodness-of-fit tests on months of historical ED arrival timestamps and treatment logs to confirm fitted distributions.
- Calibration vs. Holdout Validation: Calibrate shift-change delays using Months , then validate against holdout Months using a paired -test on average waiting time and a Turing test with ED managers.
A sound ED V&V plan combines code reviews, deterministic FEL traces, analytical cross-checks, and animation for verification, followed by expert face validity, K–S/ input validation, out-of-sample calibration/testing, paired -tests, and a Turing test under Naylor and Finger's framework.
Exercises
Practice Exercises — Verification and Validation
Clearly distinguish between verification, calibration, and validation of a simulation model, and describe five practical techniques used to verify simulation code.
Distinction:
- Verification: Confirms the computer program matches the conceptual model ("Did we build the model right?").
- Calibration: Tunes uncertain model parameters against historical data so model outputs match observed records.
- Validation: Evaluates whether the calibrated model accurately predicts the real system on independent data ("Did we build the right model?").
Five Verification Techniques:
- Peer code walk-throughs; 2. Event-by-event FEL trace compared with hand calculation; 3. Deterministic constant-input runs; 4. Checking steady-state output against analytical or Little's Law; 5. Graphical animation and boundary-condition stress tests.
Verification checks code against the conceptual model, calibration tunes parameters on historical data, and validation checks real-world predictive accuracy on independent data. Key verification methods include walkthroughs, FEL traces, deterministic runs, analytical checks, and animation.
Explain the three-step validation process proposed by Naylor and Finger, and describe how the Turing test is conducted during operational validation.
Naylor and Finger’s Three Steps:
- High Face Validity: Ensure the model appears reasonable a priori to domain experts using theory, observation, and interviews.
- Validate Model Assumptions: Test structural rules (routing, queue discipline) and input data assumptions (independence, K–S or distribution fit).
- Compare Input–Output Transformations: Statistically compare simulated responses against real-system measurements via confidence intervals, -tests, or Turing tests.
Conducting the Turing Test: Unlabeled summary reports from both the real system and the simulation model (covering the same operating conditions) are shown to experienced system operators/managers. If the experts cannot reliably tell which report came from reality and which came from the simulator—and why—the model demonstrates operational validity.
Naylor and Finger's process builds face validity, validates structural and input assumptions, and compares input–output behavior against the real system. In a Turing test, domain experts try to distinguish unlabeled real and simulated reports; inability to separate them supports validity.
Explain Type I error () and Type II error () in simulation model validation. Why are they called Model Builder’s Risk and Model User’s Risk, and how can both be reduced simultaneously?
Type I vs. Type II Validation Errors:
- Type I Error (, Model Builder’s Risk): Rejecting when the model is actually valid. It penalizes the model builder by forcing unnecessary rework.
- Type II Error (, Model User’s Risk): Failing to reject when the model is actually invalid. It penalizes the decision-maker (user) who deploys a flawed model in practice.
Reducing Both Risks Simultaneously: For a fixed sample size , decreasing increases . The only way to decrease both and at the same time is to increase the number of independent simulation replications (which shrinks the standard error and increases statistical power ).
Type I error (, Model Builder's Risk) rejects a valid model, while Type II error (, Model User's Risk) accepts an invalid model. Both risks can be reduced simultaneously only by increasing the number of replications .
To validate an inventory simulation over historical weeks, an analyst pairs weekly real-system holding cost with simulated holding cost (using historical weekly demands) and obtains differences :
Perform a paired -test at () to test .
Computing sample mean and sample standard deviation ():
Evaluating the paired -statistic:
Since , do not reject .
With (), the paired -test shows no significant difference between simulated and historical weekly costs at .
A manufacturing assembly line has a known true cycle time of . Nine independent simulation runs yield with sample standard deviation .
- Compute the confidence interval for the model's mean cycle time ().
- Based on this interval (and ), should the model be accepted or rejected at ?
Constructing the confidence interval:
Testing against : Because lies inside the confidence interval (equivalently, ), we fail to reject .
- confidence interval: .
- Since lies inside this interval (), the model is accepted (not rejected) at .