Skip to content

Verification and Validation

Source: PHY653B Ch. 1

Intuition

Two different questions, constantly confused, with different methods and different failure modes:

  • Verification: am I solving the equations right? A pure mathematics question. No experiment is involved. The answer is a convergence rate.
  • Validation: am I solving the right equations? A physics question. Requires data from the world, and no amount of numerical care can substitute for it.

A code can be perfectly verified and completely invalid: a beautifully converged solution of the wrong model. A code can also agree with experiment while being wrong — two errors cancelling is the classic way to publish something that will not reproduce.

Verification: observed order of accuracy

Do not check that the error is "small". Check that it shrinks at the rate the method promises. For a scheme of order \(p\), halving the grid should cut the error by \(2^p\):

\[p_{\rm obs} = \frac{\ln\left(E_{2h}/E_h\right)}{\ln 2}\]

Run at three resolutions, compute \(p_{\rm obs}\), and compare with the design order. This is the single most informative test you can run on a numerical code, because it fails loudly for almost every kind of bug. A sign error, a wrong boundary stencil, a mis-indexed array — all of them usually leave the answer converging at order 1 or not converging at all, even when the solution "looks right".

If your second-order scheme shows \(p_{\rm obs} = 1\), you have a first-order bug somewhere, regardless of how plausible the pictures are.

What conservation checks catch — and what they miss

Monitoring conserved quantities is cheap and worth doing, but be clear about its power:

Catches: gross indexing errors, boundary leaks, sign errors in fluxes, instability onset.

Misses: anything that respects the conservation law while being wrong. A scheme can conserve energy exactly and still put it in the wrong place, at the wrong time, with the wrong spectrum. Symplectic integrators are the standard example: excellent energy behaviour, and still capable of an \(O(\Delta t^2)\) phase error that ruins a frequency measurement.

Conservation is a necessary condition dressed up as a sufficient one. Treat a clean energy trace as evidence you have not made a catastrophic error, not as evidence you are right.

Validation: the benchmark ladder

Validate against progressively less forgiving references:

  1. Analytic solutions. Landau damping rates, Poiseuille flow, the Brio–Wu shock tube. Exact, unambiguous, and non-negotiable — see the benchmark examples.
  2. Manufactured solutions. When no analytic solution exists, make one.
  3. Other codes. Useful, but community-wide shared bugs are real.
  4. Experiment. The only true validation, and the one where the comparison is hardest, because a diagnostic measures something subtly different from what your code outputs — hence synthetic diagnostics, which apply the instrument's response to your simulation rather than comparing raw fields to processed data.

Anatomy of a defensible number

Chapter 1's checklist, and a fair summary of what separates a result from a plot:

  • the benchmark it reproduces, and to what tolerance
  • the observed order of accuracy, not the claimed one
  • resolution independence: the result does not move when you refine
  • an uncertainty, from a stated method — fit error, ensemble spread, resolution scan
  • the regime of validity, stated in dimensionless parameters
  • enough information for someone else to reproduce it: version, parameters, seed

Common mistakes

  • "The error is 10⁻⁶, so the code is verified." Small error at one resolution says nothing. It is the rate that verifies.
  • Validating before verifying. Agreement with experiment from an unverified code is luck you cannot bank.
  • Refining the grid to fix a model error. Convergence converges you to the answer of the equations you actually wrote, however wrong those are.
  • Reporting \(R^2\) as an uncertainty. See growth-rate fitting — \(R^2\) can be 0.999 on a fit whose slope is meaningless.

Knowledge graph position

Prerequisites: convergence and error. Leads to: every benchmark in the course; the discipline underlying all of Chapters 2–9.

Quiz

Q1 (conceptual). Your second-order code shows \(p_{\rm obs} = 1.0\). What does that tell you, and what does it not?

Answer

It tells you there is a first-order error contaminating the solution — commonly a boundary condition, a mis-centred stencil, or an operator-splitting error. It does not tell you the physics is wrong; the model could be perfect and the implementation broken. That is precisely why verification comes before validation.

Q2 (conceptual). Why is exact energy conservation not sufficient evidence of correctness?

Answer

Conservation constrains one scalar; a solution has enormously more freedom than that. A symplectic integrator conserves energy beautifully while accumulating phase error, so a frequency measured from it can be systematically wrong with a flat energy trace.

Q3 (MCQ). A synthetic diagnostic is used in order to:

  • (a) generate fake data when the experiment fails
  • (b) apply the instrument's actual response to simulation output, so like is compared with like
  • (c) speed up the simulation
  • (d) replace experimental validation
Answer

(b). Real instruments integrate along lines of sight, have finite resolution, and measure convolved quantities. Comparing a raw simulated field to a processed measurement compares two different things and hides real disagreement.