Diagnostic gates

Diagnostic gates classify available numerical evidence under a versioned policy. The thresholds are screening decisions, not universal criteria for identification, forecast validity or suitability for a budget decision.

The runner collects diagnostics even when the configuration omits a diagnostics block. It then uses the packaged ammm_mmm_default_v2 profile. To choose a profile and override a threshold:

diagnostics:
  gates_file: ../diagnostic_gates.yml
  thresholds:
    mcmc_max_rhat: {warn: 1.01, fail: 1.05}

Relative profile paths resolve from the model YAML. Use a complete profile, such as the demo policy. Unknown checks, missing required thresholds, non-finite values and invalid ordering fail validation.

Rules and missing evidence

For upper-bound checks, equality with warn warns and equality with fail fails. Lower-bound checks reverse the comparison direction. Divergences are special: zero passes and any positive count fails. Missing or non-finite values are skipped, with an explanation, rather than being replaced by zero.

The worst observed status sets the floor: fail > warn > pass > skipped. A pass among available checks does not imply that skipped checks were satisfied. The inference method or sampler may make particular diagnostics unavailable.

Schema 1 retains the frozen earlier check set. Schema 2 adds tail ESS, tree-depth share, in-sample 94% predictive coverage and holdout NRMSE checks; it removes the residual maximum-ACF and WAIC-warning gates. Residual and WAIC values can still be retained as descriptive diagnostics. LOO remains a predictive criterion with an importance-sampling warning, not causal validation.

The holdout input is 35_holdout_validation/holdout_predictive_report.json. Without that stage, holdout checks are skipped. NRMSE uses the outcome range in the evaluated sample; the holdout/in-sample NRMSE ratio therefore also reflects different denominators.

Read the full report and resolved policy under 50_diagnostics, then review sampling and prediction. A warning is a reason to investigate its cause. Changing a threshold after seeing a result is a policy change that needs an explicit rationale, not a repair to the underlying evidence.

Override fields and metric meanings

diagnostics accepts gates_file (default null) and thresholds (default empty). Each overridden numeric check requires warn and accepts fail (default null); values must be finite. Profile resolution validates direction/order and schema membership, so an accepted field name alone does not make it applicable to every profile (src/ammm/mmm/diagnostic_gates.py:199, src/ammm/mmm/diagnostic_gates.py:440).

Numeric check key	Quantity	Failure direction
`design_max_vif`	Maximum raw-input variance inflation factor	Upper
`design_condition_number`	Raw-input screening design condition number	Upper
`mcmc_divergence_count`	Retained divergent transitions	Any positive count in the zero-tolerance policy
`mcmc_max_rhat`	Maximum parameter R-hat	Upper
`mcmc_min_ess_bulk`, `mcmc_min_ess_tail`	Minimum bulk/tail effective sample size	Lower
`mcmc_bfmi_min`	Minimum chain energy Bayesian fraction of missing information	Lower
`mcmc_max_treedepth_share`	Fraction of draws reaching maximum tree depth	Upper
`bayesian_pareto_k_max`	Maximum importance-sampling Pareto-k estimate	Upper
`predictive_nrmse`, `holdout_nrmse`	RMSE divided by evaluated observed outcome range	Upper
`predictive_coverage_94`	In-sample 94% predictive interval coverage	Lower
`holdout_nrmse_ratio`	Holdout NRMSE divided by in-sample NRMSE	Upper
`residual_ljung_box_p`	Minimum residual Ljung-Box p-value	Lower
`residual_max_abs_acf`	Maximum absolute residual autocorrelation; schema-1 legacy check	Upper

These are the numeric override keys at src/ammm/mmm/diagnostic_gates.py:215. Boolean warning flags are evaluated separately rather than accepting arbitrary numeric thresholds. The packaged default policy contains the active version-2 values; retain the resolved file when overrides are used. The troubleshooting guide explains what to investigate when a check fails.

Implementation reference at 7cb7f20: src/ammm/mmm/diagnostic_gates.py:507, src/ammm/pipeline/stages/core.py:659.