Probabilities are easy to assume. Their reliability must be demonstrated.
Financial risk models can generate precise-looking downside estimates. But a calculated probability is not necessarily a reliable prediction. The crucial question is whether the model’s forecasts have been tested against outcomes it could not have known in advance.
Our methodology: historical out-of-sample validation
At Asset Mechanics, the risk performance we report is based on rigorous out-of-sample testing, rather than assumptions about how well a model should perform.
For macroeconomic risk factors, such as land prices, house sales and interest rates, we use up to 115 years of historical data and report on 20 years of walk-forward out-of-sample validation. At each historical forecast date, the model uses only information available up until that time, allowing us to compare its predictions with subsequently realised outcomes across different forecast horizons.
For project-specific risks, we evaluate projects that have not been part of the model development set.
Our validation focuses on two complementary measures:
- Empirical coverage: Do realised outcomes breach predicted risk thresholds approximately as often as their stated probabilities imply?
- Quantile loss: How accurately are those thresholds predicted, and does the model outperform a simple historical benchmark?
This allows us to measure predictive performance rather than merely demonstrate that a model can produce probability distributions.
Our system cards document the data, validation periods, forecast horizons, empirical performance, benchmark comparisons and limitations of each model.
Why Monte Carlo simulation is not validation
Monte Carlo simulation is widely used to quantify financial risks. It generates probability distributions by repeatedly sampling from assumptions about future outcomes.
The problem arises when the resulting percentiles are presented as scientifically grounded risk estimates without testing whether the underlying assumptions are realistic.
Running thousands of simulations does not make an assumed probability more reliable. It only calculates the implications of that assumption more precisely.
Without historical out-of-sample testing or other appropriate empirical validation, there is no demonstrated basis for treating a simulated downside percentile as a reliable probability estimate.
Transparency and accountability
We believe that probabilistic forecasts used for long-term financial decisions should be judged by their demonstrated predictive performance, not by the complexity of the model producing them.
To our knowledge, we have not identified another provider that publicly documents 20 years of out-of-sample validation of P15 forecasts across prediction horizons of 1–10 years, reporting both empirical coverage and quantile (pinball) loss as part of its long-term macroeconomic risk forecast validation.
Our methodology and headline performance results are publicly described. Detailed model-specific evidence, benchmark comparisons and limitations are documented in our system cards, with detailed reports available to clients.
The value of a risk model is not that it produces probabilities, but that those probabilities have been tested.
