Reference Class Forecasting – What is ‘Comparable’?

Estimated reading time: 7 minutes

People learn through examples

To determine the best next step in a specific situation, it helps to have concrete examples of comparable situations. Suppose we have data on 20 similar cases and find that a particular choice worked well in 18 of them (90%). We would generally have more confidence in making that choice than if it had worked in only 11 out of 20 cases (55%). The historical cases provide an empirical basis for the decision, while how we act on that evidence also depends on our attitude toward risk. We discuss this in more detail in our article on risk aversion.

The importance of Reference Class Forecasting

When we face more complex situations or projects in which we have limited or no direct personal experience it becomes a bit harder. We often try to imagine how events will unfold and make an estimate largely based on our current perspective or our gut feeling. It turns out that people can be systematically too optimistic in such situations. Daniel Kahneman describes this as the planning fallacy and makes a crucial distinction between the “inside view” and the “outside view”¹.

• In the “inside view“, we focus on the specific case, its goal and its desired success. We often overestimate the gains, underestimate the costs, and overlook the effects of uncertainties and miscalculations. This is often an important reason why projects go over budget or exceed their planned timelines.
• In the “outside view“, instead of relying mainly on our own imagination, we focus on data from comparable historical situations. By creating a pool of most comparable cases and examining their outcomes, we can then obtain a forecast² and a distribution of possible outcomes for the case at hand. In other words, a good reference class provides an empirical ‘outside view’ that helps us forecast both the expected outcome and the associated risk.

The Reference Class Problem: What Is Truly Comparable?

While the benefits of an outside view are clear, traditional Reference Class Forecasting (RCF) faces a fundamental challenge: the reference class problem ³. How do we determine which historical projects are genuinely comparable to the project at hand?

If the definition of comparability is flawed from the outset, the statistical forecast will be equally flawed—regardless of how sophisticated the underlying mathematical formulas may be. In practice, this introduces human subjectivity in two ways:

• Predefined Criteria: Comparability is determined in advance based on broad characteristics that analysts expect to matter. However, such expectations do not, in themselves, constitute empirical evidence of predictive relevance.
• Under-specified Classes: Forcing unique projects into rigid categories mixes different risk profiles. This distorts the calculated budget contingencies (uplifts) and risk percentiles for the specific project.

The Traditional Approach: Manually Filtering by Project Characteristics

In standard practice, analysts attempt to address this problem by manually filtering databases based on readily observable project characteristics. Historical projects are grouped into broad, predefined categories (static buckets) based on:

• Project Size and Capacity: For example, selecting projects with a similar number of kilometers of railway track, number of stations, or initial budgets.
• Physical Complexity: Filtering by technical characteristics, such as the proportion of underground construction (tunnels) or the type of structure.
• Geographical Location: Grouping projects located in the same region or in areas with similar urban density.

Where Manual Selection Falls Short

With this approach, analysts must decide upfront which project characteristics matter most when selecting comparable projects. The process therefore relies heavily on individual judgment and assumptions, making it vulnerable to unconscious cherry-picking (selecting projects that reinforce existing expectations) and classifications that overlook important differences in context.

Moreover, broader risk factors—such as changing market conditions, macroeconomic shifts (including inflation and interest rates), or local environmental factors—are often overlooked simply because they fall outside standard project classifications. Standard indices may lag behind market developments, fail to reflect project-specific circumstances, and offer little insight into the likelihood and magnitude of future cost overruns and delays.

The Next Step Forward: Predictive Reference Class Retrieval (Predictive RCR)

To eliminate subjectivity in determining which situations are truly comparable, Asset Mechanics has taken a major step forward in Reference Class Forecasting with Predictive Reference Class Retrieval (Predictive RCR).

Rather than relying on analysts to decide upfront which project characteristics are relevant, the engine works with a broad and diverse range of data—from physical project characteristics and environmental factors to macroeconomic indicators such as interest rates and inflation. It then automatically identifies the optimal combination of characteristics that yields comparable situations with demonstrable predictive value.

• Automated Feature Selection: The engine independently identifies combinations of features with the greatest predictive value, allowing a wide range of available data to be considered. The algorithm separates meaningful predictive signals from noise. If local soil conditions or specific market circumstances prove more predictive than broad project categories, the model automatically adjusts the reference class accordingly.
• Empirical Validation Through Out-of-Sample Testing: This is not a black box that simply generates plausible relationships. Every selected feature and constructed reference class is rigorously tested out of sample. The engine simulates historical forecasts using only information that was genuinely available at the time each prediction would have been made, then evaluates accuracy against actual project outcomes. This produces auditable, verifiable, and statistically supported validation results for auditors and regulators.

From Comparable Situations to Evidence-Based Project Risk Assessments in 4 Steps

1. Bring Together the Available Data: Gather relevant project characteristics, historical project data, environmental factors, and macroeconomic indicators, such as interest rates and inflation, that may influence project costs, timelines, and risks.
2. Identify Predictively Comparable Situations: Determine which characteristics, and combinations of characteristics, have demonstrable predictive value. Use these insights to select historical situations that are genuinely relevant to the specific project.
3. Estimate Outcomes and Uncertainties: Use the selected comparable situations to generate forecasts and their associated probability distributions. This provides insight into expected outcomes, as well as the likelihood and potential magnitude of deviations.
4. Assess Overall Project Risk: Combine the relevant risks while explicitly accounting for dependencies between them. This provides a sound statistical basis for setting budget and schedule contingencies aligned with the desired level of confidence.

Comparison of Forecasting Methods

Feature	Traditional RCF	Standard ML / LLM	Predictive RCR
Selection of Comparable Situations	Predefined by the analyst	Learned relationships, without explicitly testing the predictive comparability of the reference class	Based on demonstrable out-of-sample predictive comparability
Basis for Forecasting	Outcomes from a predefined reference class	Patterns learned from data	Historical outcomes from a project-specific reference class
Auditability of the Reference Class	Yes, selected historical projects can be inspected	No (‘Black box’)	Yes, with insight into the selected historical projects and their predictive comparability
Risk Statistics	Empirical distribution of the selected reference class	Model-derived	Empirically grounded, context-specific probability distribution

Applications of Reference Class Forecasting

Research on major infrastructure projects in the United Kingdom shows that the use of Reference Class Forecasting has been associated with substantially lower average cost overruns. However, these findings relate to the construction phase, after projects had already been fully designed. ^{4 5}

With Predictive Reference Class Retrieval (Predictive RCR), we make reliable cost estimates and risk forecasts possible even in the early stages of project development.

We have successfully applied this approach across a range of complex challenges, from:

• early cost estimates for dike reinforcements
• land development risk
• regime-dependent interest rate risks.

The predictive performance across these applications has been empirically validated and documented in our RCF-AI System Cards.

Curious how AI-driven Reference Class Forecasting (RCF-AI) can deliver faster, more auditable risk assessments tailored to your specific context? Book a demo! We’ll show you how identifying predictively comparable situations leads to more robust, evidence-based risk forecasts.

---

Footnotes

1. For more examples of the planning fallacy, see also: Daniel Kahneman ‘Thinking, fast and slow’ chapter 23 ‘the outside view’ ↩︎
2. The same principles have substantially improved recent performance of artificial intelligence algorithms. See also: Vaswani (2017): ‘Attention is all you need’. We apply this principle more broadly and predict not only the expectation, but also the risk ↩︎
3. Reichenbach, H. (1949). The Theory of Probability: An Inquiry into the Logical and Mathematical Foundations of the Calculus of Probability. University of California Press. ↩︎
4. Connoly, J. and Newman F. 2023. Spending Review 2023: An Analysis of Cost Forecasting in Major Capital Projects & Programmes. National Investment Office, Government of Ireland ↩︎
5. Park, J.E. 2021. Curbing cost overruns in infrastructure investment: Has reference class forecasting delivered its promised success?. European Journal of Transport and Infrastructure Research. 21, 2 (Jun. 2021), 120–136. DOI:https://doi.org/10.18757/ejtir.2021.21.2.5504 ↩︎

---