Reference Class Forecasting

Estimated reading time: 6 minutes

People learn through examples

To determine the best next step in a specific situation, it helps to have examples of comparable situations. Suppose we have data on 20 similar cases and find that a particular choice worked well in 18 of them (90%). We would generally have more confidence in making that choice than if it had worked in only 11 out of 20 cases (55%). The historical cases provide an empirical basis for the decision, while how we act on that evidence also depends on our attitude toward risk. We discuss this in more detail in our article on risk aversion.

The importance of Reference Class Forecasting

For many daily activities, we gain experience from childhood. We make many choices e.g. walking, cycling or driving, almost on autopilot. Even in unexpected situations, many of us adapt without thinking much about it, because we have implicitly learned which factors matter, what the risks are, and what tends to work best in different situations.

However, we also face situations where we don’t have a lot of experience. In those cases we often try to imagine how events will unfold and make an estimate largely based on our gut feeling. It turns out that people can be systematically too optimistic in such situations. Daniel Kahneman describes this as the planning fallacy and the distinction between the “inside view” and the “outside view”¹.

In the “inside view”, we focus on the specific case, its goal and its desired success. We often overestimate the gains, underestimate the costs, and overlook the effects of possible errors and miscalculations. This often causes projects to go over budget or to exceed their planned timelines.

In the “outside view”, instead of relying mainly on our own imagination, we focus on data from comparable historical situations. By creating a pool of most comparable cases and examining their outcomes, we can then obtain a forecast² and a distribution of possible outcomes for the case at hand. In other words, a good reference class provides an empirical ‘outside view’ that helps us forecast both the expected outcome and the associated risk.

How do we construct a reference class?

If we look at a new business case or project, we ideally want to search for sufficient ‘comparable’ projects. However, these are not always available. This doesn’t have to be a deal-breaker though.

To know if something is ‘comparable’, we just need to 1) focus on the most important risk factors, and 2) create reference classes on those risk factors. In many cases we can form a reference class on the underlying risk factors. Collecting sufficient comparable cases will enable us to forecast the expected value and the risks around it.

Examples of reference classes on risk factors

For some risk factors like interest, inflation and land prices we can use publicly available data. We use theregime-based method to select the most comparable years to create a reference class for the current situation. For other risk factors we might have to obtain relevant data by searching in yearly financial reports. And in other cases we can collect data through other internal or external sources.

Reference class forecasting – summary

After identifying the main risk factors and constructing their reference classes we can now obtain a good quantitative risk estimation for the whole project. In summary we carry out the following key steps:

Applications of reference class forecasting

Reference class forecasting is applied successfully in the UK for risk assessments on large infrastructure projects³. Public reports suggest average cost overruns fell (38% to 5%) after introducing reference class forecasting for major UK infrastructure projects ⁴.

In addition, our model validations show that this method can even lead to substantial improvements in market risk models making them more adaptable to changing conditions. It brings down model costs while at the same time estimating the risks more accurately.

Over the years, we have applied reference class forecasting in various projects. We have refined and validated the methodology and apply it now to a wide range of risk factors. Consider, for example, quantitative risk analyses on large projects for municipalities, AI-driven dike cost estimations, land development risk and regime-dependent interest rate riskcalculations.

If you are also interested in contextual cost and risk estimate, let us know! We are happy to show how AI-driven reference class forecasting (RCF-AI) will deliver not only faster but also more accurate and more adaptable risk estimates for your context.

---

Footnotes

1. For more examples of the planning fallacy, see also: Daniel Kahneman ‘Thinking, fast and slow’ chapter 23 ‘the outside view’ ↩︎
2. The same principles have substantially improved recent performance of artificial intelligence algorithms. See also: Vaswani (2018): ‘Attention is all you need’. We apply this principle more broadly and predict not only the expectation, but also the risk ↩︎
3. Connoly, J. and Newman F. 2023. Spending Review 2023: An Analysis of Cost Forecasting in Major Capital Projects & Programmes. National Investment Office, Government of Ireland ↩︎
4. Park, J.E. 2021. Curbing cost overruns in infrastructure investment: Has reference class forecasting delivered its promised success?. European Journal of Transport and Infrastructure Research. 21, 2 (Jun. 2021), 120–136. DOI:https://doi.org/10.18757/ejtir.2021.21.2.5504 ↩︎

---