Site Overlay

Reference Class Forecasting

Estimated reading time: 6 minutes

In this article, we look at reference class forecasting. A ‘reference class’ refers to a pool of previous situations which are sufficiently similar to the one at hand. We obtain a forecast by looking at the outcomes of that pool. Therefore it is an important method that helps us to make contextual predictions. We apply it continuously in our daily lives, and if used adequately, we can also use it to predict the course of a business case or project. It has similarity with ‘benchmark analysis’ or ‘benchmarking’ which is often used in investments context. And it has some similarity with the term ‘precedent analysis’ which is used in architecture and legal practise. It helps us to predict not only the expected value but also the risk around it.

People learn through examples

People are able to master a new subject, skill or profession relatively quickly. We generally do this by learning from experiences and examples. With enough comparable examples, we can see what can happen in practice and what works best under different situations. For example, many people learn a sport, language, or a musical instrument by making use of this predefined learning framework that has proven successful over time.

Reference class

To determine the best next step in a specific situation, it helps if we have examples of similar situations. For example, if we have data on 20 similar situations, and we know that a specific choice works well in 18 out of 20 times (90%), we are more likely to make that choice. But, if it works out right in 11 out of 20 times (55%), we would would be much more reluctant. In addition, it appears that our risk-averse behaviour also plays a role in decision-making. We  discuss this inmore detail in the article on risk aversion.

The importance of Reference Class Forecasting

For many daily activities, we gain experience from childhood. We make many choices e.g. walking, cycling or driving on autopilot. Even in unexpected situations, many of us can easily adapt without thinking much about it. We can because we have implicitly learned the underlying drivers and their risks, and know what would be best in a new situation.

However,  we also face situations where we don’t have a lot of experience. In those cases we often try to imagine the course of action and make an estimate based on our gut feeling. It often turns out that people in those cases act too optimistically. Daniel Kahneman calls this the planning fallacy and describes it as the difference between the “inside view” and the “outside view”1.

In the “inside view”, we, as involved experts, focus on the goal, on the desired success. We overestimate the gains and underestimate the costs, and often don’t oversee the effects of possible errors and miscalculations. This often causes projects to go over budget and exceed previously set project timelines.

In the “outside view”, instead of being mainly guided by our own imagination, we focus much more on collecting data of similar situations. By creating a pool of “most similar” cases and their outcomes, we can then obtain a forecast2 and a distribution of possible outcomes given a certain scenario. In other words, with a good reference class, we create an ‘outside view’ which enables us to better forecast the risk.

How do we construct a reference class?

If we look at a new business case or project, we ideally want to search for sufficient ‘comparable’ projects. However, these are not always available. This doesn’t have to be a deal-breaker though.

To know if something is ‘comparable’, we just need to 1) focus on the most important risk factors, and 2) create reference classes on those risk factors. In many cases we can form a reference class on the underlying risk factors. Collecting sufficient similar cases will enable us to forecast the expected value and the risks around it.

Examples of reference classes on risk factors

  • If the project is sensitive to inflation, we search for multiple comparable periods that give information about potential subsequent developments of the inflation
  • If the project is sensitive to financing costs, we want to know how interest can develop in comparable situations.
  • if the project depends on land purchases, we want to take into account the macro-economic context.
  • If the project is carried out in a populated area where local residents could object against the plans, it is useful to collect data about the range of additional costs made in similar situations.
  • Or, if thenumber of visitors is an important succes factor for project, it would be useful to obtain existing similar facilities with comparable characteristics.

For some risk factors like interest, inflation and land prices we can use publicly available data. We use theregime-method to select the most comparable years to create a reference class for the current situation. For other risk factors we might have to obtain relevant data by searching in yearly financial reports. And in other cases we can collect data through other internal or external sources.

Reference class forecasting – summary

After identifying the main risk factors and constructing their reference classes we can now obtain a good quantitative risk estimation for the whole project. In summary we carry out the following key steps:

How to obtain a risk estimation using reference class forecasting?

  1. Identify the most important risk factors

    Identify the top underlying risk factors that the project is most sensitive to.

  2. Create a reference class for each risk factor:

    For each underlying risk factor, collect historical data and select sufficient comparable observations to create a robust reference class.

  3. Forecast the expectation and the risk

    Obtain a distribution on every reference class to obtain a forecast of the expectation and the risk.

  4. Calculate the total risk

    Aggregate the risks taking into account the correlation benefits.

Applications of reference class forecasting

Reference class forecasting is applied successfully in the UK for risk assessments on large infrastructure projects3. It is reported that average cost overrun declined from 38% to 5% following the introduction of reference class forecasting4.

In addition, our model validations show that this method can even lead to substantial improvements in market risk models making them more adaptable to changing conditions. It brings down model costs while at the same time estimating the risks more accurately.

Over the years, we have applied reference class forecasting in various projects. We have refined and validated the methodology and apply it now to a wide range of risk factors. Consider, for example, quantitative risk analyses on large projects, the calculation of the land development risk and regime-dependent interest rate risk.

If you are also interested in contextual risk predictions, let us know! We are happy to show how data-driven analysis using reference classes can help obtain both more accurate and more adaptable risk estimations.

Footnotes

  1. For more examples of the planning fallacy, see also: Daniel Kahneman ‘Thinking, fast and slow’ chapter 23 ‘the outer view’ ↩︎
  2. The same principles have substantially improved recent performance of artificial intelligence algorithms. See also: Vaswani (2018): ‘Attention is all you need’. We apply this principle more broadly and predict not only the expectation, but also the risk ↩︎
  3. Connoly, J. and Newman F. 2023. Spending Review 2023: An Analysis of Cost Forecasting in Major Capital Projects & Programmes. National Investment Office, Government of Ireland ↩︎
  4. Park, J.E. 2021. Curbing cost overruns in infrastructure investment: Has reference class forecasting delivered its promised success?. European Journal of Transport and Infrastructure Research. 21, 2 (Jun. 2021), 120–136. DOI:https://doi.org/10.18757/ejtir.2021.21.2.5504 ↩︎
×