How Conceptual Elegance Can Lead to Risk Blindness
14/Apr/18 13:29
We’ve spent a lot of time over our careers working with risks that are, at least in theory, easy to quantify, price, and transfer. These include hazard risks for which there is substantial historical data on the frequency of their occurrence, as well as market risks where historical data sets are also very large.
In these cases, the traditional way of mitigating unwanted risk exposure is to transfer it, via insurance or financial derivative contracts. This also makes it apparently straightforward to calculate an organization’s residual or retained risk after mitigation actions are taken. In turn, this makes is apparently easy to compare the total amount of residual/retained/net risk to a board’s “risk appetite” – for example, the maximum reduction in cash flow or equity market value to which it desires to be exposed over a given period of time (with, for example, a 95% degree of confidence).
Especially after the events surrounding the 2008 global financial crisis (or the collapse of Long Term Capital Management in 1998), we are all painfully aware that in practice, things are not this easy, even in the case of risks that are apparently easy to quantify, price, and transfer.
Some real-world complications include:
Consider the case of Carillion, the UK facilities management and construction services company that recently went into liquidation with almost GBP 7 billion in liabilities.
One of the principal causes of the company’s failure was cost overruns on major projects. The potential for such overruns had previously been recognized by the company’s management as a potentially existential risk.
However, in the company’s risk management process, the size of the residual/retained risk exposure was apparently much smaller than the gross exposure. But this wasn’t because most of the risk had been transferred to a counterparty via insurance or financial derivative contracts. Rather, it was because of the assumption that internal mitigation actions would significantly reduce the risk.
Thus, the board’s apparent belief that Carillion had a small exposure to existential project cost overrun risk seems to have been based on a series of implicit assumptions that critical mitigation actions (a) would be implemented; (b) in time; and (c) would have their expected risk reducing effects.
It is also critical to recognize the enormous difference in the accuracy with which transferable risks (e.g., hazard and market) and non-transferable risks (e.g., operational and strategic) can be quantified, in order to integrated them into an overall calculation of an organization’s retained risks relative to its risk appetite.
As we have shown, the quantification of risks for which large historical data sets are available is still problematic in many ways, and subject to an unknown degree of error, which exponentially grows over time.
But for many reasons, this challenge pales in comparison with those that confront us when we try to quantify of operational and strategic risks and the potential impact of actions taken to mitigate either their probability of occurrence of the potential negative impact if they materialize. Some of the most important challenges include:
With these observations in mind, let us return to Carillion.
In reviewing what we know so far about this failure (and we will know much more when various inquests and litigation cases are completed), two critical points stand out for us.
First, it is not as though the risk of large project cost overruns sinking a company is not well-recognized or well-documented. For example, Professor Bent Flyybjerg has extensively documented the regularity with which cost overruns occur on large projects (e.g., see his paper, “Over Budget, Over Time, Over, and Over Again: Managing Major Projects”), and project revenue recognition has for years been a major preoccupation of professional accounting standards bodies.
This leads us to infer (perhaps incorrectly), that Carillion’s management and board must have been very confident that these well-known risks were adequately mitigated by the plans the company had put in place to address them. This raises questions about the evidence that provided the basis for this high degree of confidence, as well as the actions taken to confirm that these plans were being implemented (we look forward to internal audit and compliance reports eventually being publicly disclosed).
Second, the Carillion failure highlights yet again the danger of putting too much trust in enterprise risk management models that attempt to quantify and aggregate very different hazard, market, operational, and strategic risks into a unified measure of “residual/retained risk” exposure that can be compared to an equally neat “risk appetite” number.
We continue to stress that when it comes to managing and governing risk, a desire for conceptual elegance is too often achieved at the cost of dangerous risk blindness that only becomes apparent when it is too late to avoid organizational failure.
Of course, this begs the question of what constitutes a better approach to the management and governance challenges posed by various types of risk. Here's a short summary of our view:
In these cases, the traditional way of mitigating unwanted risk exposure is to transfer it, via insurance or financial derivative contracts. This also makes it apparently straightforward to calculate an organization’s residual or retained risk after mitigation actions are taken. In turn, this makes is apparently easy to compare the total amount of residual/retained/net risk to a board’s “risk appetite” – for example, the maximum reduction in cash flow or equity market value to which it desires to be exposed over a given period of time (with, for example, a 95% degree of confidence).
Especially after the events surrounding the 2008 global financial crisis (or the collapse of Long Term Capital Management in 1998), we are all painfully aware that in practice, things are not this easy, even in the case of risks that are apparently easy to quantify, price, and transfer.
Some real-world complications include:
- Use of historical data sets that do not include extreme downside losses that a given system can produce;
- Evolution in the nature of the system over time that makes historical data an increasingly inaccurate guide to what may occur in the future;
- Use of inaccurate models to forecast future risks;
- Risks whose covariance changes, both over time and as a function of conditions (e.g., remember the saying that as conditions deteriorate and uncertainty increases, correlations move towards 1.0);
- The ability of risk transfer counterparties to make good on the payments they have contractually agreed to make should a risk materialize (e.g., the case of AIG and credit default swaps in 2008).
Consider the case of Carillion, the UK facilities management and construction services company that recently went into liquidation with almost GBP 7 billion in liabilities.
One of the principal causes of the company’s failure was cost overruns on major projects. The potential for such overruns had previously been recognized by the company’s management as a potentially existential risk.
However, in the company’s risk management process, the size of the residual/retained risk exposure was apparently much smaller than the gross exposure. But this wasn’t because most of the risk had been transferred to a counterparty via insurance or financial derivative contracts. Rather, it was because of the assumption that internal mitigation actions would significantly reduce the risk.
Thus, the board’s apparent belief that Carillion had a small exposure to existential project cost overrun risk seems to have been based on a series of implicit assumptions that critical mitigation actions (a) would be implemented; (b) in time; and (c) would have their expected risk reducing effects.
It is also critical to recognize the enormous difference in the accuracy with which transferable risks (e.g., hazard and market) and non-transferable risks (e.g., operational and strategic) can be quantified, in order to integrated them into an overall calculation of an organization’s retained risks relative to its risk appetite.
As we have shown, the quantification of risks for which large historical data sets are available is still problematic in many ways, and subject to an unknown degree of error, which exponentially grows over time.
But for many reasons, this challenge pales in comparison with those that confront us when we try to quantify of operational and strategic risks and the potential impact of actions taken to mitigate either their probability of occurrence of the potential negative impact if they materialize. Some of the most important challenges include:
- We can’t be confident that we have identified all the relevant risks, mainly for two reasons. On the operational front, organizations tend to become more complex as they grow, which gives rise to both new risks and new causal pathways for ones already identified. On the strategic front, the nature of the interacting complex adaptive systems within which a company exists (e.g., technological, economic, social, and political) guarantees that new risks will continuously emerge.
- In many cases, either reference case/base rate data on which we can ground our risk and mitigation impact quantification processes either don’t exist or if they do, are inevitably incomplete.
- The subjective estimates we are usually forced to use when attempting to quantify operational and strategic risks and the potential impact of mitigation actions are almost always affected by at least five individual, group, and organizational biases, including:
- Over-optimism (e.g., the level of the mean or median estimate);
- Overconfidence (e.g., the width of the range of possible outcomes);
- Confirmation/Desirability (we pay more attention and give more weight to information that supports our view, or the outcome we desire, and less to information that does not);
- Conformity (we hesitate to deviate from the prevailing group view); and
- A strong organizational desire to avoid errors of commission (i.e., false alarms about potential risks that don’t materialize) even though this automatically increases the likelihood of errors of omission (i.e., missed alarms about potential risks that actually occur).
- Complete quantification of the relationships between operational and strategic risks, and between them and hazard and market risks, and how these relationships could vary over different situations and over time is, from both an estimation and a computational perspective, a practical impossibility.
With these observations in mind, let us return to Carillion.
In reviewing what we know so far about this failure (and we will know much more when various inquests and litigation cases are completed), two critical points stand out for us.
First, it is not as though the risk of large project cost overruns sinking a company is not well-recognized or well-documented. For example, Professor Bent Flyybjerg has extensively documented the regularity with which cost overruns occur on large projects (e.g., see his paper, “Over Budget, Over Time, Over, and Over Again: Managing Major Projects”), and project revenue recognition has for years been a major preoccupation of professional accounting standards bodies.
This leads us to infer (perhaps incorrectly), that Carillion’s management and board must have been very confident that these well-known risks were adequately mitigated by the plans the company had put in place to address them. This raises questions about the evidence that provided the basis for this high degree of confidence, as well as the actions taken to confirm that these plans were being implemented (we look forward to internal audit and compliance reports eventually being publicly disclosed).
Second, the Carillion failure highlights yet again the danger of putting too much trust in enterprise risk management models that attempt to quantify and aggregate very different hazard, market, operational, and strategic risks into a unified measure of “residual/retained risk” exposure that can be compared to an equally neat “risk appetite” number.
We continue to stress that when it comes to managing and governing risk, a desire for conceptual elegance is too often achieved at the cost of dangerous risk blindness that only becomes apparent when it is too late to avoid organizational failure.
Of course, this begs the question of what constitutes a better approach to the management and governance challenges posed by various types of risk. Here's a short summary of our view:
- Use of quantitative Enterprise Risk models that aggregate gross and net exposures to hazard and market risks still makes sense, with the caveats noted above. Given the limitations of these models, their use should be complemented with other techniques, like scenario based stress testing.
- The general category of "operational risk" encompasses a very wide range of "things that could go wrong." Where such risks can be readily quantified, priced, and transferred, they should be included in the quantitative Enterprise Risk Management models and system. Where this is not the case, risk management should focus on establishing plans, processes, and systems that are robust to potential operational failures under a wide range of scenarios, while also building in various sources of resilience when robust design falls short and failures occur. There are many techniques that can be used to analyze and manage these risks, such as failure mode and effects analysis. And key actions to mitigate operational risks should be assessed and verified at regular intervals. A final focus should be on building an adaptive organization that can constantly identify and adjust to new operational risks created by increasing internal complexity and/or a changing external environment.
- When it comes to balancing risk exposure with a board's risk appetite, strategic risks present the most vexing challenge. As we have repeatedly noted, attempts at quantifying these risks are at best highly uncertain. It must therefore be the case that a board's decisions about strategic risk exposure versus risk appetite ultimately depends on directors' subjective judgment. But that does not mean such judgments must be unstructured. Consciously or not, they will usually reflect an assessment of the degree of imbalance between the goals being pursued, the resources available, and the strategy for employing those resources in light of the uncertainties facing the organization. The greater the degree of imbalance between goals, resources, and strategy, and the higher the external uncertainty, the greater an organization's strategic risk exposure.
blog comments powered by Disqus