October 2019
Review of “Forecasting in Social Settings: The State of the Art” by Makridakis et al
30/Oct/19 17:54
In our course on Strategic Risk Management and Governance, we note the very substantial challenge of forecasting the future behavior of complex adaptive systems made up of human beings and their organizations. There are many reasons for this, including:
Taken together, these factors usually cause the accuracy of forecasts of complex adaptive system behavior to exponentially decline as the time horizon lengthens.
Given this background, we read the new paper by Makridakis and his colleagues with great interest.
At the outset, the authors clearly state that, “although forecasting in the physical sciences can attain amazing levels of accuracy, such is not the case in social contexts, where practically all predictions are uncertain, and a good number can be unambiguously wrong.”
There are a number of reasons for this. “First, there is usually a limited theoretical basis for presenting a causal or underlying mechanism” for the target variable being forecasted. “Thus we rely on statistical approximations that roughly describe what we observe, but may not represent a causal [process].” Second, “despite the deluge of data that is available today, much of this information does not concern what we want to forecast directly…Third, what we are trying to forecast is often affected by the forecasts themselves…Such feedback does not occur in weather forecasts…For these reasons, social science forecasts are unlikely to ever be as accurate as forecasts in the physical sciences, and the potential for improvements in accuracy is somewhat limited.”
The authors also note that when it comes to forecasting social systems, “unless uncertainty is expressed clearly and unambiguously, forecasting is not far removed from fortune-telling. However, uncertainty about judgmental forecasts of social system behavior is likely to be “underestimated greatly for two reasons.”
“First, our attitude to extrapolating in a linear fashion from the present to the future, and second, our fear of the unknown and our psychological need to reduce the anxiety associated with such a fear by believing we can control the future by predicting it accurately.”
Use of statistical instead of judgmental forecasting models improves the treatment of uncertainty, but this approach is far from perfect. The authors claim that, “there are at least three reasons for standard statistical models’ underestimations of the uncertainty:”
The paper also includes sections on different types of uncertainty, the challenges of incorporating causality into forecasting models, and the difficulty of predicting one off and extreme events.
In sum, the authors have produced an excellent (and extensively referenced) overview of the current state of the art of forecasting in social settings.
- Agents pursue multiple goals, with different incentives and priorities, and may change their goals and priorities over time as the system evolves;
- When deciding on actions to achieve their goals, agents differ in terms of the range of experiences they can draw on, and their cognitive ability to reason multiple time steps ahead about the likely consequences of their actions;
- Agents differ in their perceptions of the environment, and their beliefs about the future;
- Agents differ in the structure of their social networks, which also evolve over time (more technically, the data generating process in complex adaptive systems is non-stationary, which reduced the usefulness of historical results as a guide to future outcomes);
- Agents decide on their actions based not only on rational calculation, but also on their emotional reactions to competing narratives as well as the potential social impacts of their decisions;
- Agents differ in their desire to conform to the beliefs and copy the actions of other members of their group, with the latter typically increasing with the level of perceived uncertainty;
- Social feedback loops can produce emergent non-linear collective phenomena like herding, fads, booms and busts. These extreme events have been extremely hard to consistently forecast.
Taken together, these factors usually cause the accuracy of forecasts of complex adaptive system behavior to exponentially decline as the time horizon lengthens.
Given this background, we read the new paper by Makridakis and his colleagues with great interest.
At the outset, the authors clearly state that, “although forecasting in the physical sciences can attain amazing levels of accuracy, such is not the case in social contexts, where practically all predictions are uncertain, and a good number can be unambiguously wrong.”
There are a number of reasons for this. “First, there is usually a limited theoretical basis for presenting a causal or underlying mechanism” for the target variable being forecasted. “Thus we rely on statistical approximations that roughly describe what we observe, but may not represent a causal [process].” Second, “despite the deluge of data that is available today, much of this information does not concern what we want to forecast directly…Third, what we are trying to forecast is often affected by the forecasts themselves…Such feedback does not occur in weather forecasts…For these reasons, social science forecasts are unlikely to ever be as accurate as forecasts in the physical sciences, and the potential for improvements in accuracy is somewhat limited.”
The authors also note that when it comes to forecasting social systems, “unless uncertainty is expressed clearly and unambiguously, forecasting is not far removed from fortune-telling. However, uncertainty about judgmental forecasts of social system behavior is likely to be “underestimated greatly for two reasons.”
“First, our attitude to extrapolating in a linear fashion from the present to the future, and second, our fear of the unknown and our psychological need to reduce the anxiety associated with such a fear by believing we can control the future by predicting it accurately.”
Use of statistical instead of judgmental forecasting models improves the treatment of uncertainty, but this approach is far from perfect. The authors claim that, “there are at least three reasons for standard statistical models’ underestimations of the uncertainty:”
- “Probably the biggest factor is that model uncertainty is not taken into account. The prediction intervals are produced under the assumption that the model is ‘‘correct’’, which clearly is never the case.” The authors note that combining forecasts made using different models reduces this uncertainty.
- “Even if the model is specified correctly, the parameters must be estimated, and also the parameter uncertainty is rarely accounted for in time series forecasting models.” However, techniques like Monte Carlo simulation allow parameter uncertainty to be made explicit.
- “Most prediction intervals are produced under the assumption of Gaussian [normally distributed] errors. When this assumption is not correct, the prediction interval coverage will usually be underestimated, especially when the errors have a fat-tailed distribution [as is often the case in complex adaptive systems, which tend to produce outcomes that follow a Pareto/power law rather than a normal/bell curve distribution].”
The paper also includes sections on different types of uncertainty, the challenges of incorporating causality into forecasting models, and the difficulty of predicting one off and extreme events.
In sum, the authors have produced an excellent (and extensively referenced) overview of the current state of the art of forecasting in social settings.
Comments
The Critical Importance of Process in Board Decision Making
21/Oct/19 08:45
When we were researching the relationship between Board Chairs and CEOs we asked our interviewees about the processes they used in the boardroom. Mostly this elicited a response along the lines of “We have an agenda and we take minutes”. It seems most Boards of directors confuse process with procedure. Procedure may be involved but process is a much broader concept; it is a structured method or sequence of steps or activities, but not used for the purpose of administrative consistency or compliance. Process is used for quality of outcome.
There is perhaps a tendency for directors to view the discipline of process as a constraint on the free reign of experienced judgement. If so, this would be a mistake. A very wide body of experience and research has demonstrated that Board decision making is no less subject to the pitfalls of cognitive and behavioural limitations and biases than any other human activity. In certain important respects, in specific cases, Board decision making has been shown to be worse than in other, similarly important contexts. This is especially true of the performance of Boards as regards Strategic Risk.
There are two compelling reasons for Boards to take another look at their decision making processes – the first is the consequences for directors when their processes are absent or poor and the second is the real potential for improved Board effectiveness.
Traditionally, Board decision making has been defended in practice from external legal scrutiny by a convention, sometimes reinforced at common (i.e. judge made) law, that “business judgement” was not judiciable. This assumption has been long established. Yet even if it was ever wholly true, which is debatable, clear political and regulatory trends are moving against it.
In the UK, statute law has been enacted to both clarify and tighten directors’ accountabilities. Consequently, as recent research* (led by Professor Joan Loughrey at the University of Leeds) has highlighted, there has been dramatic growth in directors’ decisions being challenged in the courts. In less than a decade such cases grew by a factor of 10. Moreover, when directors are taken to court the probability is that they will lose. 63% of cases reviewed by Prof. Loughrey’s team were found against the directors, i.e. they were found liable for the consequences of their decisions. The proportion of cases being found against directors is also increasing. “Business Judgement”, at least in the UK, is no longer (if it ever really was) a blanket defence.
Even in the US, the legal environment is moving in the direction of greater director liability, especially in the arena of risk governance. The Supreme Court of Delaware, long a bell-weather for the legal approach to corporate governance in the US due to the large number of corporations domiciled in the state, has tended to err in defence of the “business judgement” principle. Historically the so-called Caremark doctrine held that directors would not be held liable for a failure in risk oversight unless there was “… an utter failure to ensure … [ ] … a reasonable system exists.” Now that seems to be changing too. In a recent judgement** the court found that “… directors must make a good faith effort to implement a [risk] oversight system and monitor it themselves.” This new ruling places the onus of accountability squarely on the Board to demonstrate an effective process is in place.
What does this mean for Board decision making? For a start it means much more than simply recording decisions in minutes. Prof. Loughrey’s team found that when cases were found in favour of directors it was most frequently because the defendants could show that they had followed a clear decision making process, supported by relevant and timely information and advice.
Thus, there is a clear case for Boards to pay greater attention to the appropriateness and rigour of their decision making processes, especially when critical strategic issues and risks are involved. Yet even if self-protection is not sufficient justification there is another good reason for Boards to up their decision making process game. Better processes produce better outcomes.
Nobel prize winner and doyen of the behavioural science underpinning human decision making, Daniel Kahneman recently published*** on why disciplined decision making processes are required. The fruits of his over 30 years of research give Kahneman deep insights into the cognitive and behavioural biases that afflict most human decision making. In the context of high stakes, high impact strategic Board-level decision making, these biases are frequently exacerbated by the individual egos, personalities and group dynamics involved. To counter these conditions Kahneman and his co-authors set out a compelling case for why decision making processes are needed.
In case after case of major corporate failure, the root cause can be found to be poor Board decision making. Yet when Boards get it wrong it is not just the individual directors who may have cause for regret. The consequences and costs fall far more widely than on the directors involved. Indeed, they are most frequently cushioned from financial impacts; employees, suppliers, customers and society at large are not. It can be argued that Boards and directors have a clear moral responsibility to improve their critical decision making processes.
* Business Judgement and the Courts; Centre for Business Law and Practice, University of Leeds; University of Liverpool Management School
** Marchand v. Barnhill, No. 533, 2018 (Del. June 19, 2019)
*** A Structured Approach to Strategic Decisions; Daniel Kahneman, Dan Lovallo, Olivier Sibony; MIT Sloan Management Review, March 04, 2019
There is perhaps a tendency for directors to view the discipline of process as a constraint on the free reign of experienced judgement. If so, this would be a mistake. A very wide body of experience and research has demonstrated that Board decision making is no less subject to the pitfalls of cognitive and behavioural limitations and biases than any other human activity. In certain important respects, in specific cases, Board decision making has been shown to be worse than in other, similarly important contexts. This is especially true of the performance of Boards as regards Strategic Risk.
There are two compelling reasons for Boards to take another look at their decision making processes – the first is the consequences for directors when their processes are absent or poor and the second is the real potential for improved Board effectiveness.
Traditionally, Board decision making has been defended in practice from external legal scrutiny by a convention, sometimes reinforced at common (i.e. judge made) law, that “business judgement” was not judiciable. This assumption has been long established. Yet even if it was ever wholly true, which is debatable, clear political and regulatory trends are moving against it.
In the UK, statute law has been enacted to both clarify and tighten directors’ accountabilities. Consequently, as recent research* (led by Professor Joan Loughrey at the University of Leeds) has highlighted, there has been dramatic growth in directors’ decisions being challenged in the courts. In less than a decade such cases grew by a factor of 10. Moreover, when directors are taken to court the probability is that they will lose. 63% of cases reviewed by Prof. Loughrey’s team were found against the directors, i.e. they were found liable for the consequences of their decisions. The proportion of cases being found against directors is also increasing. “Business Judgement”, at least in the UK, is no longer (if it ever really was) a blanket defence.
Even in the US, the legal environment is moving in the direction of greater director liability, especially in the arena of risk governance. The Supreme Court of Delaware, long a bell-weather for the legal approach to corporate governance in the US due to the large number of corporations domiciled in the state, has tended to err in defence of the “business judgement” principle. Historically the so-called Caremark doctrine held that directors would not be held liable for a failure in risk oversight unless there was “… an utter failure to ensure … [ ] … a reasonable system exists.” Now that seems to be changing too. In a recent judgement** the court found that “… directors must make a good faith effort to implement a [risk] oversight system and monitor it themselves.” This new ruling places the onus of accountability squarely on the Board to demonstrate an effective process is in place.
What does this mean for Board decision making? For a start it means much more than simply recording decisions in minutes. Prof. Loughrey’s team found that when cases were found in favour of directors it was most frequently because the defendants could show that they had followed a clear decision making process, supported by relevant and timely information and advice.
Thus, there is a clear case for Boards to pay greater attention to the appropriateness and rigour of their decision making processes, especially when critical strategic issues and risks are involved. Yet even if self-protection is not sufficient justification there is another good reason for Boards to up their decision making process game. Better processes produce better outcomes.
Nobel prize winner and doyen of the behavioural science underpinning human decision making, Daniel Kahneman recently published*** on why disciplined decision making processes are required. The fruits of his over 30 years of research give Kahneman deep insights into the cognitive and behavioural biases that afflict most human decision making. In the context of high stakes, high impact strategic Board-level decision making, these biases are frequently exacerbated by the individual egos, personalities and group dynamics involved. To counter these conditions Kahneman and his co-authors set out a compelling case for why decision making processes are needed.
In case after case of major corporate failure, the root cause can be found to be poor Board decision making. Yet when Boards get it wrong it is not just the individual directors who may have cause for regret. The consequences and costs fall far more widely than on the directors involved. Indeed, they are most frequently cushioned from financial impacts; employees, suppliers, customers and society at large are not. It can be argued that Boards and directors have a clear moral responsibility to improve their critical decision making processes.
* Business Judgement and the Courts; Centre for Business Law and Practice, University of Leeds; University of Liverpool Management School
** Marchand v. Barnhill, No. 533, 2018 (Del. June 19, 2019)
*** A Structured Approach to Strategic Decisions; Daniel Kahneman, Dan Lovallo, Olivier Sibony; MIT Sloan Management Review, March 04, 2019
ASIC's Warning on Director Oversight of Non-Financial Risks
09/Oct/19 10:34
Last week the Corporate Governance Task Force of the Australian Securities & Investments Commission (ASIC) published the first report of its Corporate Governance Task Force, on Director and Officer Oversight of Non-Financial Risks.
It's conclusions were both depressing and a clear warning to board directors around the world.
"Many directors identified challenges with overseeing non‑financial risks in large, complex organisations. Nevertheless, there was no strong, corresponding trend of directors actively seeking out adequate data or reporting that measured or informed them of their overall exposure to non‑financial risks. Fractured or informal flow of information up to the board and around the board table meant that some boards did not always have the right information to make fully informed decisions. Where information did make its way to the board, there was little evidence in the minutes of some organisations of substantive active engagement by directors…"
"We also observed that companies often had frameworks and structures in place to support board oversight of non‑financial risk; however, in practice, deficiencies arose in compliance with, or execution of, these frameworks. For example, boards approved risk appetites that were intended to articulate the level of risk acceptable for company operations, but management operated outside this appetite for years at a time with the board’s tacit acceptance…"
"We reviewed information flows from management to the board and from board committees to full boards. Our review found that material information about non‑financial risk was often buried in dense, voluminous board packs – boards did not own or control the information flows from management to the board to ensure material information was brought to their attention. Also, management reporting often did not identify a clear hierarchy or prioritisation for non‑financial risks."
You can download the full report and its associated materials here.
It's conclusions were both depressing and a clear warning to board directors around the world.
"Many directors identified challenges with overseeing non‑financial risks in large, complex organisations. Nevertheless, there was no strong, corresponding trend of directors actively seeking out adequate data or reporting that measured or informed them of their overall exposure to non‑financial risks. Fractured or informal flow of information up to the board and around the board table meant that some boards did not always have the right information to make fully informed decisions. Where information did make its way to the board, there was little evidence in the minutes of some organisations of substantive active engagement by directors…"
"We also observed that companies often had frameworks and structures in place to support board oversight of non‑financial risk; however, in practice, deficiencies arose in compliance with, or execution of, these frameworks. For example, boards approved risk appetites that were intended to articulate the level of risk acceptable for company operations, but management operated outside this appetite for years at a time with the board’s tacit acceptance…"
"We reviewed information flows from management to the board and from board committees to full boards. Our review found that material information about non‑financial risk was often buried in dense, voluminous board packs – boards did not own or control the information flows from management to the board to ensure material information was brought to their attention. Also, management reporting often did not identify a clear hierarchy or prioritisation for non‑financial risks."
You can download the full report and its associated materials here.