Review of “Forecasting in Social Settings: The State of the Art” by Makridakis et al

In our course on Strategic Risk Management and Governance, we note the very substantial challenge of forecasting the future behavior of complex adaptive systems made up of human beings and their organizations. There are many reasons for this, including:

  • Agents pursue multiple goals, with different incentives and priorities, and may change their goals and priorities over time as the system evolves;

  • When deciding on actions to achieve their goals, agents differ in terms of the range of experiences they can draw on, and their cognitive ability to reason multiple time steps ahead about the likely consequences of their actions;

  • Agents differ in their perceptions of the environment, and their beliefs about the future;

  • Agents differ in the structure of their social networks, which also evolve over time (more technically, the data generating process in complex adaptive systems is non-stationary, which reduced the usefulness of historical results as a guide to future outcomes);

  • Agents decide on their actions based not only on rational calculation, but also on their emotional reactions to competing narratives as well as the potential social impacts of their decisions;

  • Agents differ in their desire to conform to the beliefs and copy the actions of other members of their group, with the latter typically increasing with the level of perceived uncertainty;

  • Social feedback loops can produce emergent non-linear collective phenomena like herding, fads, booms and busts. These extreme events have been extremely hard to consistently forecast.

Taken together, these factors usually cause the accuracy of forecasts of complex adaptive system behavior to exponentially decline as the time horizon lengthens.

Given this background, we read the new paper by Makridakis and his colleagues with great interest.

At the outset, the authors clearly state that, “although forecasting in the physical sciences can attain amazing levels of accuracy, such is not the case in social contexts, where practically all predictions are uncertain, and a good number can be unambiguously wrong.”

There are a number of reasons for this. “First, there is usually a limited theoretical basis for presenting a causal or underlying mechanism” for the target variable being forecasted. “Thus we rely on statistical approximations that roughly describe what we observe, but may not represent a causal [process].” Second, “despite the deluge of data that is available today, much of this information does not concern what we want to forecast directly…Third, what we are trying to forecast is often affected by the forecasts themselves…Such feedback does not occur in weather forecasts…For these reasons, social science forecasts are unlikely to ever be as accurate as forecasts in the physical sciences, and the potential for improvements in accuracy is somewhat limited.”

The authors also note that when it comes to forecasting social systems, “unless uncertainty is expressed clearly and unambiguously, forecasting is not far removed from fortune-telling. However, uncertainty about judgmental forecasts of social system behavior is likely to be “underestimated greatly for two reasons.”

“First, our attitude to extrapolating in a linear fashion from the present to the future, and second, our fear of the unknown and our psychological need to reduce the anxiety associated with such a fear by believing we can control the future by predicting it accurately.”

Use of statistical instead of judgmental forecasting models improves the treatment of uncertainty, but this approach is far from perfect. The authors claim that, “there are at least three reasons for standard statistical models’ underestimations of the uncertainty:”

  1. “Probably the biggest factor is that model uncertainty is not taken into account. The prediction intervals are produced under the assumption that the model is ‘‘correct’’, which clearly is never the case.” The authors note that combining forecasts made using different models reduces this uncertainty.

  1. “Even if the model is specified correctly, the parameters must be estimated, and also the parameter uncertainty is rarely accounted for in time series forecasting models.” However, techniques like Monte Carlo simulation allow parameter uncertainty to be made explicit.

  1. “Most prediction intervals are produced under the assumption of Gaussian [normally distributed] errors. When this assumption is not correct, the prediction interval coverage will usually be underestimated, especially when the errors have a fat-tailed distribution [as is often the case in complex adaptive systems, which tend to produce outcomes that follow a Pareto/power law rather than a normal/bell curve distribution].”

The paper also includes sections on different types of uncertainty, the challenges of incorporating causality into forecasting models, and the difficulty of predicting one off and extreme events.

In sum, the authors have produced an excellent (and extensively referenced) overview of the current state of the art of forecasting in social settings.

The Critical Importance of Process in Board Decision Making

When we were researching the relationship between Board Chairs and CEOs we asked our interviewees about the processes they used in the boardroom. Mostly this elicited a response along the lines of “We have an agenda and we take minutes”. It seems most Boards of directors confuse process with procedure. Procedure may be involved but process is a much broader concept; it is a structured method or sequence of steps or activities, but not used for the purpose of administrative consistency or compliance. Process is used for quality of outcome.

There is perhaps a tendency for directors to view the discipline of process as a constraint on the free reign of experienced judgement. If so, this would be a mistake. A very wide body of experience and research has demonstrated that Board decision making is no less subject to the pitfalls of cognitive and behavioural limitations and biases than any other human activity. In certain important respects, in specific cases, Board decision making has been shown to be worse than in other, similarly important contexts. This is especially true of the performance of Boards as regards Strategic Risk.

There are two compelling reasons for Boards to take another look at their decision making processes – the first is the consequences for directors when their processes are absent or poor and the second is the real potential for improved Board effectiveness.

Traditionally, Board decision making has been defended in practice from external legal scrutiny by a convention, sometimes reinforced at common (i.e. judge made) law, that “business judgement” was not judiciable. This assumption has been long established. Yet even if it was ever wholly true, which is debatable, clear political and regulatory trends are moving against it.

In the UK, statute law has been enacted to both clarify and tighten directors’ accountabilities. Consequently, as recent research* (led by Professor Joan Loughrey at the University of Leeds) has highlighted, there has been dramatic growth in directors’ decisions being challenged in the courts. In less than a decade such cases grew by a factor of 10. Moreover, when directors are taken to court the probability is that they will lose. 63% of cases reviewed by Prof. Loughrey’s team were found against the directors, i.e. they were found liable for the consequences of their decisions. The proportion of cases being found against directors is also increasing. “Business Judgement”, at least in the UK, is no longer (if it ever really was) a blanket defence.

Even in the US, the legal environment is moving in the direction of greater director liability, especially in the arena of risk governance. The Supreme Court of Delaware, long a bell-weather for the legal approach to corporate governance in the US due to the large number of corporations domiciled in the state, has tended to err in defence of the “business judgement” principle. Historically the so-called Caremark doctrine held that directors would not be held liable for a failure in risk oversight unless there was “… an utter failure to ensure … [ ] … a reasonable system exists.” Now that seems to be changing too. In a recent judgement** the court found that “… directors must make a good faith effort to implement a [risk] oversight system and monitor it themselves.” This new ruling places the onus of accountability squarely on the Board to demonstrate an effective process is in place.

What does this mean for Board decision making? For a start it means much more than simply recording decisions in minutes. Prof. Loughrey’s team found that when cases were found in favour of directors it was most frequently because the defendants could show that they had followed a clear decision making process, supported by relevant and timely information and advice.

Thus, there is a clear case for Boards to pay greater attention to the appropriateness and rigour of their decision making processes, especially when critical strategic issues and risks are involved. Yet even if self-protection is not sufficient justification there is another good reason for Boards to up their decision making process game. Better processes produce better outcomes.

Nobel prize winner and doyen of the behavioural science underpinning human decision making, Daniel Kahneman recently published*** on why disciplined decision making processes are required. The fruits of his over 30 years of research give Kahneman deep insights into the cognitive and behavioural biases that afflict most human decision making. In the context of high stakes, high impact strategic Board-level decision making, these biases are frequently exacerbated by the individual egos, personalities and group dynamics involved. To counter these conditions Kahneman and his co-authors set out a compelling case for why decision making processes are needed.

In case after case of major corporate failure, the root cause can be found to be poor Board decision making. Yet when Boards get it wrong it is not just the individual directors who may have cause for regret. The consequences and costs fall far more widely than on the directors involved. Indeed, they are most frequently cushioned from financial impacts; employees, suppliers, customers and society at large are not. It can be argued that Boards and directors have a clear moral responsibility to improve their critical decision making processes.

* Business Judgement and the Courts; Centre for Business Law and Practice, University of Leeds; University of Liverpool Management School
** Marchand v. Barnhill, No. 533, 2018 (Del. June 19, 2019)
*** A Structured Approach to Strategic Decisions; Daniel Kahneman, Dan Lovallo, Olivier Sibony; MIT Sloan Management Review, March 04, 2019

ASIC's Warning on Director Oversight of Non-Financial Risks

Last week the Corporate Governance Task Force of the Australian Securities & Investments Commission (ASIC) published the first report of its Corporate Governance Task Force, on Director and Officer Oversight of Non-Financial Risks.

It's conclusions were both depressing and a clear warning to board directors around the world.

"Many directors identified challenges with overseeing non‑financial risks in large, complex organisations. Nevertheless, there was no strong, corresponding trend of directors actively seeking out adequate data or reporting that measured or informed them of their overall exposure to non‑financial risks. Fractured or informal flow of information up to the board and around the board table meant that some boards did not always have the right information to make fully informed decisions. Where information did make its way to the board, there was little evidence in the minutes of some organisations of substantive active engagement by directors…"

"We also observed that companies often had frameworks and structures in place to support board oversight of non‑financial risk; however, in practice, deficiencies arose in compliance with, or execution of, these frameworks. For example, boards approved risk appetites that were intended to articulate the level of risk acceptable for company operations, but management operated outside this appetite for years at a time with the board’s tacit acceptance…"

"We reviewed information flows from management to the board and from board committees to full boards. Our review found that material information about non‑financial risk was often buried in dense, voluminous board packs – boards did not own or control the information flows from management to the board to ensure material information was brought to their attention. Also, management reporting often did not identify a clear hierarchy or prioritisation for non‑financial risks."

You can download the full report and its associated materials here.

Risk Through the Lens of Strategy, Management, and Leadership

In the process of evolution that has driven human progress across the ages, there three core performance metrics that apply to all organisms and organizations:

  • Effectiveness: Achieving the goals that are required to ensure continued survival.

  • Efficiency: Achieving those goals while using as few resources as possible.

  • Adaptability: The extent to which effectiveness and efficiency are maintained in the face of changing external conditions.

There are inescapable trade-offs between these three metrics. For example, maximizing efficiency often eliminates the slack resources that are critical to adaptability.

The three major “direction setting” functions in modern organizations generally align with these three evolutionary metrics.

Strategy’s core challenge is effectiveness. Management’s is efficiency. And leadership’s is adaptability.

In this post, I’ll take a closer look at risk through each of these three lenses.

We define strategy as “a causal theory of success that exploits one or more decisive asymmetries to achieve an organization's most important goals with limited resources, in the face of uncertainty, constraints, and opposition.” If they are achieved, these goals will, at minimum, enable the organization to survive as independent entity, and ideally generate substantial value for its stakeholders.

In this context, critical risks include failing to accurately anticipate and assess emerging threats and opportunities in the external environment, setting the wrong goals, and pursuing a strategy for achieving them that doesn’t work.

The core challenge of management is translating the organization’s strategy into detailed objectives, and devising plans, processes, systems, budgets and organization structures that enable objectives to be met with scarce resources. Management also involves monitoring implementation of these plans and the evaluation of their results.

Risk in this context is well-captured by the concept of enterprise risk management, which seeks to identify, quantify, and integrate an organization’s exposure to a range of hazard, operational, and financial risks.

The fundamental challenges of leadership include aligning employees and other stakeholders around an organization’s purpose and strategic agenda; recruiting talented employees and developing them into a high-performance team; and facilitating the organization’s continuing adaptation to its changing competitive environment.

Risk in this context is, in my experience, often the most critical and also the most overlooked because of its ambiguous nature. The best strategy and most competent management in the world are all-too-easily undermined by poor leadership and a toxic culture. More importantly, metastasizing leadership and culture risk are root causes that usually precede the appearance of management and strategic risk.

For these reasons, it is critical for board directors to constantly look for signs of emerging leadership and cultural risks. While there are multiple warning indicators to monitor, over the years I’ve found two to be particularly useful. The first is the extent of blaming that you observe in an organization. The second is how difficult decisions and high consequence uncertainties are discussed. Is there a healthy exchange of different views? How is conflict managed, or is it absent?

More broadly, sustained leadership emerges from the interaction of integrity, competence, and empathy. Leadership risk is present whenever one or more of these is missing.

In sum, risk looks very different when viewed through the lens of strategy, management, or leadership. All of them are critical, and successfully addressing their different risks requires multiple approaches and skills.

Britten Coyne Partners, our focus is on strategic and leadership risk. We offer both education and consulting services to help our clients meet these challenges.


The Critical Importance of Anticipatory Intelligence in Our Complex, Uncertain World

The deceptive economic and geopolitical calm of the past decade has been an aberration, brought about by unprecedented global monetary stimulus to hold at bay the deflationary forces that have been building in the global economy. Thanks to central bankers’ efforts, volatility has remained low, and organizations have not had to worry too much about disruptive risks beyond those posed by rapid technological change. That is about to change: Brexit, the election of Donald Trump, the emergence of a new US-China Cold War, and nearly two trillion dollars of sovereign bonds bearing negative interest rates are early indications that we are entering a period of much higher uncertainty.

With this change will come much greater organizational focus on developing the processes, methods, tools, and skills needed to survive and thrive in a much more dangerous environment. Josh Kerbel, a faculty member at the United States’ National Intelligence University, recently published an article that we hope will have a substantial impact on these efforts, and closely reflects our views at Britten Coyne Partners.

In “Coming to Terms with Anticipatory Intelligence”, Kerbel notes that it is “a relatively new type of intelligence that is distinct from the “strategic intelligence” that the intelligence community has traditionally focused on. It was born from recognition that the spiking global complexity (interconnectivity and interdependence, both virtual and physical) that characterizes the post–Cold War security environment, with its proclivity to generate emergent (non-additive or nonlinear) phenomena, is essentially new. And as such, it demands new approaches.”

“More precisely, this new strategic environment means that it is no longer enough for the intelligence community to just do traditional strategic intelligence: locking onto, drilling down on, and — less frequently — forecasting the future of issues once they’ve emerged. While still important, such an approach will increasingly be too late. Rather, the intelligence community should also learn to practice foresight (which is not the same as forecasting) and imagine or envision possibilities before they emerge. In other words, it should learn to anticipate.”

Kerbel echoes longstanding concerns among some members of the intelligence community. For example, a 1983 CIA analysis of failed intelligence estimates noted that, “each involved historical discontinuity, and, in the early stages...unlikely outcomes. The basic problem was...situations in which trend continuity and precedent were of marginal, if not counterproductive value."

This distinction was also brought home to me during the four years I spend on the Good Judgement Project, which demonstrated that forecasting skills could be significantly improved through the use of a mix of techniques. But hiding in the background was an equally important question: What was the source of the questions whose outcome we were forecasting? One of my key takeaways was that anticipatory thinking – posing the right questions – was just as important to successful policy and action as accurately forecasting their outcome.

Kerbel notes that, “as clear and compelling as the case for anticipatory intelligence is, it remains poorly understood… Since the 1990s, increasing complexity has been an issue that many in the intelligence community have impulsively dismissed or discounted. Their refrain echoes: “But the world has always been complex.” That’s true. However, what they fail to understand is that the closed and discrete character of the Soviet Union and the bipolar nature of the Cold War — the intelligence community’s formative experience — eclipsed much of the world’s complexity and effectively rendered America’s strategic challenge merely complicated (no, they’re not the same). Consequently, the intelligence community’s prevailing habits, processes, mindsets, etc. — as exemplified in the traditional practice of strategic intelligence — are simply incompatible with the challenges posed by the exponentially more complex post-Cold War strategic environment.”

Kerbel’s view is that “Fundamentally, anticipatory intelligence is about the anticipation of emergence… Truly emergent issues are fundamentally new — nonlinear — behaviors that result unpredictably but not unforeseeably from micro-behaviors in highly complex (interconnected and interdependent) systems, such as the post–Cold War strategic environment. Although emergence can seemingly happen quite quickly (hence the need to anticipate), the conditions enabling it are often building for some time — just waiting for the “spark.” It is these conditions and what they are potentially “ripe” for — not the spark — that anticipatory intelligence should seek to understand… Foresight involves imagining how a broad set of possible conditions (trends, actors, developments, behaviors, etc.) might interact and generate emergent outcomes.”

This begs the question of which foresight methods and tools are most effective. We go into great detail about this in our Strategic Risk Governance and Management course. In this blog post we’ll highlight four key insights.

Traditional scenario methodologies often disappoint

  • As a general rule, when reasoning from the present to the future, we naturally (to maintain our sense psychological safety) minimize the extent of change that could occur.

  • In complex systems, it is almost always impossible to reduce the forces that could produce non-linear change to just two critical uncertainties, as is done in the familiar “2 x 2” scenario method. And in some cases, the uncertainties that most worry an organization’s senior leaders are either out of bounds for the scenario construction team, or the range of their possible outcomes is deliberately constrained.

  • I first studied the scenario methodology under Shell’s Pierre Wack back in 1983. In its early applications, this approach was often able to fulfill its goal of changing senior leaders’ perceptions. Over the years, however, I have seen what I call “scenario archetypes” become more common, which has weakened their ability to surprise leaders and change their perceptions. These archetypes result from one critical uncertainty being technological in nature, and the other being one whose negative outcome would be very bad indeed. This gives rise to three archetypes: (1) Business pretty much as usual, with current trends linearly extrapolated (this is usually the scenario that explicitly or implicitly underlies the organization’s strategy); (2) The World Goes To Hell (slow technology change and the negative outcome for the other uncertainty); and (3) Technology Saves the Day (fast technology change overcomes the negative outcome of the other uncertainty). This leaves what is usually the least well defined but potentially most important scenario, where technology rapidly develops, but the other uncertainty does not have the negative outcome. Too many organizations fail to fully explore the implications of this scenario, usually because they are more realistically threatening to the current strategy.
Historical analogies are limited by our knowledge of history

  • Whether the subject is political economic, technological, business, or military history, most of us have studied too little of it to have a rich based of historical analogies from which we can draw while trying to anticipate the future.

  • Consider some of the challenge we face at the present, including the transition from an industrial to an information and knowledge-based economy; the rapid improvement in potential “general purpose” technologies like automation and artificial intelligence; and the potential transition of the global political economy from a period of growing disorder and conflict to period of more ordered conflict due to a new Cold War between the US and China. In all these cases, the most relevant historical analogies may lie further in the past than many people realize.
Prospective hindsight – reasoning from the future to the present – is surprisingly effective

  • Research has shown that when we are given future event, told that it is true, and asked to explain how it happened, our causal reasoning is much more detailed than if we are simply asked, in the present, how this future event might happen.

  • However, that still leaves the “creative” or “imaginative” challenge of conceiving of these potential future events. We have found that starting with broad future outcomes – e.g., our company has failed; China has successfully forced the US from East Asia – generates a richer set of alternative narratives than a narrower focus on specific future events.
Explicitly focusing on system interactions helps identify emergent effects and early warning indicators

  • Quantitatively, agent-based models, which enable complex interactions between different types of agents, can produce surprising emergent effects, and, critically, help you to understand why they occur (which can aid in either their prediction or in designing interventions to promote or avoid them).

  • Qualitatively, we have found it very useful to create traditional scenarios in narrower policy areas (e.g., technology, the economy, national security, etc.) and then explicitly trace and assess overall system dynamics and how different scenario outcomes could interact across time and across policy areas (e.g., technology change often precedes economic and national security change) to produce varying emergent effects.

Kerbel concludes by noting that, “Exponentially increasing global complexity is the defining characteristic of the age.” Because of this, effective anticipatory intelligence capabilities are more important than ever before to organizations’ future survival and success – and more challenging to develop.