Failure Modes of Decision Making

The man pushing the throttles forward is the best pilot the airline has. Their magazine has his face in it, promising you are safe in his hands. It's Sunday, at the end of March and on the runway at Tenerife the fog sits so thick he cannot see the far end. The pilot believes he has been cleared for takeoff. Beside him his co-pilot says, quietly, that he is not sure they actually have clearance. But the aircraft is already moving. Eight seconds later 583 people are dead, in two planes that never saw each other until the very last moment, making it the deadliest accident in aviation history (Subsecretaría de Aviación Civil, 1978). He was one of the best pilots and knew everything about flying, but one wrong belief, the failure to stop and verify killed these 583 souls.

We do this everywhere, far outside any cockpit. We act on what we assume to be true and almost never stop to ask whether it actually is.

The one place humans built a defense against this is science. Karl Popper argued that knowledge does not grow by confirming what we already believe but by attacking it, by hunting for the error that proves us wrong (Popper, 1959) - known as the Scientific Method. But at the same time the history of science is mostly a record of confident error. The earth sat at the centre of everything for over a thousand years. Heat was once thought to be a fluid you could pour from a warm body into a cold one. Both were settled knowledge in their day and both were wrong. Laudan called this the pessimistic induction, the observation that almost every theory humans once trusted has since been overturned, which leaves little reason to believe our current ones are the exception (Laudan, 1981). ​And this is not only old history. Going through the modern research literature, Ioannidis found that most published findings do not survive a second look (Ioannidis, 2005).

Decision Failure Modes

Error detection is the most important thing because most knowledge we get wrong. But detection needs something to detect against. It is really hard to catch an error you have no frame or name for. A mechanic can name what he hears inside an engine, a worn bearing, a cylinder firing a beat too late. The same sound to somebody else is just noise. It's similar with decisions, as it is way easier to tell why one is failing if you know the shapes failure can take.

The following will focus on painting a landscape of failure modes against which decisions can be examined. Or to be more precise different backdrops for different phases of the play, that allow you to view your own decisions and decision processes in front of them.

Cognitive Failure

The pilot's wrong belief at Tenerife started where every decision does, inside a human mind working with too little and under too much. Cognitive failures are inherent to human decision-making. Bounded rationality, first formalized by Simon (1955), provides a useful framework for understanding how our cognitive capacities are insufficient to fully engage with and process the complexity of the world. Rather than optimizing across all possible alternatives, individuals operate under constraints of limited knowledge, cognitive capacity as well as time, relying on simplifying strategies that can systematically distort judgment.

Bounded Rationality

We never decide among all the options. Knowledge, attention and time shrink the field first.

It is also important to note that beyond cognitive limitations, reasoning is often directionally motivated. Motivated reasoning, as developed in social psychology, most prominently by Ziva Kunda (1990), suggests that individuals do not merely seek accuracy but coherence with their identity, prior commitments as well as desired conclusions. As a result, ambiguous evidence is interpreted in ways that preserve self-concept, status or strategic interests.

In organizational contexts, this dynamic can become amplified. New information may threaten hierarchy, reputation, past investments or previously endorsed strategies. Consequently, evidence is unconsciously filtered, reinterpreted or downplayed in order to sustain existing plans.

Epistemic Failure

In the 1950s-60s, thalidomide was prescribed for morning sickness based on animal tests showing safety. However, incomplete knowledge about its effects on human fetal development led to thousands of birth defects, today known as the Thalidomide Tragedy (McBride, 1961). The failure was in unknown unknowns, as the involved parties didn’t know enough, but also didn’t know that they didn’t know enough.

This epistemic failure, when the actors don’t recognize the limits of their own knowledge is addressed in Munger's Circle of competence and repeatedly emphasized by him, that the most dangerous state is not stupidity but misplaced confidence (Munger, 2005). As simple as this distinction between what you know and what you don’t know is in theory, it is that much harder in practice, as it requires a great deal of awareness and willingness, some might even say courage, to look at your own limitations.

Circle of Competence

Everybody's got a different circle of competence. The important thing is not how big the circle is. The important thing is staying inside the circle.

Ontological Failure

For centuries our understanding of our solar system rested on Claudius Ptolemy, who provided the dominant framework for astronomy in Europe and parts of the Islamic world. His model was mathematically sophisticated and capable of generating workable predictions for planetary motion, as it supported navigation as well as the construction of calendars. However, the structural premise was false. The Earth is not the center of the planetary system, but it took a lot of time until the heliocentric model proposed by Nicolaus Copernicus and later empirically strengthened by Galileo Galilei was fully accepted.

Geocentrism vs heliocentrism

Ptolemy ran the calendars for centuries with the earth in the wrong place. A model can be useful and still be wrong about what sits at the centre.

Ontological failure arises when the underlying model of reality is structurally incorrect. Coming from ancient Greek ὄν (on) = being and -λογία (-logia) = study of, ontological can be seen as the nature, structure or categories of being. Decisions built upon such a model may appear internally coherent and rational, yet they remain misaligned with the actual structure of the system. Ontological errors therefore operate at the level of assumptions about what exists, how it relates and which mechanisms govern it.

Agency Failure

Agency failure is what happens when a decision-maker acts on a wrong map of his own influence. He treats some variables as his to set, when they were never under his hand.

Influence mistaken for control

The wide boundary is what the decision-maker believes they control. The small disc is what they actually do. Chance governs the rest.

Ellen Langer (1975) called this the illusion of control, the tendency to overestimate how much a chance-governed outcome answers to your own actions. The pull is strong because the feedback flatters it. Success gets read as skill and the part played by luck quietly drops out of the account. Survivorship bias does the rest, since the winners are visible and tell a clean story while the ones who did everything the same and lost have already left the sample.

Agency failure also highlights a core tension in decision-making. Elevating your sense of agency can help you take more action and be beneficial, yet robustness requires accurate recognition of its limits. Overrating your control is rational for the population and ruinous for the individual. An economy needs founders who think they'll beat the odds, because the rare winners pay for the many who don't. But you don't get the average outcomeNon-ErgodicityThe Room with the Revolver and other Non-Ergodic SystemsRead the essay↗. So better have an accurate recognition of your limits while cultivating a high agency.

Feasibility Failure

Feasibility failure occurs when a decision is internally coherent yet externally impossible. The model may be logically consistent, yet the decision collapses because it neglects binding constraints imposed by physics, economics, biology, social dynamics or other structures. The problem lies in the underestimation of what can be done, as real constraints are ignored.

Penrose Tribar

Every corner is a sound on its own, yet the three cannot close into a solid that exists.

Feasibility failure often masquerades as courage or boldness and ignoring constraints can be reframed as “thinking big” or “breaking limits”, yet often at a steep cost. Entrepreneurs may assume frictionless scaling or engineers may extrapolate laboratory results into complex systems without accounting for emergent interactions. Such beliefs assume smooth transitions from decision to outcome, but reality, shaped by hard constraints, will act as a “reality check”. It demands humility toward immutable structures and attentiveness to second-order effects.

Prediction Failure

Prediction failure occurs when decision-makers believe they can reliably forecast the future. Although there are many reasons, historical extrapolation is a common source of error. There the past is used to derive “probable” futures, yet this beam represents only the realized history and not the full space of what could have happened.

The cone of potential futures

Historical extrapolation mostly lights up the probable core. High-impact but extremely rare events are almost never deduced from the past.

The distinction between probable, plausible and possible futures illustrates the problem. Predictive models typically concentrate on the probable zone, the outcomes that fit established distributions. However, complex systems contain a much wider domain of plausible and possible states. Rare, high-impact events may sit outside the probabilistic core and as Nassim Taleb (2007) argues, black swan events emerge from this neglected outer space of possibility, where historical frequency offers little guidance.

The 2008 financial crisis exemplifies this failure. Risk models built on historical housing data treated nationwide price collapses as highly improbable. By focusing on past regularities, banks underestimated interdependence and fat-tailed exposureDistributional ThinkingEvery success has siblings and often you only meet oneRead the essay↗.

Coming to a Landing

The co-pilot at Tenerife said it out loud, that he wasn't sure they were cleared. He identified the error and had the words for it. But he lacked the conviction to make sure that the fatal belief got a chance to be corrected.

Naming a failure mode does not stop you from walking into it. Every one in this essay can be seen coming and every one still gets made. What none of the names give you is the agency to hold on for a moment, challenge your decision and check before pushing the throttles forward.

You will get things wrong. That is not the bleak reading of this essay, just a reliable prediction. The edge is catching the error in time, seeing the failure a moment before it turns irreversible, holding one belief up to the light before you act on it. Error detection is the most important thing because most knowledge we get wrong.