Systemic Failure Archive and Research

Iatrogenic Failure

5 min read

Iatrogenic failure is harm produced by the intervention meant to prevent or repair a problem, rather than by the original problem itself. The word is borrowed from medicine — iatrogenesis, illness caused by the treatment or the physician — and Ivan Illich’s Medical Nemesis extended it from the individual clinic to the institution, arguing that organised medicine had itself become a threat to health. The generalisation to systems is direct: any organisation that acts on a problem is administering a treatment, and the treatment can be worse than the disease.

This is the mirror image of a latent condition. A latent condition is harm waiting dormant in a system’s fabric; an iatrogenic failure is harm actively introduced by a forceful, well-intentioned act. It is also a sharper claim than the familiar law of unintended consequences (Merton): not merely that an action has effects nobody foresaw, but that those effects run counter to the action’s own purpose — the intervention defeats itself.

Why complex systems invite it

An intervention is an action taken by an organisation on the basis of its model of the problem (see organisations as cognitive systems). In a simple, linear system the model can be good enough and the intervention does what it says on the tin. Complex systems frustrate this in characteristic ways: they contain feedback loops that absorb or reverse a push; they respond with delays, so the harm surfaces later and elsewhere than the apparent cure; and they are full of adaptive agents who change their behaviour in response to being intervened upon. Jay Forrester called the result the counterintuitive behaviour of social systems, and system dynamics gives it a name — policy resistance: the system pushes back against the lever precisely because the lever was chosen from a model that left out the loops now doing the pushing.

Force and coupling are what make it dangerous. A tentative, reversible probe can be wrong harmlessly; a forceful, irreversible intervention tightly coupled to its consequences (see tight coupling) turns a modelling error into a large, fast, unrecoverable harm.

Mechanisms of reversal

Iatrogenic failure recurs through a small set of mechanisms:

  • Acting on the wrong model. The intervention treats a simplified representation, and the real system responds differently — as when dispersants deployed against an oil slick prove more toxic to the shore than the oil they were meant to clear.
  • Adaptive response. The system’s agents adjust to the intervention and offset it: risk compensation, where a safety measure is met with riskier behaviour; or the perverse incentive of the “cobra effect,” where a bounty on a pest calls forth its breeding.
  • Scale and irreversibility. A response sound in principle is applied at a scale that becomes its own catastrophe — the mass cull, or the high-modernist scheme that razes the tacit order it depended on (Scott).
  • Added complexity. The fix introduces new coupling and new failure paths, so the defence itself becomes a hazard — the safety system that fails, the redundancy that adds opacity.
  • Delay and displacement. The harm appears later and elsewhere, so the intervention is judged a success in the window that matters and the reckoning falls to someone else.

The autonomous-systems turn

Iatrogenic risk scales with the speed, force and reach of the intervening agent — which is exactly what automation increases. An organisation that delegates intervention to an automated or algorithmic system hands the act of treatment to something that acts fast, at scale, and on a necessarily simplified model, overseen by a human prone to deferring to its output. The conditions that make a human intervention iatrogenic — a thin model, force, tight coupling and irreversibility — are precisely the conditions an autonomous decision system tends to maximise. The governance question is not only whether the model is accurate, but whether the intervention it drives can be kept small, reversible and observable before it is allowed to become large.

Not an argument against intervening

The concept is easily misused as a counsel of inaction, and it is not one. Refusing to act is itself an intervention, with its own consequences, and many problems demand a forceful response. The lesson is narrower, and it is about method. Where a system is complex, adaptive and tightly coupled, the appropriate humility is to prefer the reversible to the irreversible, to remove a known harm rather than add a speculative fix (Taleb’s via negativa), to probe at small scale before acting at large, and to treat one’s own confidence in the model as itself a risk factor. In these systems, the reflex to do something decisive is a hazard in its own right.

In this archive

  • Torrey Canyon oil spill — the cleanest exemplar: the toxic dispersants and the bombing of the wreck did much of the ecological damage the response was meant to prevent.
  • Foot-and-mouth outbreak — a control measure whose scale and manner (mass slaughter, movement bans, pyres) became a second crisis alongside the disease.

Key sources

  • Illich, I. (1976). Medical Nemesis: The Limits to Medicine. Marion Boyars.
  • Merton, R. K. (1936). The unanticipated consequences of purposive social action. American Sociological Review, 1(6), 894–904.
  • Forrester, J. W. (1971). Counterintuitive behavior of social systems. Technology Review, 73(3), 52–68.
  • Sieber, S. D. (1981). Fatal Remedies: The Ironies of Social Intervention. Plenum Press.
  • Tenner, E. (1996). Why Things Bite Back: Technology and the Revenge of Unintended Consequences. Knopf.
  • Scott, J. C. (1998). Seeing Like a State. Yale University Press.
  • Taleb, N. N. (2012). Antifragile: Things That Gain from Disorder. Random House.

Graph View

Backlinks