Desktop versionMobile version
OpenEdition Books

Proceedings of the fourth Resilience Engineering Symposium

 | 
Erik Hollnagel
, 
Éric Rigaud
, 
Denis Besnard

Fundamental On Situational Surprise: A Case Study With Implications For Resilience

Robert L. Wears and L. Kendall Webb

Abstract

Fundamental surprise is a challenge for resilience, since by definition it cannot be anticipated, and monitoring is limited by the lack of knowledge about what to target. It does, however, present opportunities for both responding and for learning. We describe an incident in which we use the co-occurrence of situational and fundamental surprise to reveal patterns about how adaptive capacity was used to meet challenges, and what barriers to learning were present. We note that temporal and cross-level factors played important roles in affecting the balance between situational and fundamental learning. Because the situational story of component failure developed first, it was difficult for the fundamental story of unknown, hidden hazards to supplant it. In addition, the situational story was easily grasped by all members of the organization, but the implications of the fundamental story were difficult for non-technical members, including senior leadership, to grasp. The responses at both the situational and fundamental level contain information about both specific vulnerabilities and general adaptive capacities in the organization.

Author's note

Note portant sur l’auteur1

Note portant sur l’auteur2

Full text

1 Introduction

“Things that never happened before happen all the time.”

1Analyses of critical incidents often distinguish between situational and fundamental surprise. Events characteristic of situational surprise might be temporally unexpected, but their occurrence and evolution are generally explicable and more importantly, compatible with the ideas generally held by actors in the system about how things work or fail, and the hazards that they face. Fundamental surprise, on the other hand, is astonishing, often inexplicable, and forces the abandonment of the broadly held notions of both how things work, and the nature of hazards that are confronted.

2If we think of a system’s resilience as its intrinsic ability “to adjust its functioning prior to, during, or following changes or disturbances, so that it can sustain required operations under both expected and unexpected conditions”, then it is clear that surprise creates unexpected demands that call for a resilient response.

3Lanir has identified 4 characteristics that distinguish situational from fundamental surprise. Fundamental surprise refutes basic beliefs about ‘how things work’, while situational surprise is compatible with previous beliefs. Second, in fundamental surprise one cannot define in advance the issues for which one must be alert. Third, situational and fundamental surprise differ in the impact of information about the future. Situational surprise can be averted by such foresight, while advance information in fundamental surprise actually causes the surprise. And finally, learning from situational surprise seems easy, but learning from fundamental surprise is difficult.

4Resilience is also characterized by 4 cardinal activities: monitoring, anticipating, responding, and learning. While effective management of situational surprise would typically involve all 4 of these activities, fundamental surprise clearly is a profound challenge for resilience, because one cannot monitor or anticipate items or events that are inconceivable before the fact. This leaves only responding and learning as the immediately available resilience activities in fundamental surprise, and explains in part why fundamental surprise is such a challenge to organisational performance. However, fundamental surprise does afford opportunities for deep learning, in particular the development of ‘requisite imagination’, an ability to picture the sorts of unexampled events that might befall

5We present a case study of the catastrophic failure of an information technology (IT) system in a healthcare delivery organisation, and of the organisation’s response to it from the point of view of resilience. The failure itself involved a combination of both situational and fundamental surprise. As might be expected, the immediate response involved both adaptations of exploitation (ie, consuming buffers and margin for manoeuvre to maintain essential operations) and adaptations of exploration (ie, novel and radical reorganisations of the way work gets done). Because fundamental surprise makes the disconnect between self-perception and reality undeniable, it affords the opportunity for a thorough-going reconstruction of views and assumptions about how things work. However, in this case the conflation of fundamental and situational surprise led to a classic fundamental surprise error – a re-interpretation of the problem in local and technical terms.

2 The Case

6In this section we describe the events and the adaptations to the interpretations made of them, based on notes, formal reviews, and interviews during and after the incident.

2.1 Events

7Shortly before midnight on a Monday evening, a large urban academic medical centre suffered a major information technology (IT) system crash which disabled virtually all IT functionality for the entire campus and its regional outpatient clinics. The outage persisted for 67 hours, and forced the cancelation of all elective procedures on Wednesday and Thursday, and diversion of ambulance traffic to other hospitals. (52 major procedures and numerous minor procedures; at least 70 incoming ambulance cases were diverted to other hospitals). There were 4 to 6 hour delays in both ordering and obtaining laboratory and radiology studies, which severely impacted clinical work. The total direct cost (not including lost revenue from cancelled cases or diverted patients) was estimated at close to $4 million. As far as is known, no patients were injured and no previously stored data were lost.

8The triggering event was a hardware failure in a network component. This interacted with modules not known to be present but left behind from an incompletely aborted (and ironically named) “high availability computing” project some years previous; this interaction prevented the system from restarting once the network component was replaced. The restart failure could not be corrected initially because of a second, independent hardware failure in an exception processer. Once this was identified and replaced, the system still could not be restarted because unbeknownst to the IT staff, the permissions controlling the start-up files and scripts had been changed during the same project, so that no one in IT was able to correct them and thus re-start the system.

2.2 Adaptations

9After a brief initial delay, the hospital was able to quickly reorganize in multiple ways to keep essential services operating for the duration. Adaptations included exploitation of existing resources or buffers; and exploration of novel, untried ways of working. These adaptations correspond roughly to the first-and second-order resilient responses described by a well-known materials science analogue.

10Adaptations of exploitation included deferring elective procedures and speeding discharges of appropriately improving inpatients. The former was limited in scope because the extent of the problem was not realized until Tuesday’s elective cases were well underway. The latter was stymied by the slow delivery of laboratory and imaging results; physicians were reluctant to discharge patients when results were still pending. This, of course, is one of the classic patterns of failure – falling behind the tempo of operations.

11Several adaptations of exploration were invoked. An incident command team was formed. Because the geographic area experiences frequent hurricanes, the incident command system was well-rehearsed and familiar, so it was adapted to manage a different type of threat.

12A similar novel use of available techniques evolved dynamically to compensate for the loss of medical record numbers (MRNs) to track patients, orders, and results while the system was down. The emergency department had been planning to implement a ‘quick registration’ method, where only basic patient information is obtained initially to permit earlier orders and treatment, and the registration process is completed at a later time. The IT failure prevented complete registration but was thought to have left the capability for quick registration. This method was very close to implementation, so it was pressed into service. However, its application in this setting uncovered a problem, in that different organisational units used the same variable to represent different information; this resulted in several patients in getting “lost” in the system. This failure led to an alternative, the use of the mass casualty incident (MCI) system.

13In many MCIs, the numbers of arriving patients would too rapidly exceed the ability to record their basic information and assign them identifying MRNs, so the organization maintained a separate system with reserved MCI-MRNs and pre-printed armbands. Although this system was envisioned for use in high demand situations, it was formally designed to accommodate any mismatch between demand and available resources. In this case, demand was normal to low, but resources were much lower, so the MCI system was used to identify and track patientsand marry them to formal MRNs after the incident had been resolved.

14The most novel adaptation of exploration included rescheduling financial staff (who now had nothing to do, since no bills could be produced or charges recorded) as runners to move orders, materials, and results around the organization that had previously been transmitted electronically.

2.3 Interpretations

15The case was viewed in multiple ways within the organisation, depending on the orientation to situational or fundamental surprise. It should be emphasized that there is not a ‘correct’ interpretation here – both views have both validity and utility, and both must be understood and held simultaneously for a full understanding of the case and its implications for organisational resilience.

16Situational Surprise. Because the triggering event was a hardware failure, and because the organisation had experienced a similar incident leading to total IT failure 13 years previously, the failure was initially interpreted as a situational surprise. It evinced no fundamental misperception of the world; it was not ‘the tip of the iceberg’ but rather a hazard about whose possibility there had always been some awareness.

17However, we should not downplay the importance of the organisation’s situational response, which was in many ways remarkably good. The organisation detected the fault and responded relatively quickly and effectively; the unfolding understanding of the situation and effectiveness of the response was monitored, and the organisation reconfigured to meet the threat. This reconfiguration involved a mixed control architecture where a central, incident command group set overall goals and made global level decisions (eg, cancelling elective procedures, reassigning financial staff) and managed communications among the various subunits of the organisation, while allowing functional units (eg, the emergency department, operating room, intensive care units, pharmacy, radiology, laboratory, and nursing) to employ a mixture of pre-planned and spontaneously developed adaptations to maintain performance.

18There was a specific attempt to capture situational learning from the incident. Each major unit conducted its own after action review to identify performance issues; the incident command group then assembled those and conducted a final, overall review to consolidate the lessons learned. This review obtained broad participation; it resulted in 104 unique items that, while locally oriented and technically specific, form the nidus of organisational memory and could inform the approach to similar future events, which are broadly anticipated in their consequences (ie, another widespread IT failure at some point seems assured) if not in their causes.

19One remarkable aspect of the response was the general absence of finger-pointing or accusatory behaviours, witch-hunts or sacrificial firings. An essay on how complex systems fail had been circulated among the senior leaders and the incident command group during the outage, with substantial agreement on how well it described the incident, its origins, and consequences.

20Fundamental Surprise. However, as a fuller understanding of the incident developed, situational gave way to fundamental surprise. The discovery of the permissions problem refuted taken-for-granted beliefs – that IT services understood and could maintain their own systems; and in particular, that restrictions to privileged (“root”) access could not be compromised except by sabotage. It raised the question of what other, previously unknown threats, installed by a parade of vendors and consultants over the years, lay lurking just beneath the surface waiting to be triggered into behaviours both unexpected and unexplainable.

21Lanir notes that “when fundamental surprises emerge through situational ones, the relation between the two is similar to that between peeled plaster and the exposed cracks in the wall. The plaster that fell enables us to see the cracks, although it does not explain their creation”. The IT unit recognized this clearly, and were astonished by the “hidden time bomb” whose presence was only fortuitously revealed by the line card failure. This triggered a deeper review of known previous changes, a new commitment to not permitting unmonitored and undocumented changes by vendors or other 3rd parties, and more stringent requirements for “as installed” documentation (including personal identification of involved parties). It led to a general awareness among IT leaders that their knowledge of their own system was incomplete and that they should therefore act in the “continuing expectation of future surprise”. This fundamental learning, however, did not spread throughout the organisation, but remained mostly encapsulated in IT.

3 Discussion

22Critical incidents are ambiguous: is stopping short of complete breakdown a story of success, or a harbinger of future failure ? Incidents embody a dialectic between resilient adaptation and brittle breakdown. In this case we see successful, resilient adaptation, but the real lesson is not in the success but rather in how adaptive capacity was used, and how it can be fostered and maintained. We also see limited fundamental learning, but the real lesson is not the failure of more broadly based learning but rather understanding what made that learning difficult.

3.1 Fundamental surprise as a challenge to resilience

23Fundamental surprise represents a major challenge to organisational resilience. Since by definition, fundamental surprise events are inconceivable before the fact, they cannot be anticipated; since it is unknown whence the come, there can be little guidance on what, exactly, to monitor to facilitate their detection.

3.2 Factors limiting fundamental learning

24There is a strong tendency to re-interpret fundamental surprise in situational terms.

25Several factors combined to limit fundamental learning in this case.

26Situational Surprise. The co-occurrence of a situational surprise (failure secondary to component failure) made it easy to redefine the issues in terms of local technical problems (eg, the lack of available spares). The easy availability of hardware failure as an explanation for the outage limited deeper analysis and understanding. In addition, the relative success of the adaptations to the failure paradoxically made deeper understanding seem less important.

27Temporal Factors. The full understanding of the incident did not develop until roughly 36 hours into the outage, so the initial characterisation of the problem as a hardware issue proved hard to dispel. In addition, the 24 x 7 x 365 nature of healthcare operations required urgent responses to prevent immediate harm to patients. This narrowed the focus of attention to actions that could be taken immediately to manage the disturbance, and moved deeper understanding to a lower priority.

28Cross-level Interactions. Different understandings were held at different levels of the organisation. The technical problem – unauthorized, unrecognized access to critical files – was harder for non-technical leadership to understand, particularly compared to the easily grasped story of component failure. Although one might suspect that the full story might have been embarrassing thus obscured or suppressed, this was not the case; the IT leadership was remarkably forthcoming in laying out the full explanation of what was known, as it became known.

29In addition, one might question whether it was even pertinent for the clinical arm of the organisation to undergo fundamental learning. Clinical operational units need to be prepared for the consequences of IT failures, but have little role in anticipating or preventing them.

30Healthcare-specific Factors. IT in healthcare has several unique characteristics that contributed to both the incident and to the difficulty of fundamental learning. In contrast to other hazardous activities, IT in health is subject to no safety oversight whatsoever. The principles of safety-critical computing are virtually unmentioned in a large medical informatics literature. Thus there is no locus in the organisation responsible for the safety of IT, and no individual or group who might be responsible for deeper learning from the incident.

31In addition, IT in healthcare is relatively new compared to other industries. The systems in use today are fundamentally “accidental systems”, built for one purpose (billing), and grown by accretion to support other functions for which they were never properly designed. This has led to “criticality creep”, in which functions originally thought to be optional gradually come to be used in mission-critical contexts, in which properties that were benign in their original setting now become hazardous.

32Diverting Factors. Finally, an external factor diverted at least senior leadership’s attention from a deeper exploration of the vulnerabilities whose presence this incident suggested. Nine months prior to this incident, the larger hospital system of which this organisation is a part made a commitment to install a monolithic, electronic medical records, order entry and results reporting system, provided by a different vendor across the entire system. Although full implementation was planned over a 5 year span, major components of the new system were scheduled to go live 9 months after the incident. This project gave the (misleading) appearance of a clean replacement of the previous system, a deus ex machina, and thus limited the felt need to understand the vagaries of the existing system more deeply, in addition to consuming a great deal of discretionary energy and resources.

Conclusion

33Fundamental surprise is a challenge for organisational resilience because anticipation is not a factor and monitoring is limited, typically, to evaluating the quality of response. Fundamental surprise also affords great opportunities for deep and fundamental learning, but it is difficult to effectively engage organisations fully in the learning process. In this case, the combination of situation and fundamental surprise blurred the distinction between them; situational adaptation and learning were remarkable, but the ease of re-interpreting fundamental as situational surprise meant fundamental learning was encapsulated, limited to only parts of the organisation.

Notes

1 University of Florida, 655 W 8th Street, Jacksonville, Florida 32209, USA wears@ufl.edu
Imperial College London, Praed Street, London W2 1NY, UK r.wears@imperial.ac.uk

2 University of Florida, 655 W 8th Street, Jacksonville, Florida 32209, USA kendall.web@jax.ufl.edu

© Presses des Mines, 2011

Terms of use: http://www.openedition.org/6540