Desktop versionMobile version
OpenEdition Books

Proceedings of the fourth Resilience Engineering Symposium

 | 
Erik Hollnagel
, 
Éric Rigaud
, 
Denis Besnard

Generating Resilience: Handling Extraordinarily High Work Load in High Three (Intensity, Technology, Reliability) Maintenance Work

Elizabeth Lay, Joshua Hecht and Selena Shaefer

Abstract

Engineering is the discipline of applying art or science to practical problems. Recently Siemens field service maintenance group found themselves a short time out from the start of a never before seen demand in workload. Multiple customers were preparing to shutdown power plants for maintenance outages. Several outages from the previous season, where schedule was extended due to emergent work, were further taxing already limited resources. Common risk assessment methods were assessed as not adequate to address this complex, urgent situation where there was a need to respond quickly to address risk and uncertainty across the entire portfolio. Siemens turned to resilience engineering for a solution. The term “resilience engineering” was foreign to most who were involved, yet they were open to learn and apply the concepts. This began with a workshop to design a strategy to respond more resiliently. This paper shares the methodology used in the workshop, the actions taken to increase resilience, and lessons in handling high stress, urgent situations.

Author's note

Note portant sur l’auteur1

Note portant sur l’auteur2

Note portant sur l’auteur3

Full text

There are no secrets to success. It is the result of preparation, hard work, and learning from failure.”
Colin Powell

Introduction

1While the characteristics of already existing High Reliability Organizations (HROs) have been thoroughly studied, purposefully engineering resilience into an organization or situation is uncommon. Anticipating reaching a significantly higher peak in work load, one Siemens field service maintenance group recognized the need to respond differently to the portfolio risk situation along with a desire to respond robustly to emerging risks during a time of high stress and fully deployed resources. Although most were unfamiliar with the philosophies of HROs, leaders in this organization were open to working with the authors of this paper to experiment in increasing resilience. The group participating in this workshop historically did well at responding to the unexpected. As this part of the business had grown, cracks from an over reliance on firefighting had begun to appear. Using the science of resilience, a workshop was held to engineer a response strategy including operational tactics.

2Background. Siemens Energy Inc services and supplies equipment to power generation, transmission, and distribution providers. This paper is written about the business of performing maintenance on turbine-generators; work that can be comparable to aviation line maintenance and hospital emergency rooms in the urgency, challenges, and possible high loss outcomes. Power generation equipment maintenance is a high 3 (intensity, technology, reliability) industry. Schedules are frequently tight; work is highly coordinated, technically complex and often performed on a 24 hr./7 day per week shift basis. Promptly after a power plant is shut-down, Siemens crews begin the business of disassembling, inspecting, modifying, repairing, reassembling, testing, and re-starting the turbine-generator. Unexpected situations can occur given the variability in work environment (occasional extreme temperatures, tight spaces), turbine-generator condition (components stuck, cracked, worn), and the mobilization of large numbers of people and specialty tools to remote areas of the world. Siemens considers power generation equipment maintenance a critical outcome industry and becoming more so as reserve margin shrinks. The work is cyclic and typically peaks opposite electric load demand; higher in spring and fall when weather is temperate.

3Resilience compared with traditional risk assessment. Traditional risk assessments typically include identifying risks (specific and detailed), analyzing the risks (qualify, quantify, rank), and designing specific responses for highly ranked risks. Risk assessment often focuses on preventing things that can go wrong.

4Resilience engineering involves designing to ensure things go right with more focus on preparedness and less on prediction. Consideration is given to broad, big picture situations and uncertainties. Responses tend to be general versus specific. Resilience engineering involves:

5Bound uncertainty and possible outcomes:

  • Consider properties of systems when analyzing possible failures.

  • Consider fundamental limitations of resources.

  • Describe possible big picture outcomes.

  • Seek to be approximately right across a broad set of eventualities. (Taleb, 2010) General response design:

  • Make the distinction between positive and negative contingencies; seize opportunities (Taleb, 2010)

  • Invest in preparedness not prediction. (Taleb, 2010)

  • Since variability is inevitable, look for ways to build structure around and plan for variability.

  • Design general responses that could address a broad set of situations (don’t focus on the precise and local).

6Uncertainty is defined as the state of having limited knowledge where it is not possible to exactly describe existing state or future outcome, it is an un-measurable risk and includes what we don’t know, ambiguity, and / or variability.

7In project management, according to De Meyer, there are 4 types of uncertainty (De Meyer et al, 2002, 61-62):

  • Variation: a range of values on a particular activity

  • Foreseen Uncertainty: identifiable and understood influences that the team cannot be sure will occur.

  • Unforeseen Uncertainty: can’t be identified in planning, team is unaware of event’s possibility or considers it unlikely. Also called “unknown unknowns”.

  • Chaos: Even the basic structure of the plan is uncertain. There is constant change, iteration, evolution. Final results may be completely different from original intent. During risk assessment, the tendency can be to act like the future can be more accurately predicted than is possible, such as when probabilities are estimated to a high degree of granularity. In risk assessment, uncertainty may be neglected. This is due, in part, to the psychological make up of humans; studies have shown people are more averse to uncertainty than to risk alone. (Platt & Huettel, 2008, 398-403) To be highly resilient is to be prepared for uncertainty. To be highly resilient is to respond robustly to the unexpected.

Workshop

8The purpose of the workshop was to evaluate portfolio risks and uncertainties and deliver a plan to increase resilience. Workshop design was based on the following properties for increased resilience (Woods, 2006, 23):

  • buffering capacity: system ability to absorb disruptions without breaking down;

  • flexibility versus stiffness: ability to restructure in response to changes;

  • margin: how close and how precarious system is operating relative to performance boundaries;

  • tolerance: system behavior near boundaries; degrades gracefully or collapses. Workshop Structure. This one day workshop was attended by representatives from operations, engineering, marketing, project management, resource planning (parts, people, and tools), and led by risk management. The workshop began with an introduction to resilience engineering including examples from other industries and Siemens’ business. Core resilience concepts were shared in the form of a Resilience Glossary.

9Questions designed around HRO principles and properties of resilient systems facilitated exploration of uncertainties, assumptions, areas of brittleness, prior and possible system failures, interface and inter-related project risks, location and indicators of approaching margins, and possible resilient solutions. See Table 1 for sample questions.

10HRO Principles (Weick K., Sutcliffe, K., 2001, 9-17):

  1. Preoccupation with Failure

  2. Reluctance to Simplify

  3. Sensitivity to Operations

  4. Commitment to Resilience

  5. Deference to Expertise

Table 1: Sample workshop questions.

Table 1: Sample workshop questions.

11Probing assumptions and uncertainties brought forth several brittle assumptions such as a short turn around time for a critical tool set (needed for several outages, no extras existed) going perfectly each time and the assumption that all outages would be completed by originally scheduled dates. While schedules include emergent work based on the prior 5 yr average, end date extensions would likely be different as emergent work is not distributed evenly in real life. Additionally, the plan did not include additional outages arising due to unplanned maintenance even though history shows this to be common. Probing cross project inter-dependencies brought forth that three projects with the same new design were beginning slightly staggered, and to be performed in parallel, leaving little time to recover should there be an unexpected issue.

12Probing for potential critical disruptions and brittle situations ran longer than expected and resulted in identifying more than 150 risks, uncertainties, and assumptions. Each participant provided their assessment of the top 3 most critical, urgent risks, uncertainties, or brittle assumptions. There was no time to design pinging (proactive probing for risk profile changes) (Wreathall et al, 2003), as had been intended, but a “straw man” for this purpose is shared below as it could be used in future workshops. Should another workshop be held, it would include one day for training and brainstorming followed by one day for designing tactics.

Table 2. “Straw man” for pinging design. Indicators that risk level is increasing.

Table 2. “Straw man” for pinging design. Indicators that risk level is increasing.

13Design to increase resilience. Using resilience engineering principles, the authors provided and helped implement suggestions that were accepted for resilient responses to the risks, uncertainties, and assumptions generated in the workshop. Several examples are provided below:

  • Pre-assign “reserve army” to planned outages to reduce flux caused by emergent work and unplanned outages. Assign qualified back office personnel with maintenance experience to planned outages scheduled during peak load, leaving active field personnel available to respond to unplanned work. Back office personnel, who work less often in the field, then have the time to plan and prepare. The more current firefighting skills of active field personnel enable them to respond to effectively to the unexpected. This also bounded the volume of back office buffer; projects were turned down that would have gone beyond this buffer and put the organization at risk of not being able to perform successfully.

  • Form crisis management and firefighting teams to bring about a heightened state of coordination and help. These teams can improve communication with front lines and can work to remove barriers and expedite solutions. Additional details provided in next section.

  • Increase time available to respond to potential issues through early inspections of new design to existing equipment interfaces and other areas where issues may be encountered.

  • Implement 24 hr./7 day per week service engineering support of the front lines. This could involve negotiating cross organizational agreements, finding funding, securing engineering buy-in, obtaining phones, and defining process for rotation and call management.

Implementing Resilience Strategy

14Once the outage season began, things really got interesting. Latent issues manifested into breakdowns at the front lines. Resources were redirected and stretched to deal with these situations. The water was coming up and there was a need to respond quickly and reinforce the dike and to move to a global assessment with a heightened state of coordinated action. Where was the dike going to breach next? Organizations have a tendency to handle issues serially versus holistically. We began questioning: (of the project engineers) what is keeping you awake at night? Who is at the point they can’t keep up? (Of the portfolio risk owners) Do you agree with the response plans?

15Mobilizing action. A sense of urgency was triggered among the stakeholders by creating and distributing “risk profiles” of the on-going outages which outlined scope, order value, project leads, issues and risks that had developed. The combined profiles were distributed quickly to a fairly high level. This caused a reaction, both positive and negative. Several project personnel believed they should have had the opportunity to review and approve the risk profiles prior to escalating. In justifying the quick escalation, one director compared it to responsibility of an individual on the turbine deck to stop a lift should they notice a problem; there are times when it is appropriate to act with the same sense of urgency in the office. Yet, this still needs to be balanced with not catching an already stressed project team off guard.

16Crisis management team. As load was ramping up, a crisis management team consisting of first and second line managers, was formed for the purpose of monitoring on-going outages for issues or trends that needed rapid responses and then guiding and coordinating actions to address the issues. This team strengthened leadership’s connection to the front lines and provided a forum for project managers to bring issues to management’s attention. These managers had the authority to add to resources where needed and to make decisions to keep actions moving. Meetings were short (30 minutes) and held on Monday, Wednesday, and Fridays. Those issues which could affect more than one outage or had the potential for significant critical path impact were reviewed. They also paid attention to multiple issues arising on singular outages that could indicate a project was not heading in the desired direction. Communication of system-wide issues enabled project managers to evaluate if their project was impacted and anticipate and act to address potential issues. The crisis team’s approach was different from traditional risk management wherein each project team addressed issues associated only with their project. While issues were fed up the management chain in each individual function, it sometimes led to inconsistent communication and possible missed impacts on other projects. Historically, cross-functional communication at the management level tended to take place only after an issue had grown; it was sometimes too late to contain it – an arrangement that did not lend itself to identification of system-wide issues.

17Dedicated rapid response team. As risks and issues multiplied, a team was formed and assigned full time to removing barriers and implementing solutions. The neutrality of this cross organization group helped smooth political tensions. Members aggressively addressed issues that had the potential to delay work in the field. As time allowed, they worked on longer term solutions to prevent reoccurrence. One team member was trained in resilience and scanned situations for areas of brittleness where buffering or other actions could help.

18Diagnosis of Brittleness. The dedicated rapid response and crisis management teams looked for signs of brittleness such as incomplete, unclear information or statuses, silo situations where workers were not optimally connected with front lines, communication issues, accuracy of assumptions, over committed critical resources, and key individuals for whom there was no back-up. One example of brittleness emerged during a document review. On-going and recently completed outages indicated an urgent need to check documents relevant to outages that had already begun or would start soon. The group responsible for checking the documents was already highly loaded in responding to emerging situations at the frontlines. In the rush to begin the reviews and with the distraction created by the dynamic situation of supporting frontline, urgent needs, details of what was to be reviewed and a realistic plan to accomplish this, was missed. A few weeks into the reviews, it was discovered that several individuals had not made progress, vacation plans would interrupt one critical review, one reviewer stated that their assignment was not totally clear to them, and a check that operations thought to be critical was not verified as having been done. A finding was that even, perhaps especially, in a rush situation, who, what, when, where, and why need to be defined and agreed to by stakeholders and those who will perform the work.

19Bring in expert help. Even if help is needed, it is difficult to bring a new person into the mix while in the midst of a critical situation. The effort to bring the person up to speed may outweigh the benefit. Several experiments were run on bringing in help. First, a list of experts (most recent retirees) was circulated among the groups known to be highly loaded. No help was requested. Next an individual, highly knowledgeable in the domain in which some issues were occurring, was brought in with this mission: talk to people, assess the situation, and determine if and where you can offer help. Within a day, several engineers requested the expert’s review of and advice on designs that were soon to be implemented. The expert identified several issues in time to address and was able to bring deep expertise and a calm mood. He speculated that his unfamiliarity with current systems and arriving in the middle of the work may have been detrimental and questioned whether he’d helped or been a hindrance. The expert was initially released after one week but called back about a week later to help again.

Conclusions

20As of the writing of this paper, the spring outage season was still in progress. While the workshop was considered successful, it could have had even more impact if it had been held earlier which would have allowed implementation of additional tactics. Actions that were taken were beneficial. Issues were identified and addressed in time to prevent incidents. The tools and philosophies of resilience engineering enabled system wide response that would not have been possible with a linear, project focus. The crisis management and dedicated rapid response teams had the power to act to move resources, remove most barriers, and set priorities. They had separation from individual projects which enabled them to find project inter-relations and determine consequences such that they could orchestrate actions and guide the business to adapt in a way that benefitted the entire portfolio. Resilience engineering is being implemented as a “grass roots” effort. One important tactic was counteracted by a manager with a different agenda and is still being worked out. Change is difficult and even more so without goal alignment. In order to have goal alignment, leadership must be onboard. This will begin with finding and enrolling an executive sponsor with the necessary authority and power remove additional barriers. Changing culture begins with language. Siemens is speaking the language of resilience and is in a different narrative in our response to challenging situations.

Bibliography

References

De Meyer, A., Loch, C. H., Pich, M. T. (2002). Managing Project Uncertainty: From Variation to Chaos, MIT Sloan Management Review, Winter Vol., 60 - 67

Platt, M. L. & Huettel, S. A. (2008). Risky business: the neuroeconomics of decision making under uncertainty. Nature Neuroscience, 11, 398 – 403.

Taleb, N. N. (2010) The Black Swan. 2nd edition. Random House.

Weick, K.. Sutcliffe, K. (2001). Managing the Unexpected: Assuring High Performance in an Age of Complexity, Jossey-Bass, 9 – 17.

Woods, D. D. (2006). Essential Characteristics of Resilience. Resilience Engineering: Concepts and Precepts. E. Hollnagel, D. Woods and N. Leveson. Burlington, VT, Ashgate Publishing Co. 21 – 34.

Wreathall, J. and A. C. Merritt (2003). Managing Human Performance in the Modern World: Developments in the US Nuclear Industry. Innovation and Consolidation in Aviation. G. Edkins and P. Pfister. Burlington, VT, Ashgate Publishing Co.

Notes

1 Siemens Energy, 4400 Alafaya Trail, Orlando, FL 32826 USA Beth.Lay@Siemens.com

2 Siemens Energy, 4400 Alafaya Trail, Orlando, FL 32826 USA Joshua.Hecht@Siemens.com

3 Siemens Energy, 4400 Alafaya Trail, Orlando, FL 32826 USA Selena.Shaefer@Siemens.com

List of illustrations

Title Table 1: Sample workshop questions.
URL http://books.openedition.org/pressesmines/docannexe/image/1034/img-1.jpg
File image/jpeg, 412k
Title Table 2. “Straw man” for pinging design. Indicators that risk level is increasing.
URL http://books.openedition.org/pressesmines/docannexe/image/1034/img-2.jpg
File image/jpeg, 72k
URL http://books.openedition.org/pressesmines/docannexe/image/1034/img-3.jpg
File image/jpeg, 229k

© Presses des Mines, 2011

Terms of use: http://www.openedition.org/6540