Version classiqueVersion mobile

Proceedings of the fourth Resilience Engineering Symposium

Erik Hollnagel
Éric Rigaud
Denis Besnard

Critical Decision Method To Access Resilience And Brittleness In Heavy Rainfall Forecast

Giovanni Dolif, Andre Engelbrecht, Alessandro Jatobá, Antônio Dias, José Orlando Gomes, Marcos R. S. Borges, Carlos A. Nobre et Paulo Victor Rodrigues de Carvalho


Natural disasters such as heavy rainfall may cause major damage and deaths. However if an adequate forecast is provided in enough time, the damage can be minimized. In Latin America and Brazil, where the socio­economic development characteristics lead to a disorderly growth of cities and, consequently, occupations and irregular buildings in risk areas, forecast of heavy rainfall and prevention and mitigation actions based on meteorological data/alerts are essential for save lives and minimize material losses, such as occurred in the mountainous region of Rio de Janeiro in 2011, where over 800 people lost their lives. This work describes the first research initiative on resilience engineering domain in systems to forecast heavy rains in Rio de Janeiro. The results indicate important sources of brittleness in the system that supports the work of meteorologists, mainly related to the technical and organizational framework and shows that the main source of resilience to deal with critical situations is the tacit knowledge of experts.

Note de l’auteur

Note portant sur l’auteur1

Note portant sur l’auteur2

Note portant sur l’auteur3

Note portant sur l’auteur4

Note portant sur l’auteur5

Note portant sur l’auteur6

Note portant sur l’auteur7

Note portant sur l’auteur8

Texte intégral

1 Introduction

1The study of meteorological phenomena and understanding the process involved in preparing their forecasts help to anticipate the warnings and improve efficiency, resulting in better prevention and mitigation of natural disasters triggered by extreme rainfall. Meteorological services are needed in several domains, from agriculture, which is directly influenced by meteorological phenomena, the ordinary person, when someone want to plan a weekend trip, passing through the industry sectors, like fashion, food, and air and land transportation among others.

2In Brazil, inadequate human settlement and urban concentrations are constantly growing. This growth enhances the vulnerability to extreme events in the atmosphere which increases the importance of the work of meteorologists.

3Natural disasters can cause major damage and deaths, but if warnings are provided within enough time, the damage can be minimized. In Latin America and Brazil, where the socio-economic and situation lead to a disorderly growth of cities and, consequently, occupations and irregular buildings in risk areas, prevention and mitigation actions based on meteorological data are essential for save lives and minimize property losses. To produce the meteorological information to be used in prevention and emergency response, the main weather forecaster’s tool is the mathematical model of the atmosphere. These numerical meteorological models were originally created and adjusted to predict weather events in the northern hemisphere mid-latitudes (Vincent and Karoly, 1998). In addition, the meteorologist in charge to deliver natural events forecast is still forced to deal with the strong pressure for accurate results in the shortest time possible.

4Heavy rainfall events have caused great tragedies in Brazil, as the floods of 2008 and 2009 in Santa Catarina and flooding in Rio de Janeiro at 1966,1988,1996,1998, 2011, and 11. On these occasions, the landslides caused huge loss of life, and economic losses and thousands homeless. The tragedy of February 1996 in Rio de Janeiro led to the creation of the Intense Rains and Landslides Alert System (Rio Alert), which aims to send warning bulletins to the population and municipal bodies about the risk of heavy rains that could trigger floods and landslides in the city.

5According to Zhu and Thoth (2001), a heavy rain event can be classified in several ways: climatologically, related to a forecast, or even according to a particular user or purpose. To the meteorologists of Rio Alert system heavy rain – large amounts of rainfall in a short time – is as important as long time rain, because both produce water accumulation in soil and slopes that lead to flooding and landslides slopes.

6In Rio Alert meteorologists are responsible for forecasting in the short term heavy rainfall events that may cause landslides in the city of Rio de Janeiro. They also have to decide the moment of issuing alert bulletins to trigger emergency procedures.

7The objective of this research has been to assess the resilience and brittleness in the work of meteorologists in the forecast of heavy rains in the city of Rio de Janeiro to suggest improvements in the Rio Alert system. To do so, we must elicit the knowledge of these meteorologists to understand how their experience, insight and knowledge are used, along with the tools available in the complex task of forecast heavy rains.

2 Method

8Combined cognitive task analysis techniques were employed such as field observation, interviews, semi-structured and structured forms of organization and knowledge representation, and the critical decision method – CDM (Crandal et al., 2006). These techniques were applied through a two-stage bootstrap process and allowed to capture data about the performance in routine tasks and during critical situations of heavy rainfall and alarm communications. The CDM was applied to rainstorm situations and contributed to the identification of aspects of cognition in the work of meteorologists, the elicitation of knowledge of experts, and the artifacts they used in these moments (Hoffman et al., 2006).

2.1 Understanding the forecaster basic knowledge

9The main objective of this phase was to gain familiarity with the domain studied, knowing the work environment, the practitioners and their routines, jargon and vocabulary, their knowledge and tools used, and build an efficient and transparent relationship with the study participants. The data collection was accomplished through the technique of storytelling, complemented by observations during weather forecast tasks. During some observation periods we request that the subject narrate what he was doing, according to the Think Aloud procedure (Someren et al., 1994). After data collection, the results obtained were presented to the forecasters for validation purpose. Based on the validation results a structured interview was prepared to increase knowledge elicited so far.

2.2 CDM in heavy rainfall

10At this stage the objective was to elicit the meteorologists’ knowledge in the forecast of heavy rain, deepening the results of phase 1. We used the Critical Decision Method (CDM), which starts by identifying an appropriate case (heavy rainfall forecast) recently experienced by a practitioner that is able to remember the case details and to retrieve data from the event. For proper CDM implementation, the case chosen should provide moments in which the practitioner had experienced a big leap in their situation comprehension or took an action that had affected the sequence of events (Crandall et al., 2006). The CDM method was applied in two sessions due to the complexity of the domain. The first session was basically for the case identification (selection of a heavy rain forecast) and the construction of the timeline, from where the critical events emerge. In the second session, a little longer, we used structured interviews to deeper our knowledge. In these interviews the meteorologist explained how he had dealt with each one of the events. After that, there were some rounds of validation, in which new aspects of knowledge were discovered.

3 Results

3.1 Phase 1 results: the meteorologist daily work

11During the first phase, the focus was the daily work of a meteorologist. The work is done by only one staff on each shift in a room with no windows, and no visualization of the weather the outside (Figure 1). The meteorological data are provided by computational tools that show the state of the atmosphere in the near past, and by mathematical models that predict their behavior in the near future. The practitioner observed does not take notes and all information is stored in his mind, forming a mental picture of the atmospheric dynamics.

12A set of documented procedures is available to help the practitioners in their decisions to change the alarm level and about the actions needed for each level. Practitioners reported that the availability of a window to outside, as an aid to the perception of cloudiness and contribution to the validation of data interpreted from the models, would be important to facilitate their work. The task of predicting the weather depends on a forecaster mental processing of the results of different atmosphere mathematical models. The tools and models they used, and the tasks order, vary from person to person, as well as the interpretation of correlated meteorological variables (e.g. temperature x wind x humidity) according to the different forecasting terms each model has. The mental model each forecaster built for consists of: (a) prior knowledge about the dynamics of the atmosphere, (b) current knowledge, received through the tools and by other information he receives, (c) theoretical basis, (d) experience, and (e) expertise. The influence of tacit knowledge or expertize appeared in the most important decision the Rio Alert meteorologist has to make: the change in alarm level. To make this decision, each meteorologist interviewed set his/her own parameters based on individual heuristics to determine if, when, where and how an intense rainfall that could reach the city of Rio de Janeiro may produce damage to a population that has to be alerted.

Figure 1 – Rio Alert system room.

3.2 Phase 2 results: CDM in a heavy rainfall event

13Using CDM we obtained details of Alert Rio practitioners’ activities during the heavy rainfall forecast. The data obtained were structured using coding schemes to facilitate analysis. This codification aims to classify the raw data organizing them to facilitate the comprehension of patterns in decision-making. We organize verbal protocols resulting from the structured interviews (based in heavy rainfall cases) in a set of categories that supported the decision making of meteorologists to issue a heavy rain warning. The categories used are: previous experience, theoretical basis, support tools, cues, procedures, and collaboration. This classification was elicited from a eight years’ experience meteorologist in weather forecasting, specialist in heavy rains forecast who is one the authors of this paper. Figure 2 shows the data share by category. We found that the previous experience of the forecaster (41 %) and the tools available (22 %) are the most important resources used by practitioners to issue a heavy rain warning.

4 Discussion

14The results suggest that the forecaster’s tacit knowledge (previous experience and cues used) is the most important factor (53 %) in carrying out a weather forecast for issuing a heavy rain warning. The decision to change a warning stage requires speed and confidence. Therefore the presence of experts in the Rio Alert system would enhance the resilience of the system because it would be able to produce good decisions on time.

Figure 2 – Content of verbal protocol transcriptions according to decision-making categories.

15The numerical weather prediction models, the most important support tool for forecasters, show deficiencies in predicting large volumes of rain and often present forecasts that are different among various models used. Problems with the use of support tools that appeared in case descriptions are related to inaccurate weather predictions, but also regarding previous information, (what had happened), and current information, (what is happening). Moreover, the number of weather stations is not always sufficient to diagnose all areas of the city. Therefore, the lack of technological support, combined with other structural problems, presses the forecasters’ work against the limits of error, and is a clearly source of system brittleness, and to overcome the problems and support tools inaccuracies, the practitioners must create and simultaneously execute new actions, i. e., they have to improvise without an adequate support.

16Work performance is needed to deal with the complexity of the real world. The normal performance of each meteorologist differs between themselves and from task prescription. The performance results from the adjustments they made based on individual heuristics to achieve success, as explained by Hollnagel’s ETTO principle (Hollnagel, 2009). Forecasters’ constantly have to made sacrifice decisions (Gomes et al, 2009) to deal with an intractable work environment (an environment in which principles of functioning are partially unknown – due its intrinsic characteristics – and because the inadequate support tools). Sacrifice decisions may lead to weather prediction failure or in predictions without enough anticipation to enable an adequate emergency response to the potential catastrophic effects of natural events. As shown in Figure 3, the more technological and structural problems accumulate, the availability of support tools decrease and the gap between the actual and required performance also increases. Improvisation and sacrifice decisions mainly based on tacit knowledge are the ways people find to do their work and close the gap, which does not necessarily prevent them from getting the job done correctly in most situations.

Figure 3 – Gap between work demands and required performance due the lack of resources.

17Moreover, there is a rising demand to issue weather conditions and warnings. In the case studied, the forecasters received more requests to issue weather conditions, outside the normal schedule established for the issuance of bulletins, and to issue earlier warnings. Both situations increase the sacrifice decisions need, and the gap as shown in Figure 3. The gap is supplied by the practitioners’ ability to adapt their expertise according to the support available in the system as a whole, improvising solutions to cope with the inadequate resources. Therefore, the quality of sacrifice decisions depends on the practitioners’ expertise, and it is variable from one professional to another. The practitioners’ expertise in weather forecast can be explained by the wide use of tacit knowledge in building the mental model of weather’s atmospheric dynamics. This model consists of prior personal knowledge that comes from study and training, and past experiences, and the knowledge about what is happening that is obtained using support tools.

5 Conclusion

18We analyzed key aspects of the meteorologists’ work in the forecast of heavy rains forecast in the city of Rio de Janeiro using Cognitive Task Analysis methods. It was possible to achieve a basic description of the forecaster performance in routine tasks and during decision-making in heavy rain situations. The case analysis of heavy rain contributed to elicit aspects of meteorologists’ cognition in the domain. Based on this analysis some sources of the system resilience and brittleness were identified like critical points in the system design and organization that contribute to push the practitioners’ performance to the boundaries of an adequate decision-making process endangering the heavy rain warning issuing. The experience and cues perception, which are based on expertise were identified as major factors for a resilient decision. Thus it is necessary to deepen the knowledge elicitation of experienced meteorologists to enable the maintenance of such knowledge and rapid learning of novices. This becomes even more necessary because often there are exchanges of meteorologists, especially in periods of renewal the outsourced service contract made by Rio de Janeiro City Administration each two years. From an organizational standpoint the results show the need to develop mechanisms to support collaboration between novice and experienced professionals, besides to keep the experienced meteorologists in the Rio Alert system staffing. The predominance of tools and models based on atmospheric dynamics of the northern hemisphere mid-latitudes requires greater cognitive effort to adapt the practitioners’ analysis for the southern hemisphere in tropical latitudes. Thus, there is a need to make an easier and faster access to accurate and correct meteorological information for the forecasters with better support tools. Better and more accurate information for checking or correcting data mentally, will naturally reduce the need for adaptation and sacrifice decisions made on poor information basis, avoiding the increased pressure on forecasters in critical situations like time pressure that also contributes to the inadequate warning decisions to anticipate heavy rains.


19The authors gratefully acknowledge the support of National Advice of Scientific and Technological Development (CNPq) and of Rio de Janeiro Research Support Foundation (FAPERJ).



Crandall, B., Klein, G., Hoffman, R. R. (2006) Working Minds. Cambridge MA.: The MIT Press.

Gomes J. O., Woods D .D., Carvalho, P.V.R., Huber, G. J., Borges, M.R.S. (2008). Resilience and brittleness in the offshore helicopter transportation system: the identification of constraints and sacrifice decisions in pilots’ work. Reliability Engineering and System Safety. 94 (2), 311-319.

Hoffman R. R., Coffey J., Ford K., Novak J. (2006) A Method for Eliciting, Preserving, and Sharing the Knowledge of Forecasters. Weather and Forecasting, 21, 416–428.

Hollnagel E. (2009). The ETTO principle. Surrey: Ashgate.

Someren, M. W., Barnard, Y. F. Sandberg, J. (1994). The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. London: Academic Press Limited. Vincent. D. and Karoly,D. (1998) Meteorology of the Southern Hemisphere. Boston, Mass.: American Meteorological Society.

Zhu, Y., Thot, Z. (2001). Extreme weather events and their probabilistic prediction by the NCEP ensembly forecast system. In: Preprints for the symposium on the precipitation extremes: Prediction, impacts and responses.


1 Centro de Ciência do Sistema Terrestre / Instituto de Pesquisas Espaciais (CCST/INPE), S. J. dos Campos, SP, Brazil

2 Programa de Pós-Graduação em Informática/Universidade Federal do Rio de Janeiro (PPGI/UFRJ), Rio de Janeiro, RJ, Brazil

3 Programa de Pós-Graduação em Informática/Universidade Federal do Rio de Janeiro (PPGI/UFRJ), Rio de Janeiro, RJ, Brazil

4 Programa de Pós-Graduação em Informática/Universidade Federal do Rio de Janeiro (PPGI/UFRJ), Rio de Janeiro, RJ, Brazil

5 Programa de Pós-Graduação em Informática/Universidade Federal do Rio de Janeiro (PPGI/UFRJ), Rio de Janeiro, RJ, Brazil

6 Programa de Pós-Graduação em Informática/Universidade Federal do Rio de Janeiro (PPGI/UFRJ), Rio de Janeiro, RJ, Brazil

7 Centro de Ciência do Sistema Terrestre / Instituto de Pesquisas Espaciais (CCST/INPE), S. J. dos Campos, SP, Brazil

8 Programa de Pós-Graduação em Informática/Universidade Federal do Rio de Janeiro (PPGI/UFRJ), Rio de Janeiro, RJ, Brazil

Table des illustrations

Légende Figure 1 – Rio Alert system room.
Fichier image/jpeg, 716k
Légende Figure 2 – Content of verbal protocol transcriptions according to decision-making categories.
Fichier image/jpeg, 128k
Légende Figure 3 – Gap between work demands and required performance due the lack of resources.
Fichier image/jpeg, 60k

Le texte et les autres éléments (illustrations, fichiers annexes importés) sont sous Licence OpenEdition Books, sauf mention contraire.

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search