1 Introduction

1A substantial amount of knowledge has accumulated on health and safety performance measurement systems (HSPMS) and performance measurement systems in general, discussing how to design, operate and evaluate such systems. However, a criticism often made of performance measurement systems concerns the inconsistency between the metrics and the company´s strategy (Neely et al., 1997). This failure means that the objects of measurement are not the most important from the standpoint of the company’s values and priorities, which, besides making the measurements irrelevant, can induce behaviours that conflict with the strategy. From the perspective of HSPMS, the priorities and values should be defined based on the HS paradigm, whether implicitly or explicitly adopted. By the HS paradigm, is understood the dominant outlook in the organization, on how accidents happen, why they happen, and how they can be avoided (Hollnagel et al., 2006). A HS paradigm contributes to HSPMS not being limited to measuring performance (are we getting better or worse?), but also may contribute to continuous improvement (how are we going to improve? Why are we good or bad?).

2This article discusses how resilience engineering (RE) can contribute to improving HSPMS. The article proposes a set of principles for designing HSPMS from the RE perspective. These principles are proposed as a way to complement the criteria that any performance measurement system should meet (e.g., the need for indicators to have goals) (Neely et al., 1997). These generic criteria, apart from not addressing the particularities of HS, place more emphasis on the design of specific indicators than on the overall design of the HSPMS. The use of the principles is illustrated by a case study, in which the HSPMS of a construction company was evaluated.

2 Principles for the design of HSPMS from the RE perspective

3The theoretical field of RE has been developed based on insights from a number of areas, such as high reliability organizations and complex systems. Due to this fact, some of the HSPMS design principles, presented below, are based on references prior to the first publication (Hollnagel et al., 2006) which coined the term RE.

1. It is impossible to monitor all hazards through an HSPMS: this principle stems from the inability to anticipate all hazards in complex systems. It can be considered a meta-principle, as its corollary is that none of the other principles can be perfectly implemented. Moreover, this principle implies that it is impossible to identify all the hazards that are not being monitored, since it is not possible to know them all. Thus, knowledge about the status of the system is always incomplete, besides being fragmented between different actors (Cilliers, 2005);

2. The HSPMS should monitor the organization's strategies for adjusting to variability: this principle follows from the above, to the extent that monitoring strategies so as to adjust to variability is a means of monitoring resources to tackle unknown hazards. Implementing this principle requires the organization to identify strategies for adjusting to variability, which, in itself, is a difficult task. The HSPMS should still be able to discern the positive variability of negative variability, so that the former is enhanced, and the latter minimized (Hollnagel et al., 2006);

3. The HSPMS must be resilient: the HSPMS should have the ability to adapt in order to continue capturing relevant information vis-à-vis the changes of a complex system. Implementing this principle requires the HSPMS to be fed by sources of information that are rich, diverse and continuous, thus keeping it compatible with the variety of the hazards. It should also be taken into account that safety is a social construct, which means that it is not something that exists independently of individuals nor can it be evaluated in a purely objective way (Rochlin, 1999). As a result, it is important that an HSPMS has mechanisms to capture the perceptions of individuals and groups on HS, which may reveal different nuances as to the hazards and different mental models that, perhaps, will conflict with each other;

4. The HSPMS should monitor vulnerabilities throughout the socio-technical system: a socio-technical system comprises sub-systems (e.g., technical, social, work organization and external environment) that interact with each other and do not have rigidly defined boundaries. However, some criteria may be used to define what makes part of the socio-technical system of interest to HSPMS. For example, this definition can be based on the criteria proposed by Hollnagel and Woods (2005) to define what should be included in the analysis of a joint cognitive system (Tabel 1).

5. The HSPMS should, as far as possible, deal with real-time monitoring: the dynamics of a complex system results in the information provided by HSPMS being out-of-step in relation to its real status. Thus, when the data are being analysed, the system is no longer as it was at the time the data were collected (Perrow, 1984). This principle also implies that, as an idealized situation, there is a continuous flow of HS information throughout all hierarchical levels, thus helping to reduce the time lag between the events and their being analysed;

6. HS management is inseparable from the management of other dimensions of the organization: as a result, the HSPMS should permeate all areas and activities, not just those normally associated with HS (Dekker, 2011). Based on this principle, one can also assume that the other performance measurement systems (e.g., those of production and environment) may, indirectly, provide important information for HSPMS;

7. The HSPMS should also be monitored: RE assumes that the organizational pressures on HS, to a greater or lesser extent, are inevitable. As a result, there is the tendency that the organization may gradually migrate towards the edge of losing control, with unsafe practices being incorporated into the routine and accepted as normal. Thus, there is a need to develop means to detect whether the HSPMS itself is deteriorating, such as external audits and metrics to assess its efficiency and effectiveness;

8. The more complex the system to be evaluated is, the HSPMS tends to be more complicated: the evaluation of HS in complex systems cannot be simplistic (e.g., by focusing on few indicators and subjects), lest it does not capture the nuances that comprise the situation. Unlike a complex system (such as a large construction project), a complicated system (such as a HSPMS) can be completely described, understood and controlled, as well as being maintained stable through compliance with a set of rules. As a main characteristic in common, both complex systems and complicated ones, consist of a large number of components that interact with each other (Dekker, 2011). Thus, an evidence of how complicated is an HSPMS, is the number of indicators it comprises. In particular, a complex system requires an HSPMS that has proactive indicators, which help predict future performance. This need is less in a stable and linear system, in which future performance is naturally more predictable (Perrow, 1984).

Table 1. Criteria for identifying sources of hazards and successes which should be monitored by the HSPMS (adapted from Hollnagel and Woods, 2005)

3 Research Method

4The company selected for the case study operates in Brazil, in the construction of high standard residential buildings. On all sites, there is a full-time safety specialist, who has responsibilities such as to collect and analyze indicators and to take part in weekly planning meetings involving both HS and production staff. This company was chosen because it has HS indicators that were not limited to those required by regulations, and has been implementing these in a standardized way on all its building sites.

5Data collection began by becoming familiar with the indicators used, their objectives, their formula, how data was collected, who was responsible for collecting data, processing and evaluating the results. Out of the company’s seventeen building sites, eight were periodically visited by two members of the research team. Over six months, each of these sites was visited at approximately fortnightly intervals (a total of 96 visits).

6With a view to reducing the level of abstraction of the HSPMS design principles, thus facilitating their empirical investigation, twenty questions were drawn up to support data collection (e.g., do the procedures for collecting, analysing and disseminating the metrics evolve over time?). The sources of data to answer each question were accessed over the six months of data collection, and consisted of: (a) documents, such as reports with the results of the indicators; (b) observing the procedures for collecting, analyzing and passing on each metric; (c) interviews with those involved in HSPMS, including safety specialists and workers. In Section 5 of this article, a discussion is made on the assessment of three out of the eight proposed principles for designing an HSPMS.

4 The HSPMS of the Company Investigated

7The company’s HSPMS has been developed incrementally and informally, based on initiatives from the HS staff, instead of top management. The information that formed the basis for calculating the indicators was generally collected directly by the safety specialists, who compiled the results of all the indicators monthly. Another specialist, allocated at the company’s headquarters, made a monthly compilation of the results from all the sites, and generated a report which provided the basis for a monthly meeting to discuss the results. On each building site, there was also a large notice board that showed the monthly results of all the indicators.

8The use of eight HS indicators was detected: (a) percentage of safety work packages (e.g., installation of guardrails) concluded (PSWc) within the deadline planned – it was calculated weekly, accompanied by an analysis and record of the causes that led to the non-implementation of the packages; (b) index of appropriateness to NR-18 (INR-18), which is the main Brazilian regulation concerned with HS in the construction industry; (c) estimate of fines because of non-compliance with NR-18; (d) reports of near-misses and frequency rate of near misses (FRNM); (e) index of training (IT), which assessed the percentage of man-hours of HS training in relation to the total of man-hours worked; (f) index of subcontractors´ performance (ISP), which allowed scores to be assigned to each subcontractor, for ten items (e.g., number of reports of near misses); (g) number of stoppages (NS) of production activities because of a lack of safety; (h) accident frequency rate (AFR).

5 Evaluation of the HSPMS Design Principles

5.1 The HSPMS should monitor the organization's strategies for adjusting to variability

9An analysis was made on how the HSPMS monitored two strategies for adjusting to the variability: adapting procedures; autonomy of the operational teams in decision making relevant to HS. The adaptation of procedures could be investigated, and measured, by monitoring normal work. However, no indicator evaluated normal work, but it did keep the focus on analyzing failures, that is to say, on situations where the adjustment to the variability did not exist or was not enough. In fact, even indicators that showed evidence of normal work and safety, rather than its absence, such as PSWc (e.g., it took into account safety packages implemented as planned), concentrated their analysis on failures (e.g., attributing causes for failing to implement safety packages), rather than on successes, thus missing opportunities to understand how adaptations were linked to normal outcomes. The analysis of normal work could also be guided by comparisons with the work prescribed. Some indicators provided information for analysis from this perspective. For example, reports of near misses and records of stoppages could be matched against the situation prescribed for performing the tasks involved in the events.

10Refusing to do a risky job was another possible way of dealing with variability. However, there were no indicators to assess whether workers refused to do tasks because of lack of safety, which was consistent with the absence of a policy to encourage this practice. The NS was the only indicator that directly reflected the autonomy of staff to take important decisions to control variability detrimental to HS, although this was restricted to the autonomy of the specialists to halt unsafe activities. Of course, this indicator could be probably more effective if workers, rather than only the specialists, had the autonomy to halt production for the sake of safety.

5.2 The HSPMS should monitor vulnerabilities throughout the socio-technical system

11As for mechanisms to broaden the perspectives on hazard identification and their possible monitoring, several indicators were based on safety specialists’ and workers’ impressions: (a) the FRNM and the ISP, to the extent that they required near-misses reports and, in order to report these events, the discernment to identify them was needed; (b) the NS contributed both to evaluating the specialists’ risk perception, and to stimulate perceiving it, since, in order to stop an activity, the specialist should make a judgement about the level of risk it involves.

12Some indicators had a narrow focus on certain elements of the socio-technical system, such as: (a) the INR-18, which detected failures related to the technological subsystem, i.e., whether physical protections were established and were in good repair; (b) the indicator that estimates fines, which helped to identify priority flaws in the technological subsystem, by using the amount of the fine as a parameter of prioritization. As to the pieces of information used to calculate three indicators (FRNM, NS, AFR), they could identify causal factors of any nature. As a result, these indicators could be interpreted as meta-monitoring mechanisms, i.e., means of monitoring the HSPMS itself. Since the data generated are not necessarily linked to certain types of vulnerabilities (as occurs, for example, with indicators that monitor specific elements of HS management, such as training), these indicators allow the tendencies of failure modes to be identified, thus contributing to keeping the HSPMS calibrated in relation to these. The NS, although it could reflect contributing factors of any nature, was the only indicator that directly evaluated by how much the external legal environment, represented by inspections by government agencies, played a role at the company´s sites. It was created since work stoppages by inspectors are frequent, in the region where this company operates.

13The analysis according to the principle of monitoring the entire socio-technical system also pointed to monitoring process safety being seen as quality monitoring, besides being conducted by an independent management structure of the kind that tracked personal safety. Similarly to what usually happens in civil construction, in the company investigated there were quality management actions related to process safety (e.g., checking the formwork supporting structure). The lack of a safety perspective on these actions reinforces the undesired separation between safety and production management, besides conveying the message that process safety hazards are either little relevant or that they are entirely under control. Two other gaps were identified concerning the principle of monitoring vulnerabilities throughout the socio-technical system: the non-existence of indicators related to occupational health; the non-existence of a measurement at a systemic level, such as an audit of the HSMS, which could help to identify the main threats and destabilization factors from a macro perspective.

5.3 Managing HS is inseparable from managing other dimensions of the organization

14From a holistic perspective, the whole HSPMS was in conflict with this principle, since it was based on the opposite assumption (i.e., HS is clearly separable from other dimensions). Indeed, all indicators were HS indicators, rather than, for example, HS plus production indicators. The design approach underlying the investigated HSPMS conveys the message that HS is a separate business area, which has a number of detrimental implications. Once the proposed principle is taken into account, the logic underlying some indicators can be questioned (e.g., why not to carry out a full assessment of subcontractors´ performance, rather than only a HS assessment?).

6 Conclusions

15Due to the abstract character of the principles, their application, whether in designing or evaluating an HSPMS, requires personnel who are familiar with RE and are domain experts, as occurred in the case study. Also, designers or evaluators should bear in mind that principles, on their own, do not lend themselves to mechanistic applications, but rather to heuristic ones. Consistent with the socio-technical approach underlying the principles, there is an inter-relationship between them. For example, the principle that the HSPMS should be resilient, is related to the principle that the HSPMS should monitor vulnerabilities throughout the socio-technical system. The use of this latter principle tends to produce a quantity and quality of information that can be used so that the HSPMS can be adjusted over time. The data collected illustrates the company managers’ lack of awareness of this relationship. In fact, the HSPMS monitored the number of training events conducted, although this, at no time, has been identified as a contributing factor in incidents. As a result, it is likely that the HSPMS could be adjusted, by excluding the indicator of the amount of training, without compromising the HS management. Because of such relationships, it is not possible that the data collected, or the various conclusions, are tightly linked to one or other principle.

16The relative importance of the eight principles is an important issue that was not dealt with in the case study. It was taken for granted that the investigated company was a complex system in all dimensions that characterize such systems, and, as result, all principles would be equally applicable for that context. However, some principles, such as real-time monitoring and resilience of the HSPMS, could not be so critical issues, since, from a macro perspective, the company operated in a fairly stable environment (e.g., it has existed for about 30 years, mastering construction technologies and product designs that are very similar across all sites). Thus, in future studies, the application of the principles should be preceded by a formal analysis of the system complexity, which might provide insights on the relative importance of each principle.