Quantifying Safety Culture

November 18, 2020 Aaron Sammut

Safety culture is often considered too difficult to quantify. Researchers have proposed a new methodology in benchmarking safety culture in organisations.

By KATARZYNA WARSZAWSKA & ANDRZEJ KRASLAWSKI
First published in the SPRING 2020 edition of Australasian Mine Safety Journal

Many events with catastrophic consequences occurred over past decades. One of them was the disaster in Bhopal. A weak safety culture is mentioned as a one of the causes of this tragic event. It is a generally accepted fact that the level of safety culture has to be continuously evaluated and improved to avoid the occurrence of such situations in the future. This paper introduces a new method – Assessment Tree Method (ATM) allowing for quantitative estimation of the level of safety culture in an organization.

The ATM enables the determination and quantification of the key aspects of safety culture on the basis of assessment trees. The structure of the trees is analogical to fault tree, a commonly used tool for the analysis of accidents. The assessment process is based on the guided interviews. As a result, a deeper insight is gained and effective identification and elimination of weak points in safety culture enabled.


“The most important advantages of the proposed method are quantification and simplicity enabling assessment of a broad scope of the specific aspects of safety culture.”


Next, the level of safety culture is calculated taking into account the results obtained for key safety aspects specific for a given organisation as well as their weights of importance. The ATM was tested for the evaluation of safety culture at the university department. There were twenty students, and three staff members responsible for safety, participating in the study. The six key aspects of safety culture were identified: knowledge & skills; awareness; flow of information; monitoring & control & supervision; management commitment and continuous improvement. On the basis of the obtained results it was concluded that the level of safety culture is high and some recommendations were given to make it even better.

The most important advantages of the proposed method are quantification and simplicity enabling assessment of a broad scope of the specific aspects of safety culture. The proposed method could be used for the standardised benchmarking of the different organisations.

Introduction

For the first time, the term “safety culture” was used in the report on the nuclear disaster at Chernobyl in 1986, published by International Atomic Energy Agency (IAEA, 1991). The definition of safety culture presented in this report is given as:

“Safety culture is that assembly of characteristics and attitudes in organizations and individuals which establishes that, as an overriding priority, nuclear plant safety issues receive the attention warranted by their significance”.

In the following years, the analysis of other disasters e.g. the explosion on the Piper Alpha offshore drilling platform (Cullen, 1990) or the Clapham Junction rail crash (Hidden, 1989), confirmed the importance of safety culture. The increasing interest in the concept of safety culture leads to intensive research in this field (Yule, 2003; Pidgeon, 1991; Turner et al., 1989). The conducted research clearly identified the strong relationship between the level of safety culture and the number of accidents. As it was to be expected, the higher level of safety culture had a positive impact on the decrease of the number of accidents (Milczarek, 2000). One of the major subjects of research is identification of the aspects of safety culture (Ejdys, 2010; Pidgeon, 1991). For example, He et al. (2012) recognized 32 elements of safety culture.

Another important aspects of research are attempts to quantify the level of safety culture (Yule, 2003; Cooper, 2008). In the past, the level of safety at work was determined on the basis of few quantitative components, such as: financial loss incurred as a result of accidents, the number of the days of the absence of employees, the days of downtimes in factories etc. (Janicak, 2003). However, such approach was not applicable to the issue of quantification of safety culture.

The existing methods of safety culture assessment are qualitative. They are using inter alia interviews, expert ratings, audits (Yule, 2003). The mentioned approaches allow to obtain a lot of information on the level of awareness of workers or motivation of their actions.

John S. Carroll (1998) proposed use of the survey consisting mainly of questions, each to be answered on a four-category scale (strongly disagree, disagree, agree, strongly agree), and interviews to measure safety culture. He also recognized the problem of a quantitative study of this issue. The suggestion to use the checklist (Score Your Safety Culture Checklist) to assess the level of safety culture was given by Reason (2011). The method consists in assessment of twenty situations related to various aspects of safety culture. The respondent gives “yes/no” answers and the final score is a sum of positive replies. The other method is Safety Culture Grid proposed by G. Kirschstein and E. Werner-Keppner (Ejdys, 2010). Cooper (2008) tried to introduce quantitative method taking into account the risk. The author proposes to conduct audit and to use surveys. The specific behavioural issues are also determined using inter alia focus group exercises. The final result is given in percentage.

The growing demand for more precise, quantitative assessment of safety culture starts to be a noticeable element of many discussions on safety in the enterprises. This commonly expressed request is a major motivation of this research.

The presented paper introduces Assessment Tree Method (ATM) for the quantitative assessment of safety culture. The proposed
method allows for assessment of the specific aspects of safety culture as well as gives it’s the numerical value.

Assessment Tree Method (ATM)

ATM enables the determination and quantification of the key aspects of safety culture. In the first step, the key safety aspects are determined on the basis of experience and knowledge of the evaluator, existing scientific literature as well as the analysed documents such as work accident and near miss reports. The examples of key aspects are: personal attitude towards safety related issues, awareness of safety, leaders’ responsibility, etc. (He et al., 2012). Next, the assessment trees are built for every key aspect of safety culture and the series of interviews are conducted to analyse all trees. The interview enables the navigation in the assessment tree. It means to analyse a path from the top of the tree to its bottom. The position at the bottom of the tree, resulting from the answers to the questions asked, determines the numeric value assigned to the given key concept of safety culture.

Basics of the method

The structure of assessment tree is analogical to fault tree, a commonly used tool for the analysis of accidents. The assessment tree is presented in Fig. 1. The assessment process is based on the guided interviews following the scheme presented in Fig. 1. Each of the rectangles in Fig.1 represents one question related to a safety aspect evaluated in a given tree. During the interview, the interviewer checks if the answer given by respondent is an “expected answer”. The “expected answer” is the response which is addressing the specific aspect of safety according to the existing rules, standards and safety regulations as well as the best practices. Then option A (Fig. 1) is followed when respondent gives “expected answer”, option B when the different answer is obtained. The expected answers should be established always before the interviews start. The answer obtained from the respondent determines the sequence of the subsequent questions to be asked.

The hierarchy of the questions in a given tree is shown in Fig. 2. The questions are arranged according to the degree of detail, starting with the most general issues at the top and ending with the detailed questions at the bottom of the pyramid. It is worth to underline that during the formulations of assessment tree, reviewer can also arrange the question according to the increasing level of detail For example the question no. 5 in Table 1 checks respondent knowledge about individual protection which should be used during the work in the lab. If respondent gives “expected answer” then the reviewer verify his/hers knowledge more deeply using question 10. It determines what the respondent should do if the individual protection is not available or damaged. When getting “expected answer” the reviewer could be sure that the respondent’s knowledge is not superficial. When the different answer is obtained then question no. 11 is asked. It must concern a totally new subject in order to verify respondent knowledge in measured aspect from a different point of view. The trees for the specific aspects are composed of the questions given in Tables 1e6. The number of the question determines its position in the tree as in Fig. 1.

The first, main question at the top of the pyramided is related to the analysed aspect of safety. Next, depending on the responses obtained, the specific path of the interview is established and the final score is obtained at the bottom of the tree. The interviews are repeated for every member of the analysed group. The interviews are designed as a discussion during which the answers to the particular questions are obtained. It requires that the interviewer is raising the issues depending on the obtained previously answers so that the process is moving in the controlled manner from the top to the bottom of the tree. In consequence, there are needed four answers in a given tree in order to assess a specified aspect of safety culture. The total number of the questions depends on the number of trees created for the assessment of various aspects of safety culture, e.g. five aspects require five trees and answer to 20 questions.


“The existing methods of safety culture assessment are qualitative.”


It is important to note that the questions placed in the tree could be arranged with respect to the level of safety culture. The different questions correspond to the different level of safety culture. It means that the positioning of the question in a given tree reflects the level culture. For example the question number 8, in Fig. 1, deal with a higher level of safety culture than the question 15. The values obtained at the bottom of the tree cover the range from 1 to 0 where 1 corresponds to the highest level of a given aspect of safety culture. The trees built on these principles create a logical entity that enables the assessment and identification of the weak points which should be eliminated in order to achieve a high level of safety culture.

Algorithm of assessment

  • The Assessment Tree Method is composed of the following steps:
  • Identification of key aspects of safety culture and creation of the corresponding assessment trees;
  • Answers to the questions given in the assessment trees and assessment of the specified aspects of safety culture;
  • Analysis of the obtained results and suggestions for the changes aimed at the improvement of safety culture.

The actions to be taken during the implementation of proposed method are presented in Fig. 3. First, the assessment team is built. It is recommended that one of the members of the team should be a person from outside of the organisation. It should ensure the impartiality of the process. Other members of the team are selected from among the employees. The membership in the assessment team is based on the employee’s duties, training, and length of service. It should be stressed that the close cooperation with the management of the assessed organisation is essential; especially, in the phase of determination of the aspects of safety culture and building the assessment tree. The second step consists in determination of the scope of assessment and the specific aspects of safety culture. The areas to be evaluated should be determined on the basis of the organisational structure of company. The key aspects of the safety culture are defined using various documents, e.g. accidents and near misses reports as well as the experience of the members of assessment team. For example, in the organisation analysed in this paper, there have been identified six aspects of safety culture as given in Fig. 4.

The presented aspects have been chosen on the basis of safety documentation and the specific working conditions at the analysed unit. The proposed aspects may be also used for the analysis of the situation in other units. The identified aspects form a system where all elements are interlinked. The omission of one of the elements may cause the deterioration of the general level of safety culture in a company. In the following step, the methods of data collection and storage should be specified, e.g. the form of the interviews, their duration, recording etc. Next, the assessment team proposes the questions to be asked during the interviews. The examples of the questions are given in the Tables 1-6.

The interviews of the employees are conducted in the next phase of the assessment exercise, and they are focused around the questions given in the assessment trees. During the interview, several trees are used, each representing the specific aspect of safety culture, e.g. awareness, management commitment, continuous improvement, etc.

In the next step, the weights of importance of the specific aspects of safety culture are determined using AHP method (Saaty, 1987/2008). After that, the calculations are conducted, and the level of safety culture is determined using five-level scale as given
in Table 7.

Finally, the comprehensive evaluation is performed and the report presenting the level of safety culture prepared. When the level of safety culture is acceptable, one should strive for continuous improvement and such assessment should be made periodically. If the level of safety culture is low then the obtained results should help to identify the weaknesses contributing to the decrease of safety culture in the organisation. In this case, recommendations given in the report should be introduced.

Then, when the changes are introduced, the assessment should be repeated in order to verify the effectiveness of the introduced changes. The deadline for the next assessment may be also proposed in the report. However, the actions should be carried out at a right pace as the changing the culture in an organisation is usually a long and complicated process.

The Eq. (1) is used to assess the value of a given key aspect of safety culture on the basis of the answers obtained from interviewing one employee:

EQ (1)

Where: ASj – value of j aspect of safety culture obtained from interviewing person k; xi k = assessment of the answer given by person k; 0.25 when the answer is compatible with “expected answer”; xi k = assessment of the answer given by person k; 0 when the answer is not compatible with “expected answer”.

The assessment of the j key aspect of a safety culture is obtained as an average of the results obtained from interviewing m employees.

EQ (2)

where m is a number of the persons interviewed. More realistic assessment of safety culture could be obtained if the different key aspects of safety culture would have the different weights of importance. Therefore, in this work Analytic Hierarchy Process (AHP) was used to identify the weights of importance, (Saaty, 2008). It is recommended that all of the respondents taking part in the interviews will participate in the paired comparison of the key aspects. Based on the results obtained from the respondents, the matrix should be formed taking into account the Saaty Ranking Scale (Saaty, 1987). Next, the calculations are conducted, in accordance with the AHP method. After that, the Consistency Index (CI) and Consistency Ratio (CR) should also be calculated (Saaty, 2008).

On the basis of CR value, it may be concluded whether the proportions between the different elements are preserved. Therefore, it is believed that the coefficient CR should not exceed 0.1. However, the exceptions from this rule are allowed (Coyle, 2004; Saaty, 1990/2008). In this case, the authors of ATM method allow for the higher CR value as it is justified to expect some incoherence of answers given by the relatively big group of respondents. Finally using the obtained weights of importance the level of the safety culture could be assessed as

EQ (3)

Where: Asc – the assessed level of safety culture; wj – weight of importance of j key aspect of safety culture; n – number of key aspects of safety culture.

Case study

The ATM was used for the assessment of safety culture at the Department of Chemical Technology, Lappeenranta University of Technology, Finland. There were twenty students participating in the described study and three staff members responsible for safety related issues. There were identified six key aspects of safety culture, Fig. 4. The students answered the questions concerning knowledge & skills; awareness; flow of information; monitoring & control & supervision while the staff members answered question on flow of information; monitoring & control & supervision; management commitment and continuous improvement.

It is worth to note that the results obtained in both groups for two aspects (flow of information and monitoring & control & supervision) were analysed separately but in Fig. 5 they are presented jointly.

There were five alternatives of answer for every key aspect of safety assessed:

  • 0 when respondent does not give any answer in accordance with the key answer;
  • 0.25 when the respondent gives only one answer which is compatible with the key answer;
  • 0.5 when respondent gives two answers which are compatible with the key answer;
  • 0.75 when respondent gives three answers which are compatible with the key answer;
  • 1 when respondent gives four answers which are compatible with the key answer.

The results obtained on the basis of students and staff members interviews are given in Fig. 5. The weights of importance of key aspects of safety culture were determined on the basis of interviews of 17 respondents and use of AHP method. The obtained weights of importance are presented in Table 8. The value of Consistency Ratio (CR) was equal to 0.085. It is worth to note, that the need for continuous improvement (0.45) was the most valued part of safety culture while the value of flow of information (0.03) was practically negligible. The assessments of key aspects of safety culture were obtained as the average value of the results obtained in the groups of students and staff. The results are given in Table 9.

Finally, the level of safety culture was calculated taking into account the results obtained for all key aspects and their weights of importance, Eq. (3). The assessed value of level of safety culture Asc was 0.82. It means that, according to Table 1, an “expected level of safety culture” was achieved. This level should not be downgraded as the values of only two key aspects were lower than 0.6.

However, the actions aimed at strengthening these aspects should be undertaken. On the basis of the obtained results the following changes/actions were proposed. Management commitment should be improved. The students should be informed about the involvement of management.

Due to the large number of foreign students, each safety training should be completed by the test, to verify the understanding of safety related issues. The exercise on the use of safety equipment should be a common practice. The students, especially those working over one year in the laboratories, should be asked for their opinions on safety issue.

Summary

Assessment Tree Method (ATM) offers a new approach to safety culture assessment enabling its quantitative evaluation. The characteristic element of this method is hierarchical, tree-based organisation of interviewing process. The most important advantage of ATM is its simplicity and simultaneously a broad scope of the assessed aspects of safety culture.

The method allows for quantitative assessment of the specific aspects of safety as well as to evaluate the level of safety culture in the organisation. As a result, it offers a deeper insight and in consequence, effective identification and elimination of weak points in safety culture. The presented algorithm could be successfully applied in the different types of the organisation thanks to its flexibility allowing to capture the specificity of the investigated enterprises.


REFERENCES
  • Carroll, J.S., 1998. Safety culture as an ongoing process: culture surveys as opportunities for enquiry and change. Work Stress 12, 272e284.
  • Cooper, M.D., 2008, April. Risk e weighted safety culture profiling. In: Paper Presented at the Meeting of International Conference on Health, Safety, and Environment in Oil and Gas Exploration and Production, France. Retrieved from. http://www.efcog.org/wg/ism_pmi/docs/Safety_Culture/Sep08/petroriskmodel. pdf.
  • Coyle, G., 2004. Practical Strategy: Structured Tools and Techniques. Prentice Hall, Glasgow. Cullen, W.D., 1990. The Public Inquiry into the Piper Alpha Disaster. H.M.S.O, London.
  • Shaping safety culture in the organization. In: Ejdys, J. (Ed.), 2010. Kszta?towanie Kultury Bezpieczenstwa I Higieny Pracy W Organizacji. Oficyna Wydawnicza
  • Politechniki Bia?ostockiej, Bia?ystok. He, A., Xu, S., Fu, G., 2012. Study on the basic problems of safety culture. Procedia Eng. 43, 245e249.
  • Hidden, A., 1989. Investigation of the Clapham Junction Railway Accident. H.M.S.O, London.
  • International Atomic Energy Agency (IAEA, 1991. Safety Culture: a Report by the International Nuclear Safety Advisory Group (Safety Series No. 75 INSAG-4). IAEA, Vienna.
  • Janicak, C.A., 2003. Safety Metrics: Tools and Techniques for Measuring Safety Performance, second ed. Government Institutes, Lanham, MD.
  • Milczarek, M., 2000. Kultura bezpieczenstwa w przedsie?biorstwie e nowe spojrzenie na zagadnienia bezpieczenstwa pracy (Safety culture in the company e a new look at safety issues). Bezpieczenstwo Pracy 10, 17e20 [in Polish].
  • Pidgeon, N.F., 1991. Safety culture and risk management in organizations. J. Cross- Cultural Psychol. 22, 129e140.
  • Reason, J., 2011. Score your safety culture in terence. In: T, N. (Ed.), Occupational Safety and Health Culture Assessment: a Review of Main Approaches and Selected Tools. European Agency for Safety and Health at Work, Luxembourg.
  • Saaty, R.W., 1987. The analytic hierarchy process e what it is and how it is used. Mathl. Model. 9, 161e176.
  • Saaty, T.L., 1990. How to make a decision: the analytic hierarchy process. Eur. J. Op. Res. 48, 9e26.
  • Saaty, T.L., 2008. Decision making with the analytic hierarchy process. Int. J. Serv. Sci. 1, 83e98.
  • Turner, B.A., Pidgeon, N., Blockley, D., Toft, B., 1989. Safety culture: its importance in future risk management. In: Paper for the Second World Bank Workshop on Safety Control and Risk Management. Karlstad, Sweden.
  • Yule, S., 2003. Senior Management Influence on Safety Performance in the UK and US Energy Sectors (Doctoral thesis). University of Aberdeen, Scotland.

Previous Article
Investigate the human aspects of mining automation
Investigate the human aspects of mining automation

As the role of automation in the mining industry grows, researchers from The University of Queensland’s Sus...

Next Article
Drilling rig accident in underground metals mine
Drilling rig accident in underground metals mine

The NSW Resources Regulator has reported a drilling rig accident that occurred in underground metals mine. ...