How Can We Make Disaster Management Evaluations More Useful? An Empirical Study of Dutch Exercise Evaluations

Ralf Josef Johanna Beerens; Henrik Tehler; Ben Pelzer

doi:10.1007/s13753-020-00286-7

Volume 11 Issue 5

Dec. 2020

Turn off MathJax

Article Contents

Article Navigation > International Journal of Disaster Risk Science > 2020 > 11(5): 578-591

Ralf Josef Johanna Beerens, Henrik Tehler, Ben Pelzer. How Can We Make Disaster Management Evaluations More Useful? An Empirical Study of Dutch Exercise Evaluations[J]. International Journal of Disaster Risk Science, 2020, 11(5): 578-591. doi: 10.1007/s13753-020-00286-7

Citation:

Ralf Josef Johanna Beerens, Henrik Tehler, Ben Pelzer. How Can We Make Disaster Management Evaluations More Useful? An Empirical Study of Dutch Exercise Evaluations[J]. International Journal of Disaster Risk Science, 2020, 11(5): 578-591. doi: 10.1007/s13753-020-00286-7

Citation:

PDF

How Can We Make Disaster Management Evaluations More Useful? An Empirical Study of Dutch Exercise Evaluations

doi: 10.1007/s13753-020-00286-7

1 Institute for Safety (IFV), 6801 HA Arnhem, The Netherlands;
2 Division of Risk Management and Societal Safety, Lund University, 221 00 Lund, Sweden;
3 Lund University Centre for Risk Assessment and Management (LUCRAM), Lund University, 221 00 Lund, Sweden;
4 Department of Sociology, Radboud University Nijmegen, 6500 HE Nijmegen, The Netherlands

Available Online: 2021-04-26
Publish Date: 2020-12-01

Abstract

Abstract

The evaluation of simulated disasters (for example, exercises) and real responses are important activities. However, little attention has been paid to how reports documenting such events should be written. A key issue is how to make them as useful as possible to professionals working in disaster risk management. Here, we focus on three aspects of a written evaluation: how the object of the evaluation is described, how the analysis is described, and how the conclusions are described. This empirical experiment, based on real evaluation documents, asked 84 Dutch mayors and crisis management professionals to evaluate the perceived usefulness of the three aspects noted above. The results showed that how evaluations are written does matter. Specifically, the usefulness of an evaluation intended for learning purposes is improved when its analysis and conclusions are clearer. In contrast, evaluations used for accountability purposes are only improved by the clarity of the conclusion. These findings have implications for the way disaster management evaluations should be documented.
- Disaster management evaluation,
- Evaluation design,
- Evaluation report,
- Exercise evaluation,
- The Netherlands,
- Usefulness

FullText(HTML)

References(55)

References

Abrahamsson, M., H. Hassel, and H. Tehler. 2010. Towards a system-oriented framework for analysing and evaluating emergency response. Journal of Contingencies and Crisis Management 18(1): 14–25.

Atzmüller, C., and P.M. Steiner. 2010. Experimental vignette studies in survey research. Methodology 6(3): 128–138.

Auspurg, K., and T. Hinz. 2015. Factorial survey experiments. Thousand Oaks, CA: Sage.

Aven, T. 2010. On how to define, understand and describe risk. Reliability Engineering and System Safety 95(6): 623–631.

Beerens, R.J.J. 2019. Does the means achieve an end—A document analysis providing an overview of emergency and crisis management evaluation practice in the Netherlands. International Journal of Emergency Management 15(3): Article 221.

Beerens, R.J.J., and H. Tehler. 2016. Scoping the field of disaster exercise evaluation—A literature overview and analysis. International Journal of Disaster Risk Reduction 19: 413–446.

Biddinger, P.D., R. Orfaly Cadigan, B.S. Auerbach, J.L. Burstein, E. Savoia, M.A. Stoto, and H.K. Koh. 2008. On linkages: Using exercises to identify systems-level preparedness challenges. Public Health Reports 123(1): 96–101.

Birkland, T.A. 2009. Disasters, lessons learned, and fantasy documents. Journal of Contingencies and Crisis Management 17(3): 146–156.

Boin, A., P. ‘t Hart, E. Stern, and B. Sundelius. 2017. The politics of crisis management, 2nd edn. Cambridge: Cambridge University Press.

Borell, J., and K. Eriksson. 2008. Improving emergency response capability: An approach for strengthening learning from emergency response evaluations. International Journal of Emergency Management 5(3–4): Article 324.

Borodzicz, E., and K. Van Haperen. 2002. Individual and group learning in crisis simulations. Journal of Contingencies and Crisis Management 10(3): 139–147.

Bovens, M., P. ‘t Hart, and S. Kuipers. 2008. The politics of policy evaluation. In The Oxford handbook of public policy, ed. M. Moran, M. Rein, and R.E. Goddin, 317–333. Oxford: Oxford University Press.

Bradshaw, C.C., and T.A. Bartenfeld. 2009. Exercise evaluation guides for public health emergency preparedness. Homeland Security Affairs Journal 5(3): 1–41.

Calidoni-Lundberg, F. 2006. Evaluation: Definitions, methods and models: An ITPS framework. Östersund: Swedish Institute for Growth Policy Studies. https://pdfs.semanticscholar.org/6847/9ec9fc396e1a3f1ab859502e2e0cb0267d41.pdf?_ga=2.138291760.1448429144.1592574523-1144041233.1589189182. Accessed 15 Jun 2020.

Davis, F.D. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13(3): Article 319.

Duarte da Costa, A., J.O. Gomes, M.R. da Silva Borges, and P.V. Rodrigues de Carvalho. 2013. ASC model: A process model for the evaluation of simulated field exercises in the emergency domain. In Proceedings of the 10th International Conference on Information Systems for Crisis Response and Management (ISCRAM), ed. T. Comes, F. Fiedrich, S. Fortier, J. Geldermann, and T. Müller, 551–555. Baden-Baden: Karlsruher Institut fur Technologie (KIT).

Etkin, J., and A. Sela. 2016. How experience variety shapes postpurchase product evaluation. Journal of Marketing Research 53(1): 77–90.

Frye, A.W., and P.A. Hemmer. 2012. Program evaluation models and related theories: AMEE guide No. 67. Medical Teacher 34(5): 288–299.

Gebbie, K.M., J. Valas, J. Merrill, and S. Morse. 2006. Role of exercises and drills in the evaluation of public health in emergency response. Prehospital and Disaster Medicine 21(3): 173–182.

Hartley, J., E. Sotto, and C. Fox. 2004. Clarity across the disciplines. Science Communication 26(2): 188–210.

Heath, R. 1998. Looking for answers: Suggestions for improving how we evaluate crisis management. Safety Science 30(1–2): 151–163.

Hertting, N., and E. Vedung. 2012. Purposes and criteria in network governance evaluation: How far does standard evaluation vocabulary takes us? Evaluation 18(1): 27–46.

Heumüller, E., S. Richter, and U. Lechner. 2012. Towards a framework for command post exercises. In Proceedings of the 9th International Conference on Information Systems for Crisis Response and Management (ISCRAM), ed. L. Rothkrantz, J. Ristvej, and Z. Franco, 1–6. Vancouver: Simon Fraser University.

Hox, J. 2002. Multilevel analysis: Techniques and applications. Mahmah, NJ: Lawrence Erlbaum Associates Inc.

IBM Corp. 2019. SPSS. Armonk, NY: IBM Corp.

Imergis.nl. 2020. Map of 25 safety regions (Fysieke Veiligheid—25 Veiligheidsregio’s 2020). Zeist: Imergis Organisatiebloei. https://www.imergis.nl/map/2020/2020-Veiligheidsregio-1200.png. Accessed 25 Jun 2020 (in Dutch).

Instituut Fysieke Veiligheid. 2017. Learning and the Fire Services: About how people learn—Also within the Fire Services (Leren and brandweer: Over hoe mensen leren—ook binnen de brandweer). Arnhem: Instituut Fysieke Veiligheid. https://www.ifv.nl/kennisplein/Documents/20170316-BA-Leren-en-brandweer.pdf. Accessed 19 Jun 2020 (in Dutch).

ISO (International Organization of Standardization). 2013. International standard, societal security—Guidelines for exercises ISO-22398-2013. Geneva: ISO.

Jasso, G. 2006. Factorial survey methods for studying beliefs and judgments. Sociological Methods and Research 34(3): 334–423.

Klein, K.R., D.C. Brandenburg, J.G. Atas, and A. Maher. 2005. The use of trained observers as an evaluation tool for a multi-hospital bioterrorism exercise. Prehospital and Disaster Medicine 20(3): 159–163.

Lin, L., A. Nilsson, J. Sjölin, M. Abrahamsson, and H. Tehler. 2015. On the perceived usefulness of risk descriptions for decision-making in disaster risk management. Reliability Engineering and System Safety 142: 48–55.

Lin, L., C. Rivera, M. Abrahamsson, and H. Tehler. 2017. Communicating risk in disaster risk management systems—Experimental evidence of the perceived usefulness of risk descriptions. Journal of Risk Research 20(12): 1534–1553.

Lincoln, Y.S., and E.G. Guba. 1980. The distinction between merit and worth in evaluation. Educational Evaluation and Policy Analysis 2(4): 61–71.

Månsson, P., M. Abrahamsson, and H. Tehler. 2019. Aggregated risk: An experimental study on combining different ways of presenting risk information. Journal of Risk Research 22(4): 497–512.

Nohrstedt, D., F. Bynander, C. Parker, and P. ‘t Hart. 2018. Managing crises collaboratively: Prospects and problems—A systematic literature review. Perspectives on Public Management and Governance 1(4): 257–271.

Perry, R.W. 2004. Disaster exercise outcomes for professional emergency personnel and citizen volunteers. Journal of Contingencies and Crisis Management 12(2): 64–75.

Peterson, D.M., and R.W. Perry. 1999. The impacts of disaster exercises on participants. Disaster Prevention and Management: An International Journal 8(4): 241–255.

Qualtrics. 2019. Qualtrics. Provo, UT: Qualtrics. https://www.qualtrics.com. Accessed 15 Jun 2020.

Rathjens, D. 1985. The seven components of clarity in technical writing. IEEE Transactions on Professional Communication PC-28(4): 42–46.

Raudenbush, S.W., and A.S. Bryk. 2002. Hierarchical linear models: Applications and data analysis methods, 2nd edn. Newbury Park, CA: Sage.

Savoia, E., F. Agboola, and P. Biddinger. 2014. A conceptual framework to measure systems’ performance during emergency preparedness exercises. International Journal of Environmental Research and Public Health 11(9): 9712–9722.

Savoia, E., L. Lin, and G.M. Gamhewage. 2017. A conceptual framework for the evaluation of emergency risk communications. American Journal of Public Health 107(S2): 208–214.

Scriven, M. 1980. The logic of evaluation. Inverness, CA: Edgepress.

Scriven, M. 1991. Evaluation thesaurus. Beverly Hills, CA: Sage.

Sinclair, H., E.E. Doyle, D.M. Johnston, and D. Paton. 2012. Assessing emergency management training and exercises. Disaster Prevention and Management: An International Journal 21(4): 507–521.

Skryabina, E., G. Reedy, R. Amlôt, P. Jaye, and P. Riley. 2017. What is the value of health emergency preparedness exercises? A scoping review study. International Journal of Disaster Risk Reduction 21: 274–283.

Suchan, J., and R. Dulek. 1990. A reassessment of clarity in written managerial communications. Management Communication Quarterly 4(1): 87–99.

Swedish Civil Contingencies Agency. 2011. Handbook evaluation of exercises. Stockholm: Swedish Civil Contingencies Agency. https://www.msb.se/RibData/Filer/pdf/25885.pdf. Accessed 13 Jun 2020.

Thomas, T.L., E.B. Hsu, H.K. Kim, S. Colli, G. Arana, and G.B. Green. 2005. The incident command system in disasters: Evaluation methods for a hospital-based exercise. Prehospital and Disaster Medicine 20(1): 14–23.

Uhr, C., H. Johansson, and L. Fredholm. 2008. Analysing emergency response systems. Journal of Contingencies and Crisis Management 16(2): 80–90.

UK Cabinet Office. 1998. The exercise planners guide. London: UK Cabinet Office. http://www.cabinetoffice.gov.uk/resource-library/exercise-planners-guide. Accessed 19 Jun 2020.

UNDRR (United Nations Office for Disaster Risk Reduction). 2020. Words into action—Design and conduct of simulation exercises—SIMEX. Geneva: UNDRR.

U.S. Departement of Homeland Security. 2020. Homeland Security Exercise and Evaluation Program (HSEEP). Washington, DC: U.S. Departement of Homeland Security.

van Duin, M., and V. Wijkhuijs. 2015. The flexibility of GRIP (De flexibiliteit van GRIP). https://www.ifv.nl/advieseninnovatie/Documents/201504-IFV-De-flexibiliteit-van-GRIP.pdf. Accessed 15 Jun 2020 (in Dutch).

Wybo, J.-L. 2008. The role of simulation exercises in the assessment of robustness and resilience of private or public organizations. In Resilience of cities to terrorist and other threats, ed. H.J. Pasman, and I.A. Kirillov, 491–507. New York: Springer.

Relative Articles

Supplements(0)

Cited By

Proportional views