Evidence-Based Risk Management: Changing the Zeitgeist
January 17, 2012
Matthew D. Griffith, M.S., RCRSP
Georgia Institute of Technology
In the past two decades, a new way of thinking has taken over the practice of medicine. The central premise is that decisions in medical care should be based on the latest and best scientific knowledge. Dr. David Sackett and colleagues define evidence-based medicine as “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients” (1996). The movement has now grown into a storm of physicians committed to conducting, disseminating, and applying valid and reliable research to clinical care.
If you found yourself thinking “what else besides evidence would guide my doctor’s decisions?” then you are naïve about how humans generally make decisions. Although there are thousands of medical studies conducted each year, physicians don’t use much of it. In How Doctors Think, Dr. Jerome Groopman references research that show only about 15% of physician’s decisions are based on sound evidence (2008). Instead of evidence, doctors more often rely on obsolete knowledge gained in medical school, long-standing but never proven traditions, patterns gathered from experience, methods they are most skilled in applying, and information from vendors with products to sell.
Unfortunately, the same behavior holds true for risk managers in recreation and sport. In fact, it could be argued, that risk managers are far more ignorant than doctors about which prescriptions are reliable–and less willing to find out. It’s time to move the zeitgeist away from making decisions based on ideology and unsystematic experience toward those based on the best available evidence.
Admittedly, the challenge of evidence-based practice in recreational risk management is greater than in medicine and some other disciplines. There is a lack of rigorous research, no formal education for so-called experts, and laws vary greatly by jurisdiction. In addition, risk managers face a vexing problem: facilities and programs have huge variations on several important dimensions. Nonetheless, if risk managers act on the best available evidence, they will better serve their users and agency.
The Evidence-Practice Gap
The concept of applying risk management practices to recreation began in the mid-1970s, but didn’t really take off in practice until the early-1980s. In the intervening three decades, significant efforts have been made to increase participant safety and reduce legal liability. Unfortunately, though, the fear of lawsuits often controls the services offered by many agencies, rather than the desirability of the services to the users. This type of “risk management” is not always based on sound evidence. Recreation practitioners continue to struggle with identifying actual risks and applying proven risk management strategies. Evidence-based risk management (EBRM) provides the needed model to guide the closing of this gap.
There are several reasons the evidence-practice gap persists in recreational risk management. The most predominate is the emphasis our profession places on practical experience. Practitioners oftentimes neglect to seek out new evidence because they trust their own experience. This is naturally human because information acquired firsthand often feels more real than words in a research journal or law review. The problem is twofold: small sample sizes and the inherent biases that characterize personal observation. The truth is that because most risks will never lead to an injury or other loss, risk managers do not have adequate personal experience identifying hazards and testing various prevention and intervention strategies.
Another primary reason for the evidence-practice gap is that numerous risk management decisions made in recreation departments are driven by dogma and belief. Many people are overly influenced by ideology and fail to question practices that fit with what they “know.” A recent study conducted by John Miller and colleagues showed that 60% of intramural sport directors did not believe waivers would protect the program or organization from legal action (2009). This is an all too common misunderstanding. The truth is that when used in the correct circumstances, well-written waivers protect the provider from liability and negligence in at least 45 states (Cotten & Cotten, 2010). Beliefs such as this, and others rooted in ideology and dogma are (to borrow the term from Chip and Dan Heath’s book Made to Stick: Why Some Ideas Survive and Others Die) “sticky” and resist disconfirmation, regardless of whether they are true.
The final reason for the persistent gap is uncritical imitation and its equivalent, casual benchmarking. Recreation professionals often look to the perceived top programs and facilities and try to emulate their practices. Although benchmarking can be a useful and cost-efficient tool, in general, it is not a source of credible evidence. Benchmarking can actually be harmful to an organization when used casually, meaning that the evidence behind what works, why it works, and whether it will work elsewhere is barely unraveled. Consider an extreme example. A large, prominent university in the south closes when a light blanket of snow accumulates on the ground. If a university in Colorado were to try to mimic this practice, they might be closed more days of the semester than they are open.
It should now be apparent why evidence-based decision making in risk management is so rare among recreation practitioners. It should also be clear why relying on any of the practices described above is not the best way to make decisions to manage risk. Adopting an evidence-based risk management approach requires a distinct mind-set. Practitioners must exhibit a willingness to set aside personal belief and conventional wisdom, and replace these with a relentless pursuit of the necessary facts to make informed and intelligent decisions.
Becoming an Evidence-Based Risk Manager
To develop an organization of evidence-based risk management, department leaders and risk managers must nurture an evidence-based approach by setting the tone. The first step is to demand evidence of efficacy for every change proposed and decision made. For example, signage is one of the areas of risk management that scientific research can tell us a lot about. (Keep in mind that of the various levels of evidence within the recreational risk management context, experimental research studies are on the upper end of the evidence spectrum). There are countless articles in psychology and human factors journals regarding efficacy of various warnings and instructions. In the United States, the American National Standards Institute has published standards and guidelines based on this research as well as legal precedent. The standard includes the proper use of signal words, layout, content, color, symbols/text, and more. Adopting this standard for safety signs within recreational facilities would lead to reduced liability and improved safety, but very few, if any, departments have implemented this research based signage on a large scale.
Simply requesting evidence is insufficient though, the evidence must also be applicable. Therefore, decision makers must look for gaps in logic and inference and generally think critically when presented with proposed changes. Time must be spent to figure out and understand the underlying assumptions that form the foundation for a proposed policy, practice, or intervention. This is particularly important when gaining advice from consultants or other professionals. It is surprising how often readers of industry publications or conference attendees are fooled by the “expert” author or presenter. Most of the risk management practices promoted have little evidence to support their widespread adoption. The widespread practice of closing indoor pools during lightning storms is one such case of a “risk management” practice that has been heavily promoted with no applicable evidence. To date, no reports of death from lightning at an indoor pool could be found. This is in spite of the fact that hundreds of pools are kept open every day during thunderstorms across the country. I co-authored a thorough article on this topic with Dr. Griffiths in the November/December 2008 issue of Aquatics International. We concluded that as long as specific design and engineering controls are in place, people swimming in an indoor pool during a thunderstorm are as safe as they can be (Griffith & Griffiths, 2008).
For some decisions in risk management, especially those lacking external evidence, the evidence must be found within the organization. In order to develop a strong internal evidence base, leaders must encourage experimentation, pilot studies, trial programs, and data collection. One barrier often cited for not experimenting is that the department adopts practices in an all-or-nothing way. This limits the ability to learn through trial and error. Looking at an example of implementing a surprise emergency drill program for intramural officials but not in other program areas shows how this trial might lead to interesting and applicable results when the intramural employees’ emergency response is compared to desk attendants who were not subject to the drills.
One interesting experiment that many aquatic facilities undertake, many times without even realizing it, is when they allow the public or groups to use diving boards and platforms for certain special events even though they are normally closed to the public. If the facility can open the 3-meter diving board and operate it safely for a few days out of the year, it begs the question, why can’t it be opened safely the rest of the year?
Using actual data collected on injuries and rates is also an important part of the internal evidence base. One all-too-common error is to mistake uncertainty for risk. The recent case of five-toe shoes shows how easily some departments implement rules without strong evidence (external or internal). In order to implement a rule such as banning toe shoes, an evidence-based risk manager must demand the evidence and interpret whether the data shows a strong cause-and-effect relationship between wearing the shoes and injury. Since this evidence does not exist externally and is unlikely to exist internally, the proposed implementation of the rule should be discarded.
Finally, leaders and risk managers must embrace what has been called the “attitude of wisdom.” For thousands of years, people have appreciated that wisdom does not come from accumulating knowledge, but from a respect for the vast amount of knowledge still unconquered. EBRM is conducted best not by know-it-alls but by people who appreciate how much they do not know. For this reason, it is important to continue professional education and identify and apply strategies for lifelong learning. Even when sound evidence is lacking, evidence-based risk managers should inquire and find other ways to gather data.
There will inevitably be critics to the evidence-based risk management I propose here, just as there are in medicine and other evidence-based disciplines. Although the theory does need to be rigorously tested, the logical argument is watertight. Decisions made on the basis of evidence will be better decisions overall. However, be careful not to misapply EBRM: it is not a one-size-fits-all approach. EBRM requires the use of the best available evidence coupled with informed judgment and local context. The promise of EBRM is to break down the “sticky” dogma and status quo for improved results. Like the QWERTY keyboard created for manual typewriters but inefficient for word processing, common risk management practices survive, despite being ineffective and out-of-line with contemporary recreation and sport. Failure to shift the zeitgeist toward evidence-based risk management, however, will have much greater costs than mere inefficiency.
References
Cotten, D. J. & Cotten, M. B. (2010). Waivers & Releases of Liability (7th ed.). Statesboro, GA: Sport Risk Consulting.
Griffith, M., & Griffiths, T. (2008, November/December). When lightning strikes. Aquatics International, 20(10), 18-21.
Groopman, J. (2008). How doctors think. Boston, MA: Houghton Mifflin Harcourt.
Miller, J. J., Young, S. J., & Martin, N. (2009). To use or not to use? The status of waivers in intramural sports. Recreational Sports Journal, 33(2), 129-138.
Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: What it is and what it isn’t. British Medical Journal, 312, 71-72.