Impact of Clinical Decision Support on Radiography for Acute Ankle Injuries: A Randomized Trial

Introduction While only 15–20% of patients with foot and ankle injuries presenting to urgent care centers have clinically significant fractures, most undergo radiography. We examined the impact of electronic point-of-care clinical decision support (CDS) on adherence to the Ottawa Ankle Rules (OAR), as well as use and yield of foot and ankle radiographs in patients with acute ankle injury. Methods We obtained institutional review board approval for this randomized controlled study performed April 18, 2012—December 15, 2013. All ordering providers credentialed at an urgent care affiliated with a quaternary care academic hospital were randomized to either receive or not receive CDS, based on the OAR and integrated into the physician order-entry system, with feedback at the time of imaging order. If the patient met OAR low-risk criteria, providers were advised against imaging and could either cancel the order or ignore the alert. We identified patients with foot and ankle complaints via ICD-9 billing codes and electronic health records and radiology reports reviewed for those who were eligible. Chi-square was used to compare adherence to the OAR (primary outcome), radiography utilization rate and radiography yield of foot and ankle imaging (secondary outcomes) between the intervention and control groups. Results Of 14,642 patients seen at urgent care during the study period, 613 (4.2%, representing 632 visits) presented with acute ankle injury and were eligible for application of the OAR; 374 (59.2%) of these were seen by control-group providers. In the intervention group, CDS adherence was higher for both ankle (239/258=92.6% vs. 231/374=61.8%, p=0.02) and foot radiography (209/258=81.0% vs. 238/374=63.6%; p<0.01). However, ankle radiography use was higher in the intervention group (166/258=64.3% vs. 183/374=48.9%; p<0.01), while foot radiography use (141/258=54.6% vs. 202/374=54.0%; p=0.95) was not. Radiography yield was also higher in the intervention group (26/307=8.5% vs. 18/385=4.7%; p=0.04). Conclusion Clinical decision support, previously demonstrated to improve guideline adherence for high-cost imaging, can also improve guideline adherence for radiography – as demonstrated by increased OAR adherence and increased imaging yield.


INTRODUCTION
Patients with foot and ankle pain often present to emergency departments (ED) and urgent care centers, accounting for nearly 2.8 million visits in 2010 (7.6% of all injury visits). 1 Despite this frequency, clinically significant fractures are only found in 15-20% of cases. 2 Validated, high quality, evidence-based guidelines for imaging patients with suspected ankle fracture (the Ottawa Ankle Rules [OAR]), 3,4,2 have been available for almost 20 years.However, their widespread adoption into practice has been suboptimal.In 2001, 96% of United States (U.S.) physicians reported familiarity with the rules; however, only 31% reported using them "always" or "most of the time." 5 In the same study, Canadian physicians reported using the rules 89% of the time.However, despite this, an analysis in Ontario showed that ED foot and ankle radiography still increased 1% annually from 2001-2007. 6he federal Health Information Technology and Economic Health (HITECH) Act of 2009 aims to improve quality of healthcare and reduce waste through meaningful use of health IT, including a major focus on clinical decision support (CDS). 7,815][16]17,18 Imaging CDS is most effective when based on high-quality evidence and embedded in provider workflow. 19However, most prior reports have focused on impact of CDS on "high cost" imaging (e.g., computed tomography, magnetic resonance imaging) with sparse use of a randomized controlled study design. 20Despite the emphasis on high-cost imaging, low-cost imaging examinations are the most common diagnostic imaging examinations performed in the U.S. and may be overused 21,22 , expose the patient to unnecessary ionizing radiation, 23,24 , may result in longer length of stay in the ED, 25,26 and result in incidental or ambiguous findings that lead to additional high-cost imaging. 27However, it remains unclear whether CDS will have a similar impact in low-cost as in high-cost imaging.
Thus, the purpose of this study was to assess the impact of CDS on radiography for acute ankle and foot injuries.More specifically, we evaluated the impact of a CDS tool on physician-documented adherence to the OAR in the evaluation of acute ankle injury in the urgent care setting.We chose the urgent care setting, as such centers are typically designed to handle relatively low acuity injuries (e.g., acute ankle and foot injuries).We hypothesized that such CDS, integrated into provider workflow, would increase adherence to the OAR.

Study Setting and Subjects
We obtained institutional review board approval for this Health Insurance Portability and Accountability Act-compliant randomized controlled study, performed between April 18,

Population Health Research Capsule
What do we already know about this issue?Clinical decision support (CDS) has been effective for improving the appropriateness of high-cost imaging, but its effect on low-cost imaging remains unclear.
What was the research question?What was the impact of randomized CDS on adherence to the Ottawa Ankle Rules (OAR)?
What was the major finding of the study?While adherence to the OAR increased with CDS, use of ankle radiographs was also higher in the CDS group.

How does this improve population health?
Evidence-based CDS can be successfully implemented for both low-and high-cost imaging in the ED.
2012, and December 15, 2013, at an urgent care center affiliated with a quaternary care, academic hospital.All providers (medical doctors and physician assistants) credentialed at the urgent care center, stratified by title, were randomized to either receive the CDS intervention at the time of ordering a foot or ankle radiograph (intervention group) or not (control group).Providers who began working at the urgent care center after the randomization period were excluded from the study.

Data Collection
Although providers were prospectively randomized, we collected data retrospectively.Data were captured concurrently with patient care, including in the CDS system for providers randomized to receive it.Therefore, we waited until all study data accrued and then collected it.Using International Classification of Diseases, Ninth Revision (ICD-9) codes, we queried the billing database for all unique patients presenting to the urgent care center during the study period with a discharge diagnosis code for a foot or ankle complaint (719.47,824.x, 825.x, 826.x, 829.x, 837.0, 838.0, 845.x, 924.2x, and 928.2x; see Supplemental eTable 1).A subsequent review of each patient's electronic health record (EHR) was performed using an explicit chart review data collection form.Data collected included patient age, chief complaint, mechanism of injury, presence of any of the exclusion criteria from the OAR (injury greater than 10 days Western Journal of Emergency Medicine prior to presentation, altered level of consciousness, multiple trauma, pregnancy, penetrating or open wounds, patients with isolated superficial skin complaints, and patients < 18 years old), tenderness over the malleoli or midfoot, and radiography utilization (eTable 2).We reviewed radiology reports to determine whether a fracture was noted and, if so, what type.Patients with chief complaints not pertaining to the foot or ankle were excluded.After chart review, patients with one or more OAR exclusion criterion were removed from the study cohort.We included only the first patient visit for each trauma episode; patient visits for re-assessment of the same ankle or foot injury were excluded.

Intervention
Providers were clustered into two subsets based on their titles (physician [MD] or physician assistant [PA]).In April 2012, providers in each subset were randomly assigned to the intervention or control groups using a random number generator function.Intervention group urgent care providers were exposed to CDS based on the OAR integrated into the computerized physician order entry (CPOE) system (Percipio; Medicalis, San Francisco CA).The CDS intervention consisted of four successive screens to capture data to determine the utility of the study according to the OAR (Figure 1A-D), and one educational screen (Figure 1E).If the utility of the study was low, the provider was exposed to the educational screen informing him/ her of such, and s/he could either cancel or proceed with the imaging order.
To determine whether any differences observed in adherence were simply a result of "gaming" the system, which we defined as inaccurate data entry into the CPOE system to avoid potentially onerous CDS interactions, one investigator (IKI) performed manual chart reviews of 158 randomly sampled charts in the CDS group, based on power calculation with alpha of 0.05,  power of 0.8, and confidence interval of 15%.We calculated the concordance and discordance rates between data in the physician note and data entered into the order entry screen (electronic orders consisting of the data entered in CPOE and CDS).Visits are concordant when the data in the visit note and the CPOE and/ or CDS system matched (based on adherence to the evidencebased guideline).They are discordant when data are conflicting.
If data entered in the CPOE and/or CDS system did not have corresponding entry in the physician note, we considered the visit to have incomplete documentation in the physician note rather than as discordant.

Outcome Measures
The primary outcome measure was adherence to the OAR, defined as the number of eligible patients with acute ankle and foot injuries whose workup was consistent with the OAR, whether or not they were imaged.We calculated the adherence rate for each group.
Secondary outcome measures were radiography utilization and fracture rates (radiography yield) among patients with acute ankle injuries.Ankle fracture, foot fracture, and clinically significant fracture rates were calculated.We defined clinically significant fractures as those with fracture displacement greater than 3 mm 2-4 .Radiography yield was calculated as the number of foot (or ankle) radiographs that detected a clinically significant fracture divided by the total number of foot (or ankle) radiographs in each group.We calculated radiography utilization rate as the number of foot (or ankle) radiographs performed divided by the total number of patient visits for acute ankle injury in each group.
Power calculation based on the rate of adherence to OAR estimated that a sample size of 334 patient visits (167 in each arm) was needed to detect a relative difference of 15% between the intervention and control groups (baseline adherence of 0.5 10,14 , alpha = .05,power = 0.8).

Statistical Analysis
We descriptively analyzed provider demographics.Chisquare was used to compare adherence to the OAR, radiography utilization rate and radiography yield of foot and ankle imaging between the intervention and control groups.We considered a two-tailed p-value of <0.05 statistically significant.All analyses were conducted using JMP Pro 11.0 (SAS Institute, Cary, NC).

Provider and Patient Characteristics
A total of 66 providers were randomized to either the  1).There were 22,982 total visits (14,642 unique patients) to the urgent care center during the 20-month study period.Of these, 988 patients were identified by the ICD-9 codes pertaining to acute ankle and foot injuries, representing 6.7% of all patients seen.We excluded 356 visits (1.5%) identified by ICD-9 codes as foot and ankle related: 177 visits were not associated with an ankle injury, 7 were multi-trauma, 26 were isolated injuries to the skin, 13 were referred with radiographs, 81 were injuries that happened more than 10 days previously, 44 were reassessment of the same injury, and 8 were associated with pregnant patients.After applying exclusion criteria, 613 patients (4.2% of all patients) representing 632 patient visits (2.7% of all patient visits) were clinically eligible for application of the OAR, of which 258 patient visits (40.8% of eligible visits) were seen by a provider in the intervention group.
Discordance between EHR and order requisition was 0% in the control and 1.3% in the intervention group while concordance was 48% and 10.7%, respectively.In the remaining cases, EHR documentation of discrete clinical information entered in CPOE was incomplete to unambiguously assess adherence to OAR.

Adherence to Evidence-based Guidelines
Rate of adherence to the OAR was higher for ankle radiography (92.6% vs. 61.8%,p=0.015) and foot radiography (81.0% vs. 63.6%,p<0.001) in the intervention group as compared to the control group (Table 2).

Radiography Use
Ankle radiography use was higher in the intervention group (64.3% vs. 48.9%,p<0.001), but foot radiography use (54.6% vs. 54.0%,p=0.950) was not significantly different (Table 3).Ankle and foot radiography was performed in 25.2% of patient visits in the intervention group compared to 15.8% in the controls (p<0.01.Only 6.6% of patient visits in the intervention group had no radiography, compared to 12.6% in the control group (p=0.0136)

DISCUSSION
Foot and ankle radiography represent low-cost, highvolume tests that -when used inappropriately -create waste, unnecessary radiation exposure, and likely increased lengths of stay in the ED/urgent care center. 4Similar to concerns regarding inappropriate use of high-cost imaging, if radiography imaging results are ambiguous or if incidental findings are discovered, potentially unnecessary downstream diagnostic and therapeutic procedures (with their associated costs and risks) may result.
We found that the implementation of a CDS tool at an urgent care center resulted in a significant increase in documented adherence to OAR, improving adherence to 93% for ankle and 81% for foot radiography for acute ankle injuries.Although we did not quantitatively compare the effort required to implement and sustain OAR deployment, associated data capture, and unambiguous calculation of adherence to OAR when using CDS compared to paper-based interventions, the relative ease of performing our experiment may encourage implementation of other decision rules using CDS infrastructure.Moreover, prior reports highlight the need for chart review when using paper forms to complete data extraction from the patient's chart as nearly 23% of data capture forms were incomplete. 4,28When using CDS, providers were required to complete entry of required data elements to place orders for imaging.Quality improvement strategies using CDS may thus provide near real-time measure of provider's adoption of evidence without the need for timeintensive retrospective chart review.
We also found that implementation of CDS reduced unnecessary foot radiography.Although the prevalence of clinically significant foot fractures was 2.3-fold higher in the CDS group, foot radiography use was similar to the control group, resulting in a higher foot radiography yield in the CDS group.
Our findings suggest overutilization of radiography even post CDS.Prior studies have reported approximately 16% Our results suggest that despite the existence of a wellknown, validated decision rule, use of radiography for the evaluation of acute ankle trauma in the urgent care setting is suboptimal.Moreover, we found that although implementation of CDS based on OAR resulted in modest improvement in use of radiography in these patients, radiography use was not optimized.This overuse of imaging may be due to a number of factors.Patients' preferences for imaging may have been a contributing factor when evidence-based guidelines were not followed; while patients are becoming aware of the risks of high cost high-radiation imaging, extremity radiographs carry a much less negative connotation.In addition, the OAR may have been suboptimally applied.Future studies would be needed to assess whether additional teaching, to both patients and providers, on use of OAR might reduce unnecessary utilization of radiography in patients with acute ankle injuries in urgent care centers.
The lower concordance between CDS-documented clinical attributes and the physician note found in the CDS group is expected.CDS required explicit documentation of relevant discrete clinical attributes, a capability absent in narrative documentation in the physician notes.This limitation of physician notes highlights the shortcomings of some current strategies for data collection (which rely on automated data extraction strategies from EHRs) as information may not be well documented and will thus often not be discoverable.

LIMITATIONS
There were a number of limitations to our study.We were unable to assess impact of OAR embedded in CDS on use and yield of radiography for evaluation of ankle fractures.The prevalence of significant fractures of ankle and foot differed significantly (near twofold) between control and intervention groups.Thus, our observed higher imaging yield of significant fractures, and higher use of radiography in the intervention group may simply reflect the higher prevalence of fractures in the intervention group.This in turn suggests that our randomization might not have been effective, with the intervention group consisting of providers evaluating patients with more significant fractures.An alternative explanation would be that a substantial number of significant fractures were missed in the control group, a very unlikely scenario, as 90% of the patients enrolled in the study were imaged, and we found no missed fractures re-presenting to the urgent care center.Secondly, we randomized on an intent-to-treat basis.As all clinicians were randomized based on being credentialed to practice at the urgent care, not every physician and PA may have worked there during the study period.Thirdly, our data was all obtained from a single site and thus may not be generalizable.Finally, we did not train our providers in interpreting OAR, which may have contributed to overuse of radiography.

CONCLUSION
We found that implementation of the Ottawa Ankle Rules embedded in clinical decision support significantly improved documented adherence to the OAR, and modestly improved use of foot radiography.The relative ease of implementation, data capture, and unambiguous measurement of provider adherence to OAR, without the need for time-consuming chart review, suggests CDS can efficiently deliver complex imaging-related decision rules embedded in provider workflow.Despite more than 20 years of experience with OAR, 4,28 , we found radiography likely remains overused in patients with acute ankle injury in urgent care centers.Future studies would be needed to assess whether additional training about the Ottawa Ankle Rules for providers and patients, or more stringent CDS-enabled interventions, can help reduce unnecessary radiography in these patients.
Figure 1A-E.Clinical decision support screens for the Ottawa ankle rule integrated in the computerized physician order-entry system.ED, emergency department
Address for Correspondence: Ali S. Raja, MD, MBA, MPH, Massachusetts General Hospital, Department of Emergency Medicine, 55 Fruit Street, Founders 110, Boston, MA 02114.Email: araja@mgh.harvard.edu.Conflicts of Interest:By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias.The authors disclosed none.Copyright: © 2017 Tajmir et al.This is an open access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) License.See: http://creativecommons.org/ licenses/by/4.0/

Table 1 .
Tajmir et al.Clinical Decision Support for Radiography of Acute Ankle Injuries Provider characteristics by group and total in study of efficacy of clinical decision support, based on the Ottawa Ankle Rules, to curb unnecessary foot and ankle radiography.
CDS, clinical decision support.*Providers randomized to each group who actually evaluated enrolled patients during the study.
Clinical Decision Support for Radiography of Acute Ankle InjuriesTajmir et al.

Table 2 .
-fold higher in the intervention Adherence to Ottawa Ankle Rules (OAR) and radiography use by group.

Table 3 .
Radiography combinations by group.

Table 4 .
4,28ography yield and fractures per patient visit of ankle and foot radiography by group.CDS, clinical decision support.Volume 18, no.3: April 2017 Clinical Decision Support for Radiography of Acute Ankle Injuries Tajmir et al.prevalence of significant fractures, and 4% prevalence of avulsion fractures in an ED cohort of patients with acute ankle injuries4,28.The intervention group in our urgent care had a 10% prevalence of significant fractures (with a nearly equal proportion of avulsion injuries), reflecting the diagnosis of less severe injuries compared to the ED.However, despite these