Summary
Objective
To determine whether specific design interventions (changes in the user interface (UI)) of an electronic health record (EHR) medication module are associated with an increase or decrease in the incidence of contradictions between the structured and narrative components of electronic prescriptions (internal prescription discrepancies).
Materials and Methods
We performed a retrospective analysis of 960,000 randomly selected electronic prescriptions generated in a single EHR between 01/2004 and 12/2011. Internal prescription discrepancies were identified using a validated natural language processing tool with recall of 76% and precision of 84%. A multivariable autoregressive integrated moving average (ARIMA) model was used to evaluate the effect of five UI changes in the EHR medication module on incidence of internal prescription discrepancies.
Results
Over the study period 175,725 (18.4%) prescriptions were found to have internal discrepancies. The highest rate of prescription discrepancies was observed in March 2006 (22.5%) and the lowest in March 2009 (15.0%).
Addition of „as directed“ option to the <Frequency> dropdown decreased prescription discrepancies by 195 / month (p = 0.0004). An non-interruptive alert that reminded providers to ensure that structured and narrative components did not contradict each other decreased prescription discrepancies by 145 / month (p = 0.03). Addition of a „Renew / Sign“ button to the Medication module (a negative control) did not have an effect in prescription discrepancies.
Conclusions
Several UI changes in the electronic medication module were effective in reducing the incidence of internal prescription discrepancies. Further research is needed to identify interventions that can completely eliminate this type of prescription error and their effects on patient outcomes.
Keywords: Medication errors, user interface, electronic medical records, electronic prescribing
1. Introduction
Electronic health records (EHRs) can improve healthcare in several ways, including facilitating access to patient information, providing clinical decision support, improving provider-provider and provider-patient communication and reducing health care costs [1-6]. Utilization of EHRs in the U.S. is increasing and is expected to continue to grow due to strong encouragement by recent federal legislation [7-9].
However, as any technology, changes in clinical workflow resulting from introduction of EHRs can have unforeseen consequences. Investigators have noted mistakes in data entry resulting from juxtaposition errors, entry of orders into the wrong patient’s record, mismatches between the real-life clinical workflow and the one „envisaged“ by the EHR, increased workload for clinicians, etc. [10-15].
One type of unintended consequence or medical error unique to EHRs that was recently identified is internal prescription discrepancies [16]. An internal prescription discrepancy arises when two components of a single prescription contradict each other: e.g. „1 capsule po tid“ vs. „take 2 tablets three times a day“. Most commonly these discrepancies arise between the structured (drop-downs that allow user to populate prescription with values from standard vocabularies) and free-text components of electronic prescriptions. Several studies representing geographically and socially diverse health care systems have found these discrepancies to be as common as one in twelve electronic prescriptions [16, 17].
As it is frequently unclear which of the two conflicting sets of medication instructions should be followed, internal prescription discrepancies carry a risk for patient harm if the patient uses incorrect instructions. Potential adverse drug events that could result range from inconsequential to mild to severe [16]. Combined with the apparently high prevalence of this phenomenon, these could have significant public health implications.
Currently it is not known how the internal prescription discrepancies could be prevented. We therefore conducted a retrospective analysis of a series of EHR user interface (UI) changes designed to decrease the incidence of internal prescription discrepancies that were introduced over several years to determine which of them were effective in accomplishing this task.
2. Methods
2.1 Design
We designed and validated a natural language processing tool to identify discrepancies between structured and narrative components of electronic medication prescriptions. We subsequently used this tool to conduct a retrospective analysis of the effects of a series of changes in the UI of the EHR medication module on the rate of internal prescription discrepancies over time.
2.2. Study Setting
The study was conducted at Partners HealthCare – an integrated healthcare delivery network in Eastern Massachusetts that was founded by the Brigham and Women’s Hospital (BWH) and Massachusetts General Hospital (MGH), and also includes a network of specialty and community hospitals and affiliated outpatient practices. Most outpatient clinicians at Partners HealthCare use Longitudinal Medical Record (LMR) – an ONC-ATCB (Meaningful Use) [8, 18] certified internally developed EHR. LMR includes a fully functional medication module that allows structural entry of the medication name, route, dose, strength and form, frequency, p.r.n. (pro re nata) / p.r.n. reason, amount to dispense and number of refills (▶ Figure 1). By the time the user interface changes described below were introduced, LMR was fully rolled out to all study practices and all healthcare providers in these practices were required to use LMR to generate prescriptions. The composition of practice setting / physician specialties of LMR users did not change significantly over the study period. There were no significant changes in LMR user training methodology over the course of the study period.
Fig. 1.
LMR Medication Module Interface: Current UI of the LMR medication module. Fields that have down pointing arrows (e.g. <Frequency>) are dropdowns that allow the user to either select a value from a controlled vocabulary or enter their own. The <Dose> field used to be implemented as a dropdown over most of the study period and now is represented by a series of radio buttons that allow for the same choice between a controlled vocabulary and user-entered values. The field <Special Instructions> allows users to enter free text of unlimited length.
In addition to the „basic“ medication entry interface designed for prescriptions that do not change over time, LMR also has user interfaces that allow structured entry of more complicated regimens including alternate dosing, tapers, sliding scale and in the most general form a „variable“ dosing interface. Finally, LMR also includes a narrative field called <Special Instructions> that serves to accommodate instructions that could not be expressed using available structured data entry mechanisms (e.g. „take at least 2 hours apart from iron containing medications“). However, it is possible for providers to also use the narrative field to enter information that could have been conveyed through the structured fields available in the module.
2.3 EHR User Interface Changes
Soon after its introduction, LMR designers became aware that entry of contradictory information in the structured and narrative fields of an electronic prescription could lead to internal prescription discrepancies. Consequently a series of UI changes designed to minimize the frequency of prescriptions with internal discrepancies (▶ Table 1) was introduced. The individual UI changes were developed based on the discussions between informatics and IT leadership and clinician stakeholders. Some of the changes that were anecdotally thought to have been ineffective were later reversed.
Table 1.
LMR Medication Module User Interface Changes Used in the Analysis
| Date UI change was released | Field(s) involved in the UI change | Description of the UI change |
|---|---|---|
| 7/1/2006 | Frequency | „As directed“ was added to the standard options in the dropdown for the <Frequency> field |
| 1/9/2008 | Special Instructions | When a cursor is placed into the <Special Instructions> field, a popup warning appears reminding the user that „Special Instructions should not conflict with SIG: NNN“. (Figure 2). |
| 11/7/2008 | None | „Renew / Sign“ button added to the Medication Module to direct the user to the Sign prompt immediately after renewing the prescription. This change was not expected to affect the rate of prescription discrepancies and was used as a negative control. |
| 6/8/2010 | Special Instructions | The pop-up added on 1/9/2008 was removed |
| Special Instructions | <Special Instructions> field was moved closer to the structured dose and frequency components of the prescription | |
| Special Instructions | Under certain circumstances the cursor placement defaulted to the <Special Instructions> field | |
| 11/12/2010 | Special Instructions | „As directed“ was added to the standard options in the dropdown for the <Dose> field |
| Special Instructions | The default cursor placement in the <Special Instructions> field introduced on 06/08/2010 was removed |
These changes were brought to the users’ attention through announcement emails broadcasted to all LMR users several days prior to the release as well as through „What’s New“ menu link on the application itself. Information about the upcoming changes was also given to „super-users“ in every department / practice who were encouraged to share it with their colleagues. The changes focused on two areas:
Adding „as directed“ options to the structured fields. Our initial analysis showed that many internal prescription discrepancies resulted from users who were writing complex prescriptions and accepted defaults in the structured fields but then entered a different set of instructions in the <Special Instructions> field [16]. Adding „as directed“ options was designed to allow users an easily accessible way to avoid conflict between the information in the <Special Instructions> and the structured field. While it was possible for users to manually enter „as directed“ in structured fields prior to the UI changes, users may not have been aware of that, and UI changes made this option explicit. <Special Instructions> field was available to users independent of whether an „as directed“ option was used in a structured field.
Focusing users’ attention on the information entered in the <Special Instructions> field – including alerts that warned about possible conflicts with the other fields in the prescription and moving the <Special Instructions> field closer to the main structured fields so that the conflicts become more apparent to the users. The EHR could not automatically identify electronic prescriptions with internal discrepancies and therefore alerts were displayed whenever a user entered information in the <Special Instructions> field. Consequently alerts were expected to overall have a low positive predictive value and were implemented in noninterruptive mode to minimize excessive disruption of clinical workflow [19].
2.4 Design and Validation of the Natural Language Processing Tool
We designed a rule-based natural language processing tool that identified prescriptions that were likely to contain discrepancies between the narrative <Special Instructions field> and structured fields in the prescription. The tool takes tabulated text files containing a list of prescriptions as input and identifies records that contain discrepancies of the narrative field with medication name, route, dose, dose unit, number of dose units to be taken at once, strength and form, frequency and p.r.n. (as needed) status. In its current implementation the tool does not identify the specific structured prescription field within which a discrepancy was found.
The tool was subsequently validated against a set of 1,000 randomly selected electronic prescriptions that were manually reviewed for internal discrepancies by two senior pharmacy students. Differences between the reviewers’ ratings were reconciled by a joint review. The manual ratings were then compared to the tool’s output for the same prescriptions to calculate its sensitivity (recall) and positive predictive value (precision).
2.5 Study Cohort
To analyze the effect of changes in EHR UI on the incidence of internal prescription discrepancies we randomly selected 10,000 electronic prescriptions that had narrative <Special Instructions> from each of 96 months between January 2004 and December 2011. We chose 10,000 as the number of prescriptions to be analyzed for each month as a balance between broader representation of underlying data and the demands on analytical resources, including natural language processing software. We excluded 34 most common <Special Instructions> that could not be in conflict with structured components of the prescription (e.g. „take as directed“, „take with food“, „no substitution“). We also excluded prescriptions written by providers who had written fewer than 10 electronic prescriptions prior to the end of the study period to limit the analysis to providers who had at least minimum of experience writing electronic prescriptions. No identifying patient information was included with the prescriptions.
2.6 Study Measurements
Each study prescription was processed by the natural language processing software that assigned a binary indicator of whether or not the prescription contained an internal discrepancy between the narrative and structured fields. Additionally we collected the following data elements for each prescription: a) year and month the prescription was written and b) fraction of complex prescriptions ever written for that medication. The fraction of complex prescriptions was calculated as the number of prescriptions that used one of the available structured entry formats for dosing regimens that change over time (e.g. taper, alternate, sliding scale and variable) divided by the total number of prescriptions for that medication. The purpose of this variable was to identify medications (e.g. prednisone, warfarin, insulins) where complex dosing was common and which therefore may have higher propensity to have internal discrepancies as we have previously shown [16].
2.7 Statistical Analysis
Summary statistics were calculated by using frequencies and proportions for categorical data and using means, standard deviations (SDs), medians, and ranges for continuous variables. Chi-square test was used to compare proportions. A multivariable autoregressive integrated moving average (ARIMA) model was used to perform time series analysis to establish association of individual changes in EHR UI with changes in the incidence of internal electronic prescription discrepancies. A single month during the study period served as the unit of analysis. Total number of prescriptions with internal discrepancies during the month served as the primary outcome variable. A series of UI changes in EHR system described above were considered as interventions. Average fraction of complex prescriptions was used as a covariate in the analysis based on the previous finding that complex prescriptions were more likely to have an internal discrepancy [16]. A prescription was considered complex if one of the following (structured) data entry interfaces was used: taper, alternating, variable or sliding scale. Fraction of complex prescriptions over the entire study period was calculated for each unique medication dictionary code in the EHR. The average fraction of complex prescriptions was then calculated for each study month based on the medication dictionary codes used in the study prescriptions during that month.
First order differencing was used to ensure that the process was stationary before testing for autoregressive and moving average parameters. Data were also seasonally adjusted using seasonal differencing in order to account for possible variations in medications prescribed over the course of the year (e.g. more antibiotics or glucocorticoids prescribed in the fall and winter) that could in turn affect frequency of internal prescription discrepancies. Goodness of fit statistics (e.g. Schwarz Bayesian criterion / SBC score) was used to indicate the adequate statistical fit of the model. No autocorrelations were detected in the residuals estimated from the models. First-order step function with both magnitude of slope and a single rate (decay) parameters was used to assess the effect of the changes in EHR UI on the frequency of internal prescription discrepancies. In order to control for possible effect of any UI change on the incidence of prescription discrepancies, we also included in the model a UI modification that was not designed to reduce the rate of discrepancies – addition of a combined „Renew / Sign“ button to the Medication Module in November 2008. P-values less than or equal to 0.05 were considered to be statistically significant. All statistical analyses were performed using SAS 9.3 (SAS Institute Inc. Cary, NC).
3. Results
3.1 Validation of Natural Language Processing Tool
The manual review identified a total of 201 discrepancies in 1,000 electronic prescriptions. Of these, the software identified 171 discrepancies with sensitivity of 0.76 (95% CI ± 0.03), precision of 0.84 (95% CI ± 0.03) and the corresponding F1 score of 0.798. Accuracy was higher for identification of discrepancies within complex prescriptions (sensitivity 0.87±0.03, precision 0.80±0.03, F1 score 0.83) and in the „as needed“ / „p.r.n.“ field (sensitivity 0.89±0.02, precision 0.83±0.02, F1 score 0.86). Reference resolution in the narrative field was the main source of errors.
3.2 Study Prescriptions
After exclusion of 777,455 electronic prescriptions with Special Instructions that could not have caused an internal prescription discrepancy and 114,532 electronic prescriptions written by providers who had written < 10 prescriptions prior to the end of the study period, 9,011,332 prescriptions were eligible for analysis. Out of these we randomly selected a total of 960,000 electronic prescriptions – 10,000 from every month between January 2004 and December 2010 – for the study. Complete information was available for a total of 954,446 prescriptions (9,141 to 9,997 prescriptions / month) that were included in the analysis.
Top three most common medications in the study dataset were azithromycin (4.37%), oxycodone (2.22%), and lorazepam (2.07%). The mean fraction of complex prescriptions for the medications in the study dataset was 1.06%. Among 1,759 unique medications that had at least 10 records in the dataset, azithromycin had the highest fraction of complex prescriptions at 39.8%, while 966 (54.9%) unique medications had no complex prescriptions.
Natural language processing software identified 175,725 (18.4% of the total) prescriptions with internal discrepancies. The month with the highest rate of prescription discrepancies (▶ Figure 3) was March 2006 (22.5%) and the month with the lowest rate of discrepancies was March 2009 (15.0%) – a significant difference (p < 0.0001). Among the medications with at least 10 records in the dataset, top 3 medications with the highest frequency of internal prescription discrepancies were liquid ferrous sulfate (96.8%), Neutra-Phos-K (96.7%) and butorphanol (95.0%). Fraction of records with structured complex prescriptions for a given medication correlated only weakly (Spearman correlation coefficient 0.26; p < 0.0001) with the fraction of records with internal prescription discrepancies. Fraction of prescriptions with internal discrepancies was slightly higher (p < 0.0001) in the winter (18.74%) and the fall (18.71%) compared to the spring (18.03%) and the summer (18.17%).
Fig. 3.
Fraction of prescriptions with internal discrepancies over the study period: Arrows indicate the month during which a particular change in the medication module interface was made.
3.3 Effects of Medication Module User Interface Changes on Frequency of Internal Prescription Discrepancies
After controlling for the average fraction of complex prescriptions and seasonal variation, two interface changes were found to have had a significant impact on the frequency of internal prescription discrepancies (▶ Table 2): a) adding a controlled vocabulary value of „as directed“ to the <Frequency> dropdown and b) pop-up warning about the potential of internal prescription discrepancies that appeared when the user placed the cursor into the <Special Instructions> field. Adding „as directed“ value to the <Frequency> dropdown had a delayed effect with a lag of 5 months (based on the best fitting ARIMA model). Introducing the pop-up warning about potential discrepancies had an immediate effect (no lag in the best fitting model).
Table 2.
Effects of UI Changes on Frequency of Internal Prescription Discrepancies
| UI Change | Month when UI change was released | Estimate of the UI change effect, prescriptions with internal discrepancies / month | 95% Confidence Interval | P-value |
|---|---|---|---|---|
| <Frequency> „as directed“ | 07/2006 | -195.52 | -299.3, –91.8 | 0.0004 |
| <Special Instructions> pop-up warning | 01/2008 | -145.42 | -275.8, –15.1 | 0.0319 |
| Negative control | 11/2008 | 5.92 | -104.4, 116.2 | 0.9164 |
| Pop-up warning removed Cursor defaults to <Special Instructions> | 06/2010 | -35.78 | -159.4, 87.9 | 0.5723 |
| <Dose> „as directed“ Cursor stops defaulting to <Special Instructions> | 11/2010 | - 50.1 | -186.5, 86.3 | 0.4738 |
Conditional least square estimates of the effects of each interface change calculated using an ARIMA model while adjusting for the average fraction of complex prescriptions and seasonal differences. P-values < 0.05 are highlighted.
The „negative control“ interface change (which was not related to internal discrepancies) did not have any effect on the frequency of internal discrepancies. Interface changes in 2010 had multiple changes carried out at the same time and two groups of interface changes were only 6 months apart, likely precluding the model from being able to establish the effect of individual interface changes.
4. Discussion
In this large retrospective study we showed that several UI changes, including addition of an „as directed“ option to the structured electronic medication module dropdowns and non-interruptive alerts that warned the user to ensure that information in structured and free-text components of the electronic prescription did not contradict each other, reduced the incidence of internal prescription discrepancies. To our knowledge, this is the first study that has demonstrated effective tools against this widespread problem which has a high potential for patient harm.
Design of the successful interventions was based on a combination of root cause analysis of internal prescription discrepancies and fundamental principles of decision support informatics. Based on our review of clinical workflows and interviews with clinicians, there are two main mechanisms through which internal prescription discrepancies are generated: a) prescriber accepts the defaults in the structured fields and enters the real prescription in the narrative field; and b) the initial prescription has concordant structured and narrative fields, while subsequent edits update only one but not the other, leading to a discrepancy. The first scenario frequently takes place when a prescription is complex [16] – that is, either the dose or the frequency of medication administration varies over the course of treatment. Many of these errors have high potential for adverse drug events [16]. Some EMRs (including the one investigated in this study) allow structured entry of complex prescriptions. However, this option is infrequently used, likely because it is typically time consuming and information entered in this way is seldom used to drive decision support.
In both cases the problem arises from the well-known limitations of the fundamental components of modern user interfaces. Menus – represented by the dropdowns for structured fields – aid decision-making process and utilize recognition rather than recall, thus minimizing mistakes. On the other hand, form fill-ins – represented by the narrative field – allow faster data entry for expert users and require minimal training, but can be more prone to errors [20].
As the providers’ preference for using the narrative field to enter complex prescriptions was clear, interventions aimed at reducing internal prescription discrepancies had to conform to this preference to be successful. Providing adjusted defaults / pick lists has previously been shown to be a successful clinical decision support strategy [21-23]. This approach was implemented by adding an „as directed“ option (which referred to the narrative field and therefore by definition would not be in contradiction with it) to the <Frequency> pick list, achieving a large and sustained decrease in the incidence of internal prescription discrepancies. It is worth noting that prescribers were also able to manually enter „as directed“ in the <Frequency> field prior to the intervention but may not have been aware of this; the change introduced was therefore primarily in their perception of how the medication module could be used. This is consistent with the previously published research that showed that provider perceptions that arise from the EHR interface have a significant impact on how it is used [24].
Alerting prescribers to a possible problem in a medication order is another venerated approach to decision support [25, 26]. Because it was impossible to determine whether an internal discrepancy was actually present in the electronic prescription in real time, a non-interruptive alert was chosen as an intervention. A significant decrease in the incidence of internal prescription discrepancies was observed when the alert was implemented followed by an increase when the alert was removed (although the latter change could also have been due in part to the cursor focus on the <Special Instructions> field that was introduced at the same time).
While the effect of the alert on the incidence of internal prescription discrepancies was immediate, the effect of adding „as directed“ option was delayed. This was likely due to the gradual adoption of the new option by the providers [27]. For the same reason the available dataset was likely insufficient to properly evaluate the effect of the last intervention – addition of the „as directed“ option in the <Dose> dropdown a year prior to the end of the study.
While the interventions analyzed in this study reduced the incidence of prescription discrepancies, they did not eliminate them. Further steps that could potentially increase the efficacy of the interventions include real-time identification of prescriptions with discrepancies (e.g. using natural language processing) that would enable more specific alerts to the providers, communication from clinical leaders, individual provider feedback and / or provider incentives [23, 28, 29]. Improvements in the EHR interface that facilitate structured entry of complex prescriptions could also reduce utilization of the narrative field for dosing / frequency information, and thus reduce the risk of discrepancies. Manual conversion of narrative descriptions of complex prescriptions to structured format by specially trained staff could be another, though potentially more expensive approach. Prescriber training is a critical component of successful interventions to reduce medical errors, but experience suggests that large scale training of busy clinicians can be challenging [30, 31]. Finally, more extensive medication instructions for patients (e.g. on how the medication should be taken relative to food) automatically pulled from EHR dictionaries could further minimize the need for narrative fields altogether.
This analysis investigated only a single type of error generated in EHR. Other types of errors resulting from suboptimal usability of EHR systems errors undoubtedly exist, and should be studied. Many EHR-specific errors could result in critical task failures – e.g. patients receiving incorrect medications – and are therefore an important performance metric for EHR usability [32].
At the same time, it is important that we continue to assess the effect of any intervention on actual patient outcomes. Medicine is a complex ecosystem and changes introduced into complex ecosystems can have unforeseen consequences. For example, previously published research indicates that presence of internal discrepancies in warfarin prescriptions may have actually reduced the risk of serious hemorrhage [33] – possibly due to the increased likelihood of pharmacist counseling when an internal prescription discrepancy is noted.
Our study has several strengths. It is the largest-to-date study of internal prescription discrepancies, enabling detection of the intervention effects in presence of significant random and seasonal variation. This was made possible by utilization of a validated natural language processing tool to analyze the prescriptions. While previously published investigations relied on manual review and were thus limited to several thousand prescriptions at the most, this study was able to analyze nearly a million prescriptions. This also enabled us to analyze prescriptions over the period of 8 years, making it unlikely that the effects we identified represented a secular trend. Finally, our analysis included a negative control – a change in the UI that was not relevant to prescription discrepancies and which, as expected, had no effect.
The results of the study have to be interpreted in the light of its limitations. Natural language processing tool used in the analysis was not perfectly accurate. However, unless the errors introduced by the tool were not random with respect to the relationship between the interface changes and prescription discrepancies, they should have had little effect on the findings of the study. The study was focused on the impact of UI changes on internal prescription discrepancies; further research is needed to study the effect of user training and other organizational changes. The study was retrospective in nature and did not include a control group; therefore it could not unequivocally establish a causal relationship between the EHR UI modifications and subsequent changes in the incidence of internal prescription discrepancies. We did not have sufficient information on several potential confounders, including provider specialty, practice setting and the length of their experience with the EHR. Finally, there may not have been sufficient follow-up time after the last interface change (addition of „as directed“ to the <Dose> dropdown) to fully estimate its effect on prescription discrepancies.
5. Conclusions
In summary, this large study has found that addition of an „as directed“ option to the structured component of the EHR prescription module and reminding users to ensure that the structured and narrative components do not contradict each other decreased the incidence of internal electronic prescription discrepancies. Further research is needed to identify interventions that could completely eliminate prescription discrepancies and their effect on patient outcomes.
Fig. 2.
Pop-up warning about potential for internal prescription discrepancies: The pop-up warning (in the red frame, indicated by the arrow) appeared whenever the cursor was placed in the <Special Instructions> field.
Acknowledgements
This study was funded in part by the Partners HealthCare Information Systems Research Council which did not play any role in study design, collection, analysis and interpretation of data, in the writing of the report, and in the decision to submit the article for publication.
Footnotes
Clinical Relevance Statement
The study presented in this manuscript identified changes in EHR user interface that decreased the incidence of errors in electronic medication prescriptions. Preventing errors in prescriptions is critical for patient safety and may also improve efficiency of clinical workflow by reducing, for example, phone calls between pharmacists and physicians.
Conflict of Interest
None.
Human Subject Protections
This Partners HealthCare System institutional review board approved this study, and the requirement for written informed consent was waived.
References
- 1.Adler-Milstein J, Salzberg C, Franz C, Orav EJ, Newhouse JP, Bates DW.Effect of electronic health records on health care costs: Longitudinal comparative evidence from community practices. Ann Intern Med. 2013; 159: 97–104 [DOI] [PubMed] [Google Scholar]
- 2.DesRoches CM, Campbell EG, Rao SR, Donelan K, Ferris TG, Jha A, Kaushal R, Levy DE, Rosenbaum S, Shields AE, Blumenthal D.Electronic health records in ambulatory care--a national survey of physicians. N Engl J Med. 2008; 359: 50–60 [DOI] [PubMed] [Google Scholar]
- 3.Wang SJ, Middleton B, Prosser LA, Bardon CG, Spurr CD, Carchidi PJ, Kittler AF, Goldszer RC, Fairchild DG, Sussman AJ, Kuperman GJ, Bates DW.A cost-benefit analysis of electronic medical records in primary care. Am J Med. 2003; 114: 397–403 [DOI] [PubMed] [Google Scholar]
- 4.Ross SE, Moore LA, Earnest MA, Wittevrongel L, Lin C-T. Providing a web-based online medical record with electronic communication capabilities to patients with congestive heart failure: Randomized trial. Journal of medical Internet research 2004; 6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Hillestad R, Bigelow J, Bower A, Girosi F, Meili R, Scoville R, Taylor R.Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Affairs. 2005; 24: 1103–1117 [DOI] [PubMed] [Google Scholar]
- 6.Bates DW, Ebell M, Gotlieb E, Zapp J, Mullins H.A proposal for electronic medical records in us primary care. Journal of the American Medical Informatics Association. 2003; 10: 1–10 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bauer JC., Arra, hitech A medical economist’s analysis. J Healthc Inf Manag. 2010; 24: 4–5 [PubMed] [Google Scholar]
- 8.Blumenthal D, Tavenner M.The „meaningful use“ regulation for electronic health records. N Engl J Med. 2010; 363: 501–504 [DOI] [PubMed] [Google Scholar]
- 9.Blumenthal D.Launching hitech. N Engl J Med. 2010; 362: 382–385 [DOI] [PubMed] [Google Scholar]
- 10.Ash JS, Berg M, Coiera E.Some unintended consequences of information technology in health care: The nature of patient care information system-related errors. J Am Med Inform Assoc. 2004; 11: 104–112 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH.Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc. 2006; 13: 547–556 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Yackel TR, Embi PJ.Unintended errors with ehr-based result management: A case series. J Am Med Inform Assoc. 2010; 17: 104–107 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Embi PJ, Yackel TR, Logan JR, Bowen JL, Cooney TG, Gorman PN.Impacts of computerized physician documentation in a teaching hospital: Perceptions of faculty and resident physicians. Journal of the American Medical Informatics Association. 2004; 11: 300–309 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Magrabi F, Ong M-S, Runciman W, Coiera E.An analysis of computer-related patient safety incidents to inform the development of a classification. Journal of the American Medical Informatics Association. 2010; 17: 663–670 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Sittig DF, Singh H.Electronic health records and national patient-safety goals. New England Journal of Medicine. 2012; 367: 1854–1860 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Palchuk MB, Fang EA, Cygielnik JM, Labreche M, Shubina M, Ramelson HZ, Hamann C, Broverman C, Einbinder JS, Turchin A.An unintended consequence of electronic prescriptions: Prevalence and impact of internal discrepancies. J Am Med Inform Assoc. 2010; 17: 472–476 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Singh H, Mani S, Espadas D, Petersen N, Franklin V, Petersen LA.Prescription errors and outcomes related to inconsistent information transmitted through computerized order entry: A prospective study. Arch Intern Med. 2009; 169: 982–989 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Marcotte L, Seidman J, Trudel K, Berwick DM, Blumenthal D, Mostashari F, Jain SH.Achieving meaningful use of health information technology: A guide for physicians to the ehr incentive programs. Archives of internal medicine. 2012; 172: 731–736 [DOI] [PubMed] [Google Scholar]
- 19.Shah NR, Seger AC, Seger DL, Fiskio JM, Kuperman GJ, Blumenfeld B, Recklet EG, Bates DW, Gandhi TK.Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc. 2006; 13: 5–11 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Galitz WO: The essential guide to user interface design: An introduction to gui design principles and techniques. John Wiley & Sons, 2007 [Google Scholar]
- 21.Chertow GM, Lee J, Kuperman GJ, Burdick E, Horsky J, Seger DL, Lee R, Mekala A, Song J, Komaroff AL, Bates DW.Guided medication dosing for inpatients with renal insufficiency. Jama. 2001; 286: 2839–2844 [DOI] [PubMed] [Google Scholar]
- 22.Peterson JF, Kuperman GJ, Shek C, Patel M, Avorn J, Bates DW.Guided prescription of psychotropic medications for geriatric inpatients. Arch Intern Med. 2005; 165: 802–807 [DOI] [PubMed] [Google Scholar]
- 23.Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, Spurr C, Khorasani R, Tanasijevic M, Middleton B.Ten commandments for effective clinical decision support: Making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003; 10: 523–530 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Skentzos S, Shubina M, Plutzky J, Turchin A.Structured vs. Unstructured: Factors affecting adverse drug reaction documentation in an emr repository. Proc AMIA Symp 2011: 1270–1279 [PMC free article] [PubMed] [Google Scholar]
- 25.Schedlbauer A, Prasad V, Mulvaney C, Phansalkar S, Stanton W, Bates DW, Avery AJ.What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior? J Am Med Inform Assoc. 2009; 16: 531–538 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Seidling HM, Phansalkar S, Seger DL, Paterno MD, Shaykevich S, Haefeli WE, Bates DW.Factors influencing alert acceptance: A novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc. 2011; 18: 479–484 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Palchuk MB, Turchin A, Alexeyev A, Galperin I, Hamann C.Reducing unintended consequences of e-prescribing on the path to nuanced prescriptions. AMIA Annu Symp Proc 2008: 1079. [PubMed] [Google Scholar]
- 28.Brennan N, Mattick K.A systematic review of educational interventions to change behaviour of prescribers in hospital settings, with a particular emphasis on new prescribers. British Journal of Clinical Pharmacology. 2013; 75: 359–372 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Flodgren G, Eccles MP, Shepperd S, Scott A, Parmelli E, Beyer FR.An overview of reviews evaluating the effectiveness of financial incentives in changing healthcare professional behaviours and patient outcomes. Cochrane Database Syst Rev 2011; 7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Times LT.Usability of electronic medical records. Journal of usability studies. 2009; 4: 70–84 [Google Scholar]
- 31.Horsky J, Kuperman GJ, Patel VL.Comprehensive analysis of a medication dosing error related to cpoe. Journal of the American Medical Informatics Association. 2005; 12: 377–382 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Albert W, Tullis T.Measuring the user experience: Collecting, analyzing, and presenting usability metrics. Newnes, 2013 [Google Scholar]
- 33.Turchin A, Shubina M, Goldberg S.Unexpected effects of unintended consequences: Emr prescription discrepancies and hemorrhage in patients on warfarin. AMIA Annu Symp Proc. 2011; 2011: 1412–1417 [PMC free article] [PubMed] [Google Scholar]