Primary Care Practices’ Abilities And Challenges In Using Electronic Health Record Data For Quality Improvement

Since 2008, adoption of office-based physician electronic health records (EHRs) has more than doubled.¹ Federal investment played a critical role in accelerating EHR adoption through a combination of financial incentives (the EHR Incentive Program) and technical assistance programs (Regional Extension Centers).^2–6 The expectation was that widespread adoption of EHRs would efficiently generate meaningful data, enabling accurate measurement of quality, informing practice quality improvement efforts, and ultimately leading to improved care processes and outcomes. Yet little is known about how well EHRs meet these expectations, particularly among primary care practices with scarce technical resources.^7–11

The EHR Incentive Program set standards for the meaningful use of EHRs, which included implementing an EHR system and demonstrating its use to improve care. There were seventeen core standards defined in stages 1 and 2 of the meaningful-use program (2015–17). Stage 3 began in 2017 and expanded the requirements to include health information exchange, interoperability, and advanced quality measurement to maximize clinical effectiveness and efficiency by supporting quality improvement. As of 2017 the EHR Incentive Program defined sixty-four electronic clinical quality measures¹² that are aligned with national quality standards. The rationale behind using these measures was to reduce the need for clinicians’ involvement in reporting by using data already collected within the EHR and automating the electronic submission of results.

Quality measurement for payment grew with the 2006 implementation of the Physician Quality Reporting System, as an increasing number of clinicians and practices reported their quality data electronically. In 2016 the Quality Payment Program⁶ was developed as a way to streamline quality reporting programs while expanding the expectations of electronic reporting as defined by the Merit-based Incentive Payment Program. A core expectation of meaningful use and the subsequent Quality Payment Program was for EHRs to have the capability to measure and report electronic clinical quality measures and for practices to use these data to improve quality. To that end, the Office of the National Coordinator for Health Information Technology (ONC) worked with the Centers for Medicare and Medicaid Services (CMS) and stakeholders to establish a set of certification criteria for EHRs. Use of an ONC-certified EHR was a core requirement of meaningful use. The functionality of certified EHR systems’ reporting of electronic clinical quality measures was aligned with CMS-based incentives and quality criteria; it is anticipated that these quality-based incentives will continue as meaningful use evolves into the Quality Payment Program.

Clinicians participating in the Quality Payment Program were required to report on the full 2017 performance period by March 31, 2018. In addition to meeting external reporting requirements, EHRs must help practices identify delivery gaps and “bright spots” of performance^13,14 that are critical for quality improvement. This requires the ability to produce patient-, clinician-, and practice-level reports across various measurement periods and at different frequencies and to allow for customized specifications to conduct improvement cycles.¹⁵ EHR systems often fail to meet these expectations, but it is unclear whether this is because of implementation differences, providers’ lack of knowledge about capabilities, or lack of capabilities in the EHRs themselves.^16–18

We explore how well EHRs—as currently implemented—meet the measurement-related quality improvement needs in primary care practice. To do so, we examined survey data from 1,492 practices and combined this information with qualitative data to gain a richer answer than surveys alone could provide. Our findings highlight the challenges that practices face as value-based payment replaces volume-based systems.

Study Data And Methods

Study Design And Cohort

In 2015 the Agency for Healthcare Research and Quality (AHRQ) launched EvidenceNOW: Advancing Heart Health in Primary Care. EvidenceNOW is a three-year initiative dedicated to helping small and medium-size primary care practices across the US use the latest evidence to improve cardiovascular health and develop their capacity for ongoing improvement. AHRQ funded seven grantees (called cooperatives) that span seven US regions (and twelve states). Cooperatives were tasked with developing and leveraging sustainable infrastructure to support over 200 practices in their regions in improving electronic clinical quality measures endorsed by CMS and the National Quality Forum for aspirin use,¹⁹ blood pressure monitoring,²⁰ cholesterol management,²¹ and smoking screening and cessation support²² (the ABCS measures).

AHRQ also funded an evaluation of this initiative called Evaluating System Change to Advance Learning and Take Evidence to Scale (ESCALATES) to centralize, harmonize, collect, and analyze mixed-methods data with the goal of generating cross-cooperative, generalizable findings.²³ ESCALATES started at the same time the cooperatives’ work began. The goals of ESCALATES included identifying facilitators of and barriers to implementing regionwide infrastructure to support quality improvement among primary care practices, of which health information technology (IT) was a central component.

Data Sources

ESCALATES compiled quantitative survey data collected by the cooperatives from the 1,492 practices. While cooperative study designs (for example, stepped wedge, group randomized trials) varied, all cooperatives used their first year (May 2015–April 2016) for recruitment and start-up activities, and all staggered the time at which practices received the intervention. Survey data were collected from practices before the start of the intervention (that is, at baseline), which ranged from September 2015 to April 2017. We collected complementary qualitative data (observation, interview, and online diary) for this study in the period May 2015–April 2017.²³ We chose this time period because it gave us exposure to the data issues that manifested themselves during start-up and implementation.

Qualitative Data Collection And Management

We conducted two site visits with every cooperative. The first site visit occurred before implementation of the intervention (August 2015–March 2016) and focused on understanding the cooperative, its partners, regional resources (including EHR and data capacities), and approach to supporting large-scale practice improvement. The second site visit was conducted during implementation of the intervention (July 2016–April 2017) and focused on observing practice facilitators work with practices. We observed forty-one facilitators conducting sixty unique practice quality improvement visits. During site visits we took field notes and conducted and recorded (and later transcribed the recordings of) semistructured interviews with key stakeholders (for example, investigators, facilitators, and health IT experts).

To supplement observation and interview data, we attended and took notes at a meeting of an AHRQ-initiated cooperative work group to discuss health IT challenges, and we implemented an online diary²⁴ for each cooperative that included documentation by key stakeholders (such as investigators, health IT experts, and facilitators) of implementation experiences in real time (approximately twice a month).

Online diary data, interviews, meeting notes, and field notes were deidentified for individual participants and reviewed for accuracy. To confirm our findings, cooperative representatives completed a table that characterized obstacles to using EHR data for quality improvement.

We used Atlas.ti for data management and analysis. The Oregon Health & Science University Institutional Review Board approved and monitored this study.

Survey Measures

Cooperatives administered a survey to all of their practices. The survey, completed by a lead clinician or practice manager, consisted of a subset of questions from the National Ambulatory Medical Care Survey’s Electronic Medical Records Questionnaire^25–28 and assessed practice characteristics, EHR characteristics,²⁹ and reporting capabilities (see online appendix exhibit A1 for survey items).³⁰

Qualitative Data Analysis

Three authors (Deborah Cohen, Jennifer Hemler, and Jennifer Hall) analyzed qualitative data in real time following an immersion-crystallization approach³¹ and coded data to identify text related to clinical quality measurement, quality improvement, and EHRs. We analyzed data within and across cooperatives to identify nuanced findings and variations regarding usage of EHRs for quality improvement. Data collection and analysis were iterative; initial findings prompted additional questions that were later answered in the online diaries and during site visits to cooperatives.³² We triangulated data with other sources, discussing differences until we reached saturation—the point at which no new findings emerged.³² Qualitative findings informed the selection of variables for quantitative analyses, and both quantitative and qualitative data informed interpretations.

Quantitative Data Analysis

Two authors (Bijal Balasubramanian and Miguel Marino) used descriptive statistics to characterize the EvidenceNOW practice sample and used multivariable logistic regression to evaluate the association between practice characteristics and EHR reporting capability, measured as a “yes” or “no” response to the following question: “Does your practice have someone who can configure or write quality reports from the EHR?” Indicator variables for cooperatives were included in the logistic model to account for regional variability, and we used multiple imputation by chained equations to account for missing data (see appendix exhibit A2).³⁰ We performed statistical analyses using R, version 3.4.0.

Limitations

Our study had several limitations. First, our findings may have underestimated the challenges that practices face in using EHRs for quality measurement, as the practices recruited to participate in EvidenceNOW may have self-selected based on their greater quality improvement and health IT confidence.

Second, our understanding of practices’ challenges in using EHRs for quality measurement was based on the views of cooperative experts and does not necessarily represent the practices’ perspectives. Thus, we were unable to quantify the extent to which practices experienced these problems. Yet it is from the cooperatives’ vantage point that we identified problems that are often difficult to characterize using practice-level surveys, and it may be that solutions are most effective at the regional rather than practice level.

Third, our primary survey outcome—the response to the question “Does your practice have someone who can configure or write quality reports from the EHR?”—combines workforce and reporting capacity in a single item. While it might be preferred to parse these issues in separate items, we did not do this because of concerns about response burden. Our qualitative data suggest that directing more survey questions to practices might not have been useful, since practices lack staff with the expertise to answer more technically complex questions. Data collected from cooperatives’ health IT experts complemented practice survey data, shedding light on this complex issue.

Fourth, our study findings were also limited by our inability to identify whether some EHRs faced more or fewer challenges than others, and by the fact that some survey items had more than 10 percent missing data. However, our conclusions were based on one of the largest studies of geographically dispersed primary care practices, and the use of multiple imputation leveraged this scale to minimize potential bias due to missing data.

Study Results

Of the 1,710 practices recruited to EvidenceNOW, 1,492 (87.3 percent) completed the practice survey. The majority of these practices had ten or fewer clinicians (84 percent), were located in urban or suburban areas (71 percent), and were owned by clinicians (40 percent) or hospital/health systems (23 percent) (exhibit 1). Over 93 percent used EHRs, of which 81 percent were certified by the ONC for 2014. While sixty-eight different EHRs were represented, Epic, eClinicalWorks, and NextGen were the most commonly used systems. The number of different EHR systems among practices within a cooperative ranged from four to thirty-two. Sixty percent of practices participated in stages 1 and 2 of meaningful use. (More detailed findings are in exhibit 1 and appendix exhibit A2.)³⁰

Challenges Using Electronic Clinical Quality Measures For Quality Improvement

Practices and quality improvement facilitators experienced significant challenges using EHRs to generate tailored reports of electronic clinical quality measures for quality improvement, which led to substantial delays in reporting quality measures and engaging in measurement-informed quality improvement activities (exhibit 2).

Generating Reports Of Electronic Clinical Quality Measures For Quality Improvement

Practices participating in stages 1 and 2 of meaningful use were more likely to report being able to generate reports of electronic clinical quality measures at the practice and clinician levels, compared to practices not participating (odds ratio: 1.65) (exhibit 3). Similarly, practices participating in quality improvement demonstration projects or in external payment programs that incentivized quality measurement had 51–73 percent higher odds of reporting an ability to generate reports of electronic clinical quality measures (exhibit 3). Facilitators and health IT experts working directly with practices noted that practices could produce reports that complied with meaningful use. However, EHR reporting tools did not meet practices’ needs for quality improvement measurement.

Practices reported needing reports with customizable time frames, which could be repeated as desired, to align with quality improvement activities. Cooperative experts reported that some ONC-certified EHRs, as implemented, could generate Physician Quality Reporting System or meaningful-use clinical quality reports only for a calendar year. When functions were available to customize measurement periods, significant manual configuration or additional modules were required. According to a report on measurement challenges from cooperative 3, “out of the box tools are inadequate to use for routine quality improvement. This necessitated working with vendors to deploy reports in the linked reporting tool, which required expertise in database query writing, which is almost universally absent from the skillset of staff at independent small practices.”

EHR vendors charged extra fees to access these tools, and smaller practices could not pay for this assistance. Additionally, some EHRs could generate meaningful-use metrics only for patients with Medicare or Medicaid coverage (often a minority of practice patients). Many vendors were resistant to making software changes beyond what was required for Physician Quality Reporting System or meaningful use reporting. Thus, most practices were unable to query EHR data for measurement in rapid-cycle tests of change.

Practices owned by health/hospital systems had higher odds of reporting the ability to generate reports of electronic clinical quality measures, compared to clinician-owned practices (OR: 2.88), while solo and rural practices were less likely than practices with six or more physicians and those in urban areas to report being able to generate such reports (exhibit 3). Complementary qualitative data showed that system-owned practices had greater health IT and data capability than solo and rural practices did, but these resources were centralized. These practices and facilitators experienced substantial and repeated delays in getting access to data needed for quality improvement, as organizational priorities took precedence (particularly when tied to payment), and their experts were overwhelmed with other demands.

New Clinical Guidelines

Quality measurement was complicated by changes in clinical guidelines. The American College of Cardiology and American Heart Association guidelines on cardiovascular disease risk changed dramatically in 2013.³³ At the start of EvidenceNOW in 2015, measurements for the A, B, and S parts of the ABCS measures were routinely part of the Physician Quality Reporting System. However, CMS did not publish the criteria for the C part (the cholesterol measure) until May 4, 2017. The measure chosen for the EvidenceNOW initiative matched the 2013 guideline, but lack of a complementary official CMS measure meant that no EHR had yet implemented a similar measure in their system. Some practices created their own measures based on all or part of the new guidelines to inform quality improvement, but this was not useful for benchmarking.

Validity Across Different Electronic Health Record Systems

Facilitators and health IT experts often found verifiable problems in clinical quality reports. For example, a representative of cooperative 6 told us in an interview: “Doctors always look at our data and say it’s not [correct]…. Unless you put [information] in the exact spot, it doesn’t pull it [for the electronic clinical quality measures]…. They didn’t hit the little cog-radio button. It takes [you] to a template that you have to complete. In order to pull the data it has to be on there.”

It was common for there to be specific locations (for example, checkboxes) where structured data elements had to be recorded to be counted in a calculation of electronic clinical quality measures. The combination of vendor-standardized documentation requirements for the measures, lack of alignment of these requirements with clinical workflows, and clinical teams’ lack of awareness of documentation rules and the consequences of recording patterns on quality measurement led to many examples of unreliable reports of the measures.

Challenges Developing Regional Data Infrastructure For Quality Improvement

Cooperatives that used data warehouses, hubs, or health information exchanges in their regions did so to provide practices with clean data and tools for measurement. To develop this type of data infrastructure, cooperatives worked with EHR vendors to access back-end EHR data. Exhibit 2 summarizes the challenges cooperatives faced in using EHR data for this purpose.

Cooperatives reported that EHR vendors and other organizations charged high fees ($5,000–$15,000) for accessing data, and cooperatives found it difficult to export continuity-of-care documents (electronic documents standardized for patient information exchange) in batch format. Exporting batches of these documents—a requirement for ONC certification²⁹—means that EHR data can be extracted from multiple patient records simultaneously and pulled into one file. The documents are meant to include most commonly needed patient information in a form that can be shared across computer applications. Yet cooperatives found that the documents met only minimum requirements. One representative of cooperative 7 said in an interview that one vendor “will only send ten [documents] at a time. Another only does, like, one an hour or something. There are these bizarre kinds of things where they’re meeting the requirement, but they aren’t useful.” Cooperatives and practices queried vendors about these issues, but vendors were not responsive. Lack of efficient, mass export of continuity-of-care documents meant that cooperatives were unable to help practices use their underlying data for quality improvement.

Cooperatives that developed data management infrastructure also created the capacity for combining and comparing data (performance benchmarking) and found differences in electronic clinical quality measures across EHRs. One expert attributed differences to variations in implementation of the EHR system and noted in cooperative 1’s report on measurement challenges that “editorial decisions” made by vendors about which EHR fields to pull data from when calculating the measures led to problems: “We have experienced challenges in defining the measures and achieving accurate results. We began with pre-built measures from our software vendor but often found differences in definition for a commonly named measure.” Extra steps were needed to ensure uniform definitions of measures for performance benchmarking. Without regional infrastructure to extract and normalize ABCS measures, performance data from different EHR systems might not be correct or comparable.

Discussion

Primary care practices have to exert too much effort to get usable data from their EHRs to improve care quality and meet reporting requirements. Despite the large national investment in health IT and substantial investments of time and expertise by practices and cooperatives, it has been difficult for them to generate timely and usable data for quality measurement and improvement. These findings are particularly salient given that the majority of practices in this large national sample used EHRs certified by the Office of the National Coordinator for Health Information Technology, and more than half reported participating in stages 1 and 2 of meaningful use and having the ability to produce reports of electronic clinical quality measures. Yet EHR reports that complied with meaningful use generally did not allow practices to customize date ranges or report frequency, and they rarely provided functionality for measuring performance by individual clinicians. In cases where this functionality was available, it was a costly upgrade and typically required health IT expertise to use. These resources are not often present in practices—particularly solo, rural, and clinician-owned practices. Additionally, practices that questioned the validity of meaningful-use reports did not have feasible ways to validate them. These factors inhibited practices’ ability to make the measurements—the ongoing identification of quality gaps and monitoring of the effects of changes made in care processes—that are essential for quality improvement.

These persistent challenges should be cause for concern. Our study amplifies findings from prior research that documented challenges in using EHRs in general and for quality improvement in particular.^{3,16–18,34–37} To our knowledge, previous studies have not examined this problem in a sample of the size and diversity that our sample attained, especially not with mixed methods. Our study shows that survey data alone are inadequate for fully understanding the problem. Furthermore, most studies of these challenges are nearly a decade old and do not reflect the impact of more recent federal programs.^3,38

Regional and national organizations that connect disparate practices’ EHR systems to a central data repository offer a potential solution for mitigating measurement challenges through shared infrastructure for data extraction, normalization, validation, analysis, and reporting. However, most states in our sample did not have regional data infrastructure, and those that did had limited reach; the time, effort, and investment required to build these resources are extensive.⁸ Regional leaders struggle with financing, vendor relations, and governance structures.

To improve EHRs’ ability to achieve their potential and support sustainable payment reforms, policy makers should consider empowering the ONC and CMS to expand their standards and requirements for, and monitoring of, EHR vendors.³⁹ The agencies need to make it more efficient for practices to generate quality reports with up-to-date definitions,⁴⁰ ensure that organizations can extract data from batches of continuity-of-care documents for secondary use, and create explicit requirements to support quality improvement and practice population health measurement. New initiatives from the ONC are encouraging vendors to facilitate this process through standard application programming interfaces and mapping (for example, Fast Healthcare Interoperability Resources). With several new reporting requirements for clinicians (including those for recognition as a patient-centered medical home), payer requirements, and other federal demonstration projects (which have differing reporting requirements), the ONC should focus not just on EHR capacities to serve the Quality Payment Program but also on quality improvement and reporting needs generally.

Cooperatives’ current experience is that EHR data are “locked up,” which prevents even a well-resourced initiative from being able to use data for quality measurement across diverse practice settings. The first Quality Payment Program reporting period ended in December 2017, and many clinicians may be unable to achieve their full potential related to quality improvement despite having certified EHRs. CMS may need to be prepared to help practices not only comply but also use data for quality improvement—perhaps by loosening reporting options, expanding exclusion criteria, and allocating additional funds for technical assistance.

Conclusion

Primary care is an essential part of healthy communities. With federal value-based payment programs such as the Quality Payment Program poised to motivate clinicians to improve care quality, investment is needed to ensure that the health IT clinicians use delivers credible clinical quality data and has the functionality necessary to inform quality improvement efforts as well as external reporting for payment and other purposes without adding to an already high burden.

ACKNOWLEDGMENTS

A version of the findings presented in this article was reported at the 44th Annual North American Primary Care Research Group Meeting, Colorado Springs, Colorado, November 12–16, 2016. This research was supported by the Agency for Healthcare Research and Quality (Grant No. R01HS023940-01). This article could not have been completed without the help of people from the seven EvidenceNOW cooperatives, to whom the authors are greatly indebted. Members of the national evaluation team were also important in supporting this study, including Rachel Springer, David Cameron, Bernadette Zakher, Rikki Ward, Benjamin Crabtree, Kurt Stange, and William Miller. Without their efforts, this work would not have been possible. Finally, the authors acknowledge Amanda Delzer Hill, who assisted with copy editing.

NOTES

• 1 Office of the National Coordinator for Health Information Technology. Office-based physician electronic health record adoption [Internet]. Washington (DC): Department of Health and Human Services; 2016 Dec [cited 2018 Feb 6]. (Health IT Quick-Stat No. 50). Available from: https://dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php Google Scholar
• 2 Hsiao CJ, Jha AK, King J, Patel V, Furukawa MF, Mostashari F. Office-based physicians are responding to incentives and assistance by adopting and using electronic health records. Health Aff (Millwood). 2013;32(8):1470–7. Go to the article, Google Scholar
• 3 Lynch K, Kendall M, Shanks K, Haque A, Jones E, Wanis MGet al. The Health IT Regional Extension Center program: evolution and lessons for health care transformation. Health Serv Res. 2014;49(1 Pt 2):421–37. Crossref, Medline, Google Scholar
• 4 Centers for Medicare and Medicaid Services. Electronic Health Records (EHR) Incentive Programs [Internet]. Baltimore (MD): CMS; [last updated 2017 Nov 29; cited 2018 Jan 23]. Available from: https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/index.html?redirect=/ehrincentiveprograms Google Scholar
• 5 Xierali IM, Hsiao CJ, Puffer JC, Green LA, Rinaldo JC, Bazemore AWet al. The rise of electronic health record adoption among family physicians. Ann Fam Med. 2013;11(1):14–9. Crossref, Medline, Google Scholar
• 6 CMS.gov. Quality Payment Program resource library [Internet]. Baltimore (MD): Centers for Medicare and Medicaid Services; [last modified 2018 Feb 6; cited 2018 Feb 6]. Available from: https://www.cms.gov/Medicare/Quality-Payment-Program/Resource-Library/Resource-library.html Google Scholar
• 7 Rao SR, Desroches CM, Donelan K, Campbell EG, Miralles PD, Jha AK. Electronic health records in small physician practices: availability, use, and perceived benefits. J Am Med Inform Assoc. 2011;18(3):271–5. Crossref, Medline, Google Scholar
• 8 Mostashari F, Tripathi M, Kendall M. A tale of two large community electronic health record extension projects. Health Aff (Millwood). 2009;28(2):345–56. Go to the article, Google Scholar
• 9 Howard J, Clark EC, Friedman A, Crosson JC, Pellerano M, Crabtree BFet al. Electronic health record impact on work burden in small, unaffiliated, community-based primary care practices. J Gen Intern Med. 2013;28(1):107–13. Crossref, Medline, Google Scholar
• 10 Friedman A, Crosson JC, Howard J, Clark EC, Pellerano M, Karsh BTet al. A typology of electronic health record workarounds in small-to-medium size primary care practices. J Am Med Inform Assoc. 2014;21(e1):e78–83. Crossref, Medline, Google Scholar
• 11 Heisey-Grove D, Patel V. National findings regarding health IT use and participation in health care delivery reform programs among office-based physicians. J Am Med Inform Assoc. 2017;24(1):130–9. Crossref, Medline, Google Scholar
• 12 eCQI Resource Center. 2017 performance period EP/EC eCQMs [Internet]. Baltimore (MD): Centers for Medicare and Medicaid Services; [cited 2018 Feb 6]. Available from: https://ecqi.healthit.gov/eligible-professional-eligible-clinician-ecqms/2017-performance-period-epec-ecqms Google Scholar
• 13 McNamara P, Shaller D, De La Mare J, Ivers N. Confidential physician feedback reports: designing for optimal impact on performance [Internet]. Rockville (MD): Agency for Healthcare Research and Quality; 2016 Mar [cited 2018 Feb 6]. (AHRQ Publication No. 16-0017-EF). Available from: https://www.ahrq.gov/sites/default/files/publications/files/confidreportguide_0.pdf Google Scholar
• 14 Krist AH, Green LA, Phillips RL, Beasley JW, DeVoe JE, Klinkman MSet al. Health information technology needs help from primary care researchers. J Am Board Fam Med. 2015;28(3):306–10. Crossref, Medline, Google Scholar
• 15 Higgins TC, Crosson J, Peikes D, McNellis R, Genevro J, Meyers D. Using health information technology to support quality improvement in primary care [Internet]. Rockville (MD): Agency for Healthcare Research and Quality; 2015 Mar [cited 2018 Feb 6]. (AHRQ Publication No. 15-0031-EF). Available from: https://pcmh.ahrq.gov/page/using-health-information-technology-support-quality-improvement-primary-care Google Scholar
• 16 Parsons A, McCullough C, Wang J, Shih S. Validity of electronic health record–derived quality measurement for performance monitoring. J Am Med Inform Assoc. 2012;19(4):604–9. Crossref, Medline, Google Scholar
• 17 Roth CP, Lim YW, Pevnick JM, Asch SM, McGlynn EA. The challenge of measuring quality of care from the electronic health record. Am J Med Qual. 2009;24(5):385–94. Crossref, Medline, Google Scholar
• 18 Chan KS, Fowles JB, Weiner JP. Review: electronic health records and the reliability and validity of quality measures: a review of the literature. Med Care Res Rev. 2010;67(5):503–27. Crossref, Medline, Google Scholar
• 19 eCQI Resource Center. Ischemic vascular disease (IVD): use of aspirin or another antiplatelet [Internet]. Baltimore (MD): Centers for Medicare and Medicaid Services; [last updated 2017 Oct 25; cited 2018 Feb 6]. Available from: https://ecqi.healthit.gov/ecqm/measures/cms164v5 Google Scholar
• 20 eCQI Resource Center. Controlling high blood pressure [Internet]. Baltimore (MD): Centers for Medicare and Medicaid Services; [last updated 2017 Jul 12; cited 2018 Feb 6]. Available from: https://ecqi.healthit.gov/ecqm/measures/cms165v3 Google Scholar
• 21 eCQI Resource Center. Statin therapy for the prevention and treatment of cardiovascular disease [Internet]. Baltimore (MD): Centers for Medicare and Medicaid Services; [last updated 2017 Oct 25; cited 2018 Feb 6]. Available from: https://ecqi.healthit.gov/ep/ecqms-2018-performance-period/statin-therapy-prevention-and-treatment-cardiovascular-disease Google Scholar
• 22 eCQI Resource Center. Preventive care and screening: tobacco use: screening and cessation intervention [Internet]. Baltimore (MD): Centers for Medicare and Medicaid Services; [last updated 2017 Jul 12; cited 2018 Feb 6]. Available from: https://ecqi.healthit.gov/ecqm/measures/cms138v3 Google Scholar
• 23 Cohen DJ, Balasubramanian BA, Gordon L, Marino M, Ono S, Solberg LIet al. A national evaluation of a dissemination and implementation initiative to enhance primary care practice capacity and improve cardiovascular disease care: the ESCALATES study protocol. Implement Sci. 2016;11(1):86. Crossref, Medline, Google Scholar
• 24 Cohen DJ, Leviton LC, Isaacson N, Tallia AF Crabtree BF. Online diaries for qualitative evaluation: gaining real-time insights. Am J Eval. 2006;27(2):163–84. Crossref, Google Scholar
• 25 Nutting PA, Crabtree BF, Miller WL, Stange KC, Stewart E, Jaén C. Transforming physician practices to patient-centered medical homes: lessons from the national demonstration project. Health Aff (Millwood). 2011;30(3):439–45. Go to the article, Google Scholar
• 26 Balasubramanian BA, Chase SM, Nutting PA, Cohen DJ, Strickland PA, Crosson JCet al. Using Learning Teams for Reflective Adaptation (ULTRA): insights from a team-based change management strategy in primary care. Ann Fam Med. 2010;8(5):425–32. Crossref, Medline, Google Scholar
• 27 Shaw EK, Ohman-Strickland PA, Piasecki A, Hudson SV, Ferrante JM, McDaniel RR Jret al. Effects of facilitated team meetings and learning collaboratives on colorectal cancer screening rates in primary care practices: a cluster randomized trial. Ann Fam Med. 2013;11(3):220–8, S1–8. Crossref, Medline, Google Scholar
• 28 National Center for Health Statistics. Ambulatory health care data [Internet]. Hyattsville (MD): NCHS; [last updated 2017 Dec 12; cited 2018 Feb 6]. Available from: https://www.cdc.gov/nchs/ahcd/index.htm Google Scholar
• 29 Office of the National Coordinator for Health Information Technology. ONC fact sheet: 2015 edition health information technology (health IT) certification criteria, base electronic health record (EHR) definition, and ONC health IT certification program modifications final rule [Internet]. Washington (DC): Department of Health and Human Services; 2015 Oct [cited 2018 Feb 6]. Available from: https://www.cdc.gov/ehrmeaningfuluse/docs/onc_factsheet_2015_cehrt.pdf Google Scholar
• 30 To access the appendix, click on the Details tab of the article online.
• 31 Borkan J. Immersion/crystallization. In: Crabtree BF, Miller WL, editors. Doing qualitative research. 2nd ed. Thousand Oaks (CA): Sage Publications; 1999. p. 179–94. Google Scholar
• 32 Cohen DJ, Crabtree BF. Evaluative criteria for qualitative research in health care: controversies and recommendations. Ann Fam Med. 2008;6(4):331–9. Crossref, Medline, Google Scholar
• 33 Stone NJ, Robinson J, Lichtenstein AHet al. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014;129(25 Suppl 2):S1–45. Crossref, Medline, Google Scholar
• 34 Baron RJ. Quality improvement with an electronic health record: achievable, but not automatic. Ann Intern Med. 2007;147(8):549–52. Crossref, Medline, Google Scholar
• 35 Fernandopulle R, Patel N. How the electronic health record did not measure up to the demands of our medical home practice. Health Aff (Millwood). 2010;29(4):622–8. Go to the article, Google Scholar
• 36 Krist AH, Beasley JW, Crosson JC, Kibbe DC, Klinkman MS, Lehmann CUet al. Electronic health record functionality needed to better support primary care. J Am Med Inform Assoc. 2014;21(5):764–71. Crossref, Medline, Google Scholar
• 37 Lyons SS, Tripp-Reimer T, Sorofman BA, Dewitt JE, Bootsmiller BJ, Vaughn TEet al. VA QUERI informatics paper: information technology for clinical guideline implementation: perceptions of multidisciplinary stakeholders. J Am Med Inform Assoc. 2005;12(1):64–71. Crossref, Medline, Google Scholar
• 38 Heisey-Grove D, Danehy L-N, Consolazio M, Lynch K, Mostashari F. A national study of challenges to electronic health record adoption and meaningful use. Med Care. 2014;52(2):144–8. Crossref, Medline, Google Scholar
• 39 Washington V, DeSalvo K, Mostashari F, Blumenthal D. The HITECH era and the path forward. N Engl J Med. 2017;377(10):904–6. Crossref, Medline, Google Scholar
• 40 HealthIT.gov. ONC regulation FAQs: #42 Question [06-13-042-1] [Internet]. Washington (DC): Department of Health and Human Services; [last updated 2013 Nov 14; cited 2018 Feb 7]. Available from: https://www.healthit.gov/policy-researchers-implementers/42-question-06-13-042 Google Scholar