Evaluation Of The Behavioral Health Integration And Complex Care Initiative In Medi-Cal

The Affordable Care Act (ACA) has provided new opportunities to improve the health of low-income Americans.^1–3 In California, the Inland Empire Health Plan (IEHP) has been an active participant in these health reform efforts. The plan serves over one million Medi-Cal (California Medicaid) beneficiaries in San Bernardino and Riverside Counties, a large geographic area containing urban, rural, and wilderness areas and having high rates of unemployment, poverty, and social challenges.⁴ The IEHP has a network of over 4,000 providers and added 350,000 Medi-Cal beneficiaries in 2014, approximately two-thirds of whom were newly eligible as a result of the ACA.

Despite gains in insurance coverage, Medi-Cal beneficiaries in these counties continue to face challenges in accessing health care because of a shortage of primary care physicians and specialty providers.⁴ As is the case in other counties in California, the health care systems serving the Inland Empire have funding streams that have historically been siloed and that effectively separate physical health, behavioral health, and substance use treatment providers and disconnect them from each other.⁵

In response to concerns about access to and quality of care for Medi-Cal beneficiaries, the IEHP engaged in a system-level transformation program called the Behavioral Health Integration and Complex Care Initiative (BHICCI). The initiative aims to support the development of health homes that provide integrated physical and behavioral health care and complex care management.^6–8 It supports practice transformation in twelve health care organizations across thirty clinical sites, including federally qualified health centers, multispecialty clinics, and behavioral health clinics. The initiative aims to improve both physical and behavioral health outcomes through the use of an integrated complex care team that uses a whole-person approach to address physical, behavioral, social, and environmental needs. Preparing participating organizations for value-based case-rate funding as health homes will ensure the sustainability of the initiative.⁹ It received the 2017 Quality Innovation Award from the California Department of Health Care Services. In this article we describe the BHICCI model and the results from a Learning Evaluation of the BHICCI, which the authors conducted under contract.¹⁰

The Behavioral Health Integration And Complex Care Initiative

The BHICCI approach to practice transformation was based on the Institute for Healthcare Improvement’s Breakthrough Series Learning Collaborative Model, which emphasizes the use of collaborative learning to achieve major changes in health care delivery.¹¹ Implementing the initiative required practice changes within multiple parts of the health care system, including engaging leaders, hiring the “right people” for the care team, training providers in collaborative care and complex care management, and designing new workflows to support data collection and population health measurement.

Each BHICCI care team comprises three people: a care manager, behavioral health clinician, and care coordinator. The care manager works with patients to identify their health and wellness goals and incorporates these goals into shared care plans that facilitate communication among providers. The care manager supports patient activation, encourages patients to achieve their goals (often using motivational interviewing) and provides patient and family education about chronic medical and behavioral health conditions to improve health literacy. The behavioral health clinician provides evidence-based treatments such as behavioral activation, motivational interviewing, problem-solving therapy, and cognitive behavioral therapy, using a whole-person approach for mental health and substance abuse conditions. The care coordinator performs patient screenings and vital measurements, assists in the coordination of appointments and referrals, and ensures that appropriate releases of information are obtained to facilitate record sharing across providers and facilities. Patients eligible for the BHICCI had at least one chronic medical condition and one behavioral health condition.

Another participating provider was the practice coach. These coaches facilitated collaboration and supported practice changes among the care teams.¹² The coaches provided individualized, hands-on guidance to support the successful implementation of integrated chronic care management and person-centered health care. They fostered collaborative relationships, empowered teams, and facilitated whole-person care using a relational coaching approach that was guided by several core values: learner-centered, collaborative problem solving, partnering and mutuality, intentionality, social justice, and team vitality. Coaching activities were tailored for each organization to support its unique context in practice transformation as needed to achieve the Triple Aim of improving population health and the patient experience with care while remaining cost-effective.¹³

Study Data And Methods

Participating Organizations

Exhibit 1 lists the organizations participating in the BHICCI. These included federally qualified health centers that provided primary care (four organizations, with nineteen clinics collectively), multispecialty clinics that provided specialized care (two organizations, with four clinics collectively), and behavioral health clinics that provided mental health and substance use care (six organizations, with seven clinics collectively). Each participating organization engaged in quality improvement activities using the Model for Improvement.¹⁴ Practice coaches challenged care teams to identify change ideas and test them using the plan-do-study-act (PDSA) framework. Teams received ongoing training in identifying appropriate change ideas, making hypotheses, testing the ideas, and building upon previous test cycles in a process known as “PDSA ramps.” Implementation of quality improvement efforts within the care teams helped ensure the consistency and sustainability of ongoing testing and refinement of practice changes. Teams also participated in semiannual learning sessions, which were used to introduce change ideas and were designed as an opportunity for organizations to develop collaborative relationships through small-group discussions and activities.

This study used multiple methods to evaluate the implementation of the BHICCI, with a focus on the Triple Aim.¹³ Quantitative data were used to describe BHICCI patients, programs, and changes in clinical outcomes and costs. Qualitative data were used to describe the patient experience with care.

Identifying Patients And Measuring Clinical Outcomes

BHICCI patients were identified from a study roster of patients enrolled in the initiative between January 1, 2016, and April 6, 2018. Clinical outcomes data were captured in a registry. BHICCI teams were coached to monitor the following outcomes: depressive symptoms, using the nine-item Patient Health Questionnaire-9, recommended every month for patients with assessment scores above the clinical threshold for depression; glycosylated hemoglobin (HbA1c), recommended every three months for patients with known diabetes or with certain risk factors for the disease; and systolic blood pressure and body mass index, recommended at every visit for every patient.

The registry included clinical outcomes data recorded through April 6, 2018. Clinical outcomes were examined at the patient level at baseline, defined as within ninety days of BHICCI enrollment, and the most recent follow-up assessment within 365 days of BHICCI enrollment. The following process measures were examined: rates of screening at baseline, percentage of patients with clinical values exceeding clinical thresholds at baseline, and percentage of patients exceeding thresholds with a follow-up measure within one year. The outcomes were compared among patients with baseline and follow-up measures and clinical values exceeding clinical thresholds at baseline, using paired t-tests. These analyses were limited to patients with 180 days or more of enrollment in the BHICCI to allow time to observe meaningful changes in clinical outcomes.

Health Care Costs

Changes in per capita costs among BHICCI patients were compared to changes among a matched comparison group of enrollees in the Inland Empire Health Plan who were not in the initiative (described below), within a difference-in-differences framework. Administrative data used for these analyses included demographic, eligibility, claims, encounter, and pharmacy data for the period June 2015–December 2017. These data encompassed services covered by the IEHP that were either reimbursed on a fee-for-service basis or delivered under capitated contracts and recorded as encounters. Prices in the encounter data were set at the procedure-code level, using the average fee-for-service payment for a given procedure. The data did not include services that were not covered by the IEHP, such as specialty mental health or substance use services.

The data included information about both BHICCI patients and IEHP enrollees who were assigned to a participating organization but did not themselves participate in the initiative. The IEHP is sufficiently large that it was possible to identify patients with similar characteristics who were and were not enrolled in the initiative. Pre and post periods were created for BHICCI patients for up to 365 days before and after enrollment in the initiative. To create pre and post periods for the comparison group, IEHP enrollees not in the initiative were randomly assigned an enrollment date according to the distribution of enrollment dates observed among BHICCI patients. To allow comparable pre-post comparisons, we limited the cost analyses to BHICCI patients with 180 or more days of enrollment in the BHICCI before December 2017 and 180 days of enrollment in the IEHP before enrollment in the initiative, and to patients in the comparison group with 180 days of enrollment in the IEHP before and after their randomly assigned start date.

Propensity Score Matching

Propensity scores and nearest neighbor matching were used to identify a comparison group of IEHP enrollees with demographic, clinical, and service use characteristics similar to those of the BHICCI patients.^15,16 Demographic characteristics included age, sex, race/ethnicity, and Medi-Cal eligibility category: adult, family, aged, or disabled. Clinical diagnoses were measured using the Chronic Illness and Disability Payment System.¹⁷ Data on service use included the numbers of inpatient admissions, emergency department (ED) visits, days with office visits, and pharmaceutical fills in the year before enrollment.

Logistic regression was used to predict BHICCI enrollment. Nearest neighbor matching was used to match one IEHP enrollee to each BHICCI patient based on the closest predicted probabilities of BHICCI enrollment. Matching occurred without replacement. Matches were limited by calipers to within 0.2 of the standard deviation of propensity scores and to a common support. The BHICCI and IEHP samples were well matched: The average absolute standardized difference in predictor variables was 1.3 percent, with a maximum of 6.5 percent (see the online appendix).¹⁸

Cost Analysis Methods

Generalized linear models were used to estimate the effect of participation in the BHICCI on costs.¹⁹ Dependent variables were monthly inpatient, ED, outpatient, pharmacy, and total costs. Independent variables included indicators for month of enrollment, enrollment in the BHICCI, the post period, and the interaction between BHICCI enrollment and the post period. Costs were estimated using two-part models: The probability of use was estimated using logistic regression, while costs for patients with one or more services were estimated using a gamma distribution and a log link function. Standardized estimates of costs were calculated over both parts of the model and over all individuals as they were alternatively assigned to the BHICCI and comparison groups in the pre and post periods. Standard errors were estimated using a nonparametric bootstrap, and p values were estimated using the percentile method.²⁰ Standard accounting methods were used to estimate the cost of the new staffing for BHICCI care teams.

Patient Experience

The patient experience with the BHICCI was derived from eight focus groups conducted in 2017. These focus groups were convened at five sites with early implementation of the BHICCI that were purposefully sampled so that participants would be able to speak to any changes they experienced with their health care.²¹ Patients were purposefully recruited by care teams based on their availability and their knowledge of the BHICCI. Six focus groups (with forty members, collectively) were conducted in English, and two (with fourteen members, collectively) were conducted in Spanish.

During the focus groups, participants were asked to describe their experiences with the BHICCI, including those with care management for their complex health conditions and with care coordination, and their involvement with care planning. Sessions lasted approximately sixty minutes and were audiorecorded and professionally transcribed. Thematic analysis of transcripts and field notes was conducted using Nvivo software. The credibility of the findings was supported by several strategies of rigor, including member checking, independent cocoding of transcripts, and consensus-driven findings.²² Transcripts from the Spanish-language groups were analyzed in Spanish.

Study Results

As of April 6, 2018, 6,699 IEHP members had enrolled in the BHICCI (exhibit 2). Their mean age was forty-eight years. Fifty-nine percent were female, 41 percent Latino, 11 percent African American, and 24 percent disabled. Of the patients, 3,570 received services in federally qualified health centers, 1,848 in multispecialty clinics, and 1,281 in behavioral health clinics (data not shown). There were no significant differences between all of the BHICCI patients and those who were included in the analyses of clinical or cost outcomes.

Clinical Outcomes

Rates of screening were greatest for systolic blood pressure (87 percent) and lowest for HbA1c (41 percent) (exhibit 3). Among those screened, 60 percent exceeded the clinical threshold for major depression in the Patient Health Questionnaire-9, while fewer patients exceeded the clinical threshold for systolic blood pressure (24 percent), HbA1c (41 percent), or body mass index (55 percent). Among those who exceeded the threshold at baseline, rates of follow-up screening varied by measure, from 92 percent for systolic blood pressure to 79 percent for HbA1c. And among those patients whose scores exceeded the threshold and had follow-up measures for depression and systolic blood pressure, there were significant and clinically meaningful improvements.

Screening rates varied by type of organization (data not shown). Multispecialty clinics had the highest rates of screening for depression (81 percent) and systolic blood pressure (99 percent), while federally qualified health centers had the highest rates for HbA1c (51 percent) and body mass index (90 percent). Rates of screening were lowest in behavioral health clinics for all four clinical outcomes: depression (54 percent), systolic blood pressure (62 percent), HbA1c (13 percent), and body mass index (51 percent). Among patients who were screened at baseline, there were no consistent or meaningful differences across types of organizations in rates of follow-up screening or improvements in clinical outcomes.

Health Care Costs

The effect of the BHICCI on costs varied by type of organization and service. Inpatient per member per month costs declined by $26 overall and by $224 among patients in multispecialty clinics (exhibit 4). ED costs increased by $7 overall and by $10 among patients in federally qualified health centers. Outpatient costs increased by $67 overall, by $87 among patients in federally qualified health centers, and by $66 among patients in behavioral health clinics. Total costs increased by $189 among patients in federally qualified health centers. The average cost of staffing the BHICCI team was $266 per member per month (data not shown).

SOURCE Authors’ analysis of data from the Inland Empire Health Plan (IEHP). NOTES A two-part, generalized linear model was used to examine the use of services and costs among Behavioral Health Integration and Complex Care Initiative (BHICCI) patients compared to a matched comparison group of IEHP enrollees identified using propensity scores and nearest neighbor matching. Methods are described more fully in the text. FQHC is federally qualified health center. *$p<0.10$ **$p<0.05$ ***$p<0.01$

Patient Experience

Patient Voice:

A recurring theme in the patient focus groups was the perceived benefits of a care manager who would hear patients’ expressed care goals, needs, and preferences and incorporate them as patients navigate the health care system. One patient described the care management experience this way: “I really think they saved my life because at the time I was just so frustrated with trying to tell someone something, and then they’re trying to tell me like they know my body better than I do. …So this team, they came into the doctor’s office—and that’s when they introduced themselves, while I was waiting to see the doctor. They came in before the doctor. And it’s awesome.”

Care Coordination:

Focus-group participants also reported that it was helpful to have a care coordinator to facilitate appointments with primary care providers and specialists. This support was especially helpful when embarking on a relationship with a new type of specialist. Patients particularly valued the care coordinator’s check-in calls.

Communication Among Providers:

BHICCI patients were impressed with the level of communication among their care team members. One patient commented: “I’ve been in pain management for over thirty years…. The fact [is] that there are multiple doctors here that actually talk to each other and kind of bounce ideas off of each other, that they work together to get a treatment program.” A patient from one of the Spanish-language focus groups observed, “All of the information, appointments, doctors, and reports are in harmony.”

A Desire For Direct Personal Involvement In Care:

While patients consistently noted the benefits of a more coordinated health care experience, some articulated a desire to have more direct personal involvement in communication that pertained to their health care needs. One patient suggested that care coordination could be improved by having more frequent meetings with providers to discuss care planning: “I would actually like to meet with all of them at [the] same time…maybe once a quarter…to make sure they all knew what I wanted and all knew the goals set, because maybe they’re having meetings, maybe they’re not. So at least once every three months, or four months, please have all of you for fifteen minutes sitting down in the same room with me.”

Discussion

The Behavioral Health Integration and Complex Care Initiative represented a comprehensive effort by the Inland Empire Health Plan to implement integrated complex care management across a diverse group of health care organizations. The initiative employed an established model of practice change that was augmented with practice coaching. Several indicators suggest that the implementation of the initiative has been successful: the health management of a large population with complex chronic conditions, improvements in clinical indicators for common chronic conditions, and reductions in inpatient costs in some organizations. Patients value the changes they experienced in their health care, including improved care management, care coordination, provider communication, and access to behavioral health care.

The behavioral health clinics participating in the initiative were less successful than the participating multispecialty clinics and federally qualified health centers were, with lower rates of screening and fewer documented improvements in clinical outcomes. This may have been the result of challenges in hiring, credentialing, and retaining primary care providers in the behavioral health clinics, as well as challenges related to providing measurement-based care. Routine primary care workflows readily accommodate repeated measures of clinical outcomes, such as blood pressure, depressive symptoms, and HbA1c. Behavioral health clinics do not typically measure clinical outcomes, so the care teams in those settings were challenged by a lack of familiarity with these measures and the need to develop new workflows to track clinical outcomes.

In addition, the Patient Health Questionnaire-9 might not have been the best measure of mental health outcomes for a population with severe mental illness, including schizophrenia and bipolar disorder as well as severe major depression. The questionnaire is primarily used to screen for depression and measure its severity. Moreover, it is a measure focused on symptoms and deficits, and it lends itself to brief, structured interventions. While this approach is consistent with primary care, it runs counter to the culture of specialty behavioral health care, which involves strengths-based models of recovery and longer courses of treatment. Future efforts should consider flexible partnerships to support a primary care workforce, focused training on measurement-based care specific to behavioral health, and alternative measures of mental health outcomes that are more appropriate for use with a population with severe mental illness.

Significant reductions in inpatient costs were demonstrated only in multispecialty clinics. These settings served a population with a high concentration of patients with complex conditions, which made it more likely that patients who would benefit most from care management would be identified and enrolled in the BHICCI. In contrast, the federally qualified health centers struggled to identify and enroll appropriate patients. This highlights the need to clearly define which patients in which settings would benefit most from complex care management and to determine the most cost-effective strategies for identifying these patients. The cost data did not include information about psychiatric admissions, so we were less likely to observe reductions in inpatient costs among patients in behavioral health clinics.

The ACA created an optional Medicaid state plan benefit to establish health homes to coordinate care for Medicaid beneficiaries with multiple chronic conditions. California’s Health Homes Program is designed to serve Medi-Cal beneficiaries with multiple chronic conditions who are frequent utilizers and who may benefit from care management and coordination. A primary goal of the BHICCI was to prepare participating organizations to become eligible for funding as health homes. Thus, BHICCI eligibility criteria specified having multiple chronic conditions including one medical and one behavioral health condition. Additionally, BHICCI services are consistent with those covered under the Health Homes Program: comprehensive care management, care coordination, health promotion, comprehensive transitional care, individual and family support, and referral to community and social support services such as housing. Health care provided under the BHICCI is also consistent with a whole-person philosophy, whereby providers integrate and coordinate physical and behavioral health care—including substance use treatment—and long-term services and supports to treat the whole person.

Our evaluation of the BHICCI had several limitations. First, the timeline for the analysis was one year, whereas the initiative’s effectiveness may increase over time. Second, propensity scores were used to identify a comparison group. However, the comparison sample might have differed in unobservable characteristics that could have biased the estimates of the BHICCI’s effect on health care costs. Third, the cost analysis was limited to services covered by the IEHP and thus did not include behavioral health care. Fourth, because of the focus on clinical outcomes for common chronic conditions, the evaluation might have missed changes in clinical outcomes related to complex chronic conditions. Finally, BHICCI patients might have benefited from care management in ways that were not captured by the focus groups.

Despite these limitations, the BHICCI evaluation provides useful insights into the reality of systems-level practice change across multiple complex organizations. Over the implementation period, the BHICCI transitioned from a pilot program to a full-scale initiative of the IEHP. Activities that were performed by outside consultants have now been institutionalized by IEHP staff. These include internalizing the tracking of clinical outcomes data, hiring some of the lead consultants who designed and implemented the BHICCI to fill key plan leadership positions, and hiring practice coaches for continued support of the sustainability and spread of practice change. The BHICCI represents a new type of partnership between the IEHP and its providers and patients that has the potential to transform care for Medi-Cal beneficiaries with complex chronic conditions.

Conclusion

The BHICCI has demonstrated that it is possible to implement integrated care across multiple provider types in a challenging environment with siloed funding streams. The initiative has shown improvements in multiple clinical outcomes and patient satisfaction and, in some organizations, reductions in inpatient costs. The BHICCI may be better understood as an ongoing and evolving strategic partnership than as a point-in-time intervention. Changes in the funding model to support value-based care will be required to sustain these efforts in the long term.

ACKNOWLEDGMENTS

An earlier version of this article was presented as part of the Michael M. Davis Lecture Series at the Center for Health Administration Studies, University of Chicago. Todd Gilmer, Elizabeth Siantz, Benjamin Henwood, and Kimberly Center received financial support to conduct the evaluation of the Behavioral Health Integration and Complex Care Initiative under contract with the Inland Empire Health Plan. Marc Avery was a consultant to the Inland Empire Health Plan for the implementation of the Behavioral Health Integration and Complex Care Initiative. Elise Pomerance and Jennifer Sayles are employees of the Inland Empire Health Plan. The authors acknowledge the extraordinary vision and transformative leadership of Bradley Gilbert, whose steadfast support of this initiative from conception to planning, implementation, and evaluation made it all possible.

NOTES

• 1 Decker SL, Lipton BJ, Sommers BD. Medicaid expansion coverage effects grew in 2015 with continued improvements in coverage quality. Health Aff (Millwood). 2017;36(5):819–25. Go to the article, Google Scholar
• 2 Han X, Luo Q, Ku L. Medicaid expansion and grant funding increases helped improve community health center capacity. Health Aff (Millwood). 2017;36(1):49–56. Go to the article, Google Scholar
• 3 Takach M. About half of the states are implementing patient-centered medical homes for their Medicaid populations. Health Aff (Millwood). 2012;31(11):2432–40. Go to the article, Google Scholar
• 4 Felland L, O’Malley A, Cross D, Perera I (Center for Studying Health System Change, Washington, DC). Riverside/San Barnardino: vast region, market fragmentation add to access woes [Internet]. Oakland (CA): California HealthCare Foundation; 2012 Sep [cited 2018 Jun 22]. (Regional Markets Issue Brief). Available from: https://www.calhospital.org/sites/main/files/file-attachments/chcf_regional_market_report_riverside_san_bernardino_september_2012.pdf Google Scholar
• 5 Gilmer TP, Henwood BF, Goode M, Sarkin AJ, Innes-Gomberg D. Implementation of integrated health homes and health outcomes for persons with serious mental illness in Los Angeles County. Psychiatr Serv. 2016;67(10):1062–7. Crossref, Medline, Google Scholar
• 6 Bodenheimer T (University of California San Francisco). Strategies to reduce costs and improve care for high-utilizing Medicaid patients: reflections on pioneering programs [Internet]. Hamilton (NJ): Center for Health Care Strategies, Inc.; 2013 Oct [cited 2018 Jun 22]. (Policy Brief). Available from: https://www.chcs.org/media/HighUtilizerReport_102413_Final3.pdf Google Scholar
• 7 Bleich SN, Sherrod C, Chiang A, Boyd C, Wolff J, DuGoff Eet al. Systematic review of programs treating high-need and high-cost people with multiple chronic diseases or disabilities in the United States, 2008–2014. Prev Chronic Dis. 2015;12:E197. Crossref, Medline, Google Scholar
• 8 McCarthy D, Ryan J, Klein S. Models of care for high-need, high-cost patients: an evidence synthesis [Internet]. New York (NY): Commonwealth Fund; 2015 Oct 29 [cited 2018 Jun 22]. Available from: https://www.commonwealthfund.org/publications/issue-briefs/2015/oct/models-care-high-need-high-cost-patients-evidence-synt Google Scholar
• 9 California Department of Health Care Services. Health homes for patients with complex needs: California concept paper—final [Internet]. Sacramento (CA): DHCS; 2016 Mar [cited 2018 Jun 22]. Available from: http://www.dhcs.ca.gov/services/Documents/HealthHomesforPatients_Final.pdf Google Scholar
• 10 Balasubramanian BA, Cohen DJ, Davis MM, Gunn R, Dickinson LM, Miller WLet al. Learning Evaluation: blending quality improvement and implementation research methods to study healthcare innovations. Implement Sci. 2015;10(10):31. Crossref, Medline, Google Scholar
• 11 Institute for Healthcare Improvement. The Breakthrough Series: IHI’s collaborative model for achieving breakthrough improvement. Boston (MA): IHI; 2003. Google Scholar
• 12 Baskerville NB, Liddy C, Hogg W. Systematic review and meta-analysis of practice facilitation within primary care settings. Ann Fam Med. 2012;10(1):63–74. Crossref, Medline, Google Scholar
• 13 Berwick DM, Nolan TW, Whittington J. The Triple Aim: care, health, and cost. Health Aff (Millwood). 2008;27(3):759–69. Go to the article, Google Scholar
• 14 Langley GJ, Moen RD, Nolan KM, Nolan TW, Norman CL, Provost LP. The improvement guide: a practical approach to enhancing organizational performance. 2nd ed. San Francisco (CA): Jossey-Bass; 2009. Google Scholar
• 15 Rubin DB. Estimating causal effects from large data sets using propensity scores. Ann Intern Med. 1997;127(8 Pt 2):757–63. Crossref, Medline, Google Scholar
• 16 Rubin DB, Thomas N. Matching using estimated propensity scores: relating theory to practice. Biometrics. 1996;52(1):249–64. Crossref, Medline, Google Scholar
• 17 Kronick R, Gilmer T, Dreyfus T, Lee L. Improving health-based payment for Medicaid beneficiaries: CDPS. Health Care Financ Rev. 2000;21(3):29–64. Medline, Google Scholar
• 18 To access the Appendix, click on the Details tab of the article online.
• 19 Manning WG, Basu A, Mullahy J. Generalized modeling approaches to risk adjustment of skewed outcomes data. J Health Econ. 2005;24(3):465–88. Crossref, Medline, Google Scholar
• 20 Efron B, Tibshirani RJ. An introduction to the bootstrap. Boca Raton (FL): Chapman and Hall; 1993. Crossref, Google Scholar
• 21 Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health. 2015;42(5):533–44. Crossref, Medline, Google Scholar
• 22 Patton MQ. Qualitative research and evaluation methods. 3rd ed. Thousand Oaks (CA): Sage Publications; 2002. Google Scholar