{"subscriber":false,"subscribedOffers":{}} Payment Reform In Transition—Scaling ACOs For Success | Health Affairs

If any vignette defines the provider experience in a Medicare accountable care organization (ACO), it is probably the anxious waiting, digesting, and analyzing of the Centers for Medicare and Medicaid Services (CMS) quarterly reports of ACO performance against its benchmark. In offices across the United States, ACO executive directors and analytics teams pore over tables and cross-tabs trying to assess their results, asking “are we meeting our goals for the year? Are we on track to generate shared savings?”

The reality is that most of these conversations are moot because the vast majority of Medicare ACOs are too small to conclude with any certainty that their measured performance is reflective of their true performance, instead of the result of statistical variation.

Fortunately, this problem is solvable but only if ACOs are large enough to reliably interpret the performance of their networks. While the majority of ACOs do not serve large enough populations to manage the number of lives needed for statistical reliability, providers can do so without acquisition, consolidation, or any other threat to independence by creating collaborative ACOs. Collaborative ACOs—which unlike other new Medicare concepts such as “virtual groups” already exist—are particularly effective in supporting smaller and rural providers for whom ACO participation would normally be out of reach, allowing them to leverage a shared pool of resources and relieving them of pressure to join a regional mega-system. Participants in collaborative ACOs do not need to share patients to be successful, or even be contained within state lines. They do need shared governance, however, and a mutual commitment to measurable clinical objectives with transparency and accountability to their partners.

Tensions Facing ACOs In 2018

For more than 10 years, ACOs have been touted as a key to health system transformation. CMS launched the ACO concept more than six years ago, kicking off Pioneer ACOs in January 2012 and opening the Medicare Shared Savings Program (MSSP) later that same year. CMS has experimented with at least six different types of ACOs since 2012. David Muhlestein, Robert Saunders, and Mark McClellan reported that the number of ACOs grew from 61 in 2011 to 923 in 2017, including both Medicare and other contracts, covering 32 million lives. CMS has nurtured the growth of the MSSP through several channels, including by offering small and rural providers start-up loans; a “no risk” upside-only track in which participants can receive a share of any savings achieved but are not penalized for overages; favorable treatment under the new Merit-Based Incentive Payment System (MIPS); learning networks; and technical support.

Despite the widespread adoption of ACOs—with more than 550 currently in the MSSP alone—opinions remain mixed as to whether these entities have moved the needle on health care costs and quality. The secretary of the Department of Health and Human Services (HHS), Alex Azar, has acknowledged concerns with provider engagement and ACO results, while reasserting HHS’s commitment to its health care transformation efforts: “Results for the early stages of federal efforts to encourage accountable care organizations have been, to be honest, underwhelming… . But there is no turning back to an unsustainable system that pays for procedures rather than value. In fact, the only option is to charge forward—for HHS to take bolder action, and for providers and payers to join with us.”

We propose that the secretary follow his call for bold action by taking this opportunity to apply lessons learned from the first six years of ACO operations. Participants in ACOs represent the largest cadre of providers in payment-reform initiatives to date. The doctors and hospitals in Medicare ACOs currently provide care to more than 10 million beneficiaries, or nearly one-third of the program’s fee-for-service enrollees. Safety-net providers are well represented in the models, including federally qualified health centers, critical access hospitals, and rural health clinics. Given its success in attracting providers and improving patient care, we encourage HHS to focus its efforts on the MSSP and on assisting providers who have not yet joined with learning how to succeed in population health models.

Below, we describe the lessons learned from Caravan Health’s experience managing 38 ACOs across the country that serve more than one million Medicare patients, and we offer our recommendations for improving the success and sustainability of the program within the current regulations.

Lessons Learned From The Shared Savings Program

Shared Savings Don’t Always Come From Improvement In Performance (And Vice Versa)

It happens every year—the provider that did everything right comes in over the benchmark for the year. Alternatively, the one that made no meaningful changes somehow earns savings. The following year, winners and losers change places. In the Shared Savings Program, the only certainty is that someone will be surprised by the year’s end.

CMS does its best to weed out random winnings and losings. To avoid handing out “unearned” shared savings, particularly in the no-risk Track 1, CMS requires savings to exceed anywhere from 2.0 percent to 3.9 percent before they are shared. However, even those corridors allow for a large degree of random earnings due to chance each year. In the initial rulemaking for the program, CMS accepted that an ACO with 5,000 lives would have a 10.0 percent chance of random winnings in any given year in establishing the 3.9 percent threshold, while a 50,000-life ACO would only have a 1.0 percent chance.

Participants in the ACO program face great frustration from the high variability of spending outcomes in the program. Many participants, eager to deliver better preventive care for their patients and believing that 5,000 lives is sufficient, form an ACO with the best intentions. However, the unpredictability of these results indicates that investments cannot reliably deliver returns, meaning many ACOs that are making the right choices are getting the “wrong” results. The effect of statistical variation on these small ACOs can create spurious results that wrongly penalize or reward participants, and offer little insight into how well population health initiatives may be working.

Most existing ACOs are too small to reliably interpret performance in any one year relative to their own benchmark. In 2016, the most recent year of published data by CMS, 73 percent of Medicare Shared Savings Program ACOs had fewer than 20,000 attributed lives. The roughly 400 ACOs with fewer than 20,000 lives routinely experience savings and losses of 10 percent to 20 percent simply due to statistical variation in health care spending. Exhibit 1 displays the performance of ACOs in the Shared Savings Program against their benchmarks over 2013–16. As Exhibit 2 illustrates, the average deviation from the mean for ACOs with fewer than 10,000 lives is 5 percent, larger than the minimum savings/loss rate for ACOs of that size, with a range of variation of greater than 50 percent overall.

Exhibit 1: ACO Savings Or Losses As A Percent Of Benchmark Versus Attributed Lives, 2013–16

Source: Authors’ analysis of CMS Medicare Shared Savings Program ACO Public Use File.

Exhibit 2: Savings And Losses As A Percent Of Benchmark By Number Of Attributed Lives

Source: Authors’ analysis of Medicare Shared Savings Program ACO Public Use File.

Examining the year-over-year performance of ACOs tells the same story (see Exhibit 3). An ACO should ideally achieve consistent improvement over the three-year agreement period. In fact, for smaller ACOs, performance changes randomly year over year. In the period between 2013 and 2016, 45 percent of ACOs saw their savings rate go up or down by more than three percentage points, including 21 percent that saw changes in excess of 5 percentage points.

Exhibit 3: Year-Over-Year Changes In ACO Savings Or Losses As A Percent Of Benchmark Versus Attributed Lives, 2013–16

Source: Authors’ analysis of CMS Medicare Shared Savings Program ACO Public Use File.

Exhibit 4 displays the statistical confidence that an ACO’s true performance will fall within a range of measured performance. For example, an ACO with 10,000 lives that identifies and executes on 4 percent of savings opportunities can only be 90 percent confident that it will beat its benchmark, and there is a greater than 25 percent chance it will not beat its minimum savings rate of 3 percent. Even at 30,000 lives, an ACO that is performing “well enough” to save 4 percent—a very significant sum for one year—will have barely 75 percent confidence it will beat its 2.4 percent mid-session review. At 100,000 lives, however, an ACO can be 90 percent confident that its program results will fall within about 1 percent of its true performance.

Exhibit 4: Confidence Interval Around Savings Rates In The MSSP Program 2014–15 Versus ACO Attributed Lives

Source: Sampling of Medicare 5 percent claims file from 2014 and 2015 performed by external actuaries under contract to Caravan Health.

Collaboration Does Not Require Consolidation

The hospital industry is consolidating, and consolidation is driving concern about pricing and anti-competitive behaviors. (It is important to note that despite significant concern about ACOs potentially increasing the market power of hospitals, the only study of which we are aware on the question found no evidence of this occurring in practice.) These trends have led many to prioritize the survival of independent physician practices, but the same argument applies for the need to maintain independent hospitals. Those hospitals feel the same pressures for consolidation affecting the industry at large. Much as small physician practices are seeking new care models and shared services to add capabilities and reduce costs, hospitals outside large systems also need to use shared services to gain the advantages of scale while maintaining independence. 

CMS has expressed interest in supporting independent practices, presumably due in part to concerns about consolidation. Policy makers need to create appropriate incentives for independent providers to participate in payment reform programs while avoiding the pressure to consolidate. We have seen strong interest from rural providers in participating in care transformation, and success often follows when they try. Caravan’s small and rural clients have achieved excellent clinical quality scores above national averages even as they beat their spending benchmarks.

Alternatives to consolidation that do not require succumbing to random statistical variation already exist. “ACO enablers” including Caravan Health, Aledade, Collaborative Health Systems, Evolent, Imperium, Premier, and others have designed different models to support providers in ACOs through shared services. In the case of Caravan Health, our participating hospitals faced a dilemma of shared performance among unaffiliated providers and needed to ensure that all members “pulled their weight,” while avoiding free-loading in collaborative ACO groupings. Over time, we developed shared governance and objective metrics, leading indicators, and scorecards that assessed each member’s effort and demonstrated change—and tied those metrics to the overall success of the ACO.

CMS has recognized the role of convening organizations in multiple contexts and has created explicit roles for “conveners” in bundled payment programs. Research has also noted that shared services organizations support the continued existence of small, physician practices. We believe that shared services across many hospitals and physician practices offers one viable path forward for the ACO program. These collaborative organizations can solve the problem of scale for ACOs without the need for acquisition or consolidation.

Two-Sided Risk Models Are Not Sufficiently Attractive To Bring A Critical Mass Of Providers Into A Voluntary Program

While Congress and CMS increasingly emphasize two-sided risk models as the future of health care, the statistical disconnect between performance and results means that few ACOs are well-positioned to consistently succeed in risk-bearing models. Although CMS apparently wants providers to participate in two-sided risk ACO contracts, it has not yet created a sufficiently attractive offer (or downside to staying out) that will outweigh the consequence of writing a large check to CMS. Similarly, creating more barriers to participating in the ACO program in a “no risk” arrangement without strong rewards for participation will simply drive participants out of the model. Each year, usually around the time when CMS gives previews of new benchmarks, some ACOs decline to enter the program or leave it after one or two years of experience. In March 2018, seven of 58 risk-bearing Next Generation ACOs left the program amid a common refrain of finger pointing and disagreement over risk coding policy.

Consider the 101 ACOs participating in risk-bearing contracts in 2018. These ACOs presumably entered the program intending to reduce spending, increase quality, and share in 50–75 percent of the savings to the program. Assuming these ACOs perform in line with historical experience, about 10 of them will have spending results sufficiently beyond their minimum loss rates to require them to write a check to CMS—averaging $7 million based on an analysis from Avalere Health.

No hospital or physician practice administrator can sustain participation in a voluntary program if that organization is writing large checks back to a payer, particularly if the ACO cannot explain the loss despite large investments in analytics and change management. ACO’s successes hinge on very thin margins—one evaluation found that in 2014, the program reduced spending in Medicare by approximately 0.7 percent, or $67 per ACO-attributed patient. For ACOs with a minimum loss rate (the mirror image of the minimum savings rate) of 2 percent, most good performers will not have to write CMS a check; but some will, and some small ACOs early in their performance period will be in a repayment position purely due to bad luck. Conversely, with the minimum loss rate set at 2 percent, many good performers are unlikely to get shared savings if true savings is only 0.7 percent, unless they are also lucky by chance.

A Sustainable Alternative: The Large, Collaborative ACO

In our view, the problem of small numbers means that ACO participants need to team up, create larger, collaborative ACOs with shared governance, shared clinical programs, and more than 100,000 lives. The 100,000 lives gives an ACO manager and its participants 90 percent confidence that its measured spending results will be within about 1 percent of its true performance. Accountability for primary care needs to be central to large ACOs, as does robust monitoring, tracking, and continuous quality improvement—activities already undertaken in most ACOs.

This approach offers several advantages to participating physicians and hospitals, including:

  • A more sustainable business model, in which performance is connected to results with greater confidence;
  • Greater likelihood of predictable shared savings, through lower minimum savings rates;
  • A reduction of more than 80 percent of the administrative burden of quality reporting as quality reporting requirements are spread over a larger set of providers; and
  • The ability to assume acceptable downside risk with low minimum savings rates.

For policy makers, this model also offers important benefits, including:

  • A model that can help sustain the largest ongoing initiative in payment reform and care transformation by reducing random variation in savings and losses;
  • A mechanism to support rural and independent community hospitals and clinicians that would otherwise face overwhelming pressure to join or be sold to large systems, reducing pressure to consolidate; and
  • Better accountability for program participants, as deficits in performance will directly implicate the ACO’s strategy and execution in the performance year and can’t be chalked up to bad luck.


CMS should recognize some of the challenges inherent in the ACO program, chief among them the fact that most ACOs are simply too small to prosper as currently constructed and will never be suited to take risk. Health care providers and CMS should support large ACOs built out of independent practices and hospitals and embrace them as a path to sustainably improve quality and control costs for independent practices, community hospitals, rural providers, safety-net facilities, and the millions of patients they serve.

Authors’ Note

Thanks to Louise Yinug for assistance in the preparation of this post.



Peterson Logo

Featured Blog topic supporter. Learn more.

Sponsored Content: PwC