Embargoed Until:
August 11, 2009
12:01 a.m. Eastern Time



Christopher Fleming

Are Health Policy Researchers Doing A Good Job Of Providing Decisionmakers With Relevant And Reliable Information?

Researchers Say Studies Are Too Often Flawed And Insufficiently Targeted Toward The Needs Of Decisionmakers

Bethesda, MD -- As the health reform debate proceeds and private-sector payers face ever-increasing financial pressures, relevant and timely health policy research is more important than ever. But according to two studies published today on the Health Affairs Web site, research is too often flawed and poorly targeted toward the questions that are most important to decisionmakers.

"The potential for a disconnect between the information needs of decisionmakers and the research being produced for them may stem, in large part, from fundamental differences in the priorities and cultures of researchers and decisionmakers, whether from the public or private sector," say Claudia Schur, vice president and director of the Center for Health Research and Policy, Social and Scientific Systems (SSS), in Silver Spring, MD, and coauthors in a study supported by the California HealthCare Foundation (CHCF). Schur and her colleagues point to the need for "decision-based evidence making," defined by CHCF President Mark Smith as seeking "to figure out which decisions need making, and what evidence is necessary in order to make them in the real world, in our lifetime." http://content.healthaffairs.org/cgi/content/abstract/hlthaff.28.5.w886

Schur and her colleagues say that more attention has been paid to the research needs of public-sector health decisionmakers than to the needs of their private-sector counterparts. As a first step toward learning about the research needs of private-sector decisionmakers, the researchers designed a multi-step inquiry culminating in an online survey of respondents from California’s larger health plans and employers.

Respondents were asked to rank twenty issues in terms of importance to the decisions they are required to make in their current positions. "Engaging consumers in their own care management" was ranked the highest, followed closely by "understanding underlying drivers of health care costs." For more than half of the twenty issues -- and three of the five issues ranked highest in importance – fewer than 10 percent of respondents reported that existing research was adequate for their needs.

Because the health care landscape continually evolves, and the issues that are important can change, there is a need for a long-term, "systematic approach to linking the needs of private-sector decisionmakers, funders of research, and those who conduct the research." Two such strategies garnered majority support from survey respondents: 58 percent of respondents "strongly supported both the sharing of data by employers and health plans in order to create new sources of information, and the creation of a clearinghouse for compiling, assessing, and disseminating research, with the clearinghouse sponsored by a foundation or another nonprofit organization," say Schur and coauthors Marc Berk of SSS, former SSS research analyst and current Emory University doctoral student Lauren Silver, Jill Yegian of CHCF, and Michael O’Grady of NORC in Bethesda, MD.

Health Policy Research Studies Are Too Often Fundamentally Flawed

Because the designs of many health policy research studies are fundamentally flawed, policymakers and the public do not always know if they can believe even work appearing in the highest-impact journals, Sumit Majumdar and Stephen Soumerai say in a second study published today on the Health Affairs Web site. http://content.healthaffairs.org/cgi/content/abstract/hlthaff.28.5.w886

Majumdar and Soumerai reject the argument that only randomized controlled trials (RCTs) produce trustworthy evidence. They say that RCTs are rarely possible in health policy research, and they defend the use of valid nonrandomized or quasi-experimental study designs that seek to learn from the natural experiments created by health policy interventions.

Insisting on the exclusive use of RCTs "inappropriately lumps together valid evidence from strong nonrandomized designs (that is, before-and-after studies with concurrent controls or the interrupted time series study in which a policy causes a sudden, visible change in trend) with evidence from weak designs that permit little causal influence (that is, the commonly conducted cross-sectional analysis that looks at outcomes only after a policy has been implemented)," say Majumdar, an associate professor in the Department of Medicine, University of Alberta in Edmonton, and Soumerai, a professor in the Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, in Boston, MA.

The problem, say Majumdar and Soumerai, is that researchers too often use weak study designs. For example, according to a recent systematic review of 13 studies of financial incentives for physicians to improve the quality of pharmaceutical care, not one study "met minimal methodological standards to ensure validity. … Most consisted of cross-sectional analyses or longitudinal studies comparing providers’ activities before and after financial incentives were introduced, without controlling for pre-existing trends in study outcome or important factors (‘confounders’) that might have accounted for the policy ‘effects.’"

Similarly, under economic stimulus legislation passed earlier this year, the government is spending tens of billions of dollars on nationwide implementation of predominately commercial health information technology systems, despite the fact that this has not yet been proven to increase the quality or quantity of life, the two researchers say.

Majumdar and Soumerai suggest a number of potential steps "to improve the health of health policy research," including providing researchers, clinicians, and policymakers "with a more solid grounding in the comparative rigor of various quasi-experimental policy designs." They also call for better visual displays of time trends before and after policy implementation, and other strategies.

In addition, Majumdar and Soumerai suggest the creation of "secondary review" health policy journals, akin to highly trusted publications such as ACP Journal and Journal Watch that exist in the clinical world. These publications "apply quality filters and review many articles; summarize the results of important studies; and comment on the validity and broader context of the work," the two researchers explain.

After the embargo lifts, you can read the article by Schur and coauthors at http://content.healthaffairs.org/cgi/content/abstract/hlthaff.28.5.w886

You can read the article by Majumdar and Soumerai at http://content.healthaffairs.org/cgi/content/abstract/hlthaff.28.5.w900


Health Affairs, published by Project HOPE, is the leading journal of health policy. The peer-reviewed journal appears bimonthly in print with additional online-only papers published weekly as Health Affairs Web Exclusives at www.healthaffairs.org.


©2009 Project HOPE–The People-to-People Health Foundation, Inc.