Advertisement
Policy Forum

Policy Forum Policy Forum articles provide a platform for health policy makers from around the world to discuss the challenges and opportunities in improving health care to their constituencies.

See all article types »

How Can We Support the Use of Systematic Reviews in Policymaking?

  • John N. Lavis mail

    lavisj@mcmaster.ca

    Affiliations: McMaster Health Forum, McMaster University, Hamilton, Ontario, Canada, Centre for Health Economics and Policy Analysis, McMaster University, Hamilton, Ontario, Canada, Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada, Department of Political Science, McMaster University, Hamilton, Ontario, Canada

    X
  • Published: November 17, 2009
  • DOI: 10.1371/journal.pmed.1000141

Summary Points

  • Policymakers need many types of research evidence—synthesized and packaged for them—and the use of this evidence supported in multiple complementary ways. Stakeholders who seek to influence the policymaking process have the same requirements.
  • Policymakers and stakeholders need many types of systematic reviews. For example, reviews of qualitative studies can help to identify alternative framings of the problem, to understand how or why a policy or program option works, and to appreciate stakeholders' perspectives on particular options.
  • Policymakers and stakeholders now have access to many review-derived products: (1) summaries of systematic reviews highlighting decision-relevant information; (2) overviews of systematic reviews providing a “map” of the policy questions addressed by systematic reviews and the insights derived from them; and (3) policy briefs drawing on many systematic reviews to characterize a problem, policy or program options to address the problem, and implementation strategies.
  • A range of activities are being undertaken to support the use of reviews and review-derived products in policymaking, all of which warrant rigorous evaluation.
  • Future challenges include: (1) examining whether and when any apparent duplication of efforts occurs in the production of review-derived products at the international level; and (2) scaling up activities that are found to be effective in supporting the use of reviews and review-derived products in policymaking.

Introduction

In the last few years the landscape has changed dramatically for policymakers seeking to use research evidence in the policymaking process. The landscape has also changed for the many stakeholders seeking to use research evidence to influence the policymaking process. The task once seemed overwhelming given the dearth of synthesized research evidence on the “big”, typically multifaceted, questions that matter to policymakers and stakeholders [1],[2]. Now it isn't uncommon for these groups to find dozens of systematic reviews that address the governance, financial, and delivery arrangements within health systems that can determine whether a cost-effective program, service, or drug reaches those who need it. For example, teams of African policymakers, stakeholders, and researchers drew on 30 reviews for what at first glance seems a straightforward question: how to support the widespread use of artemisinin-based combination therapy (ACT) to treat uncomplicated falciparum malaria. The review of qualitative studies of people's views about and experiences with medicine sellers provided insights that were as central to the process as reviews of the effectiveness of a particular ACT formulation or the home-based management of malaria [3][5].

For policymakers and stakeholders the challenge in using research evidence has shifted from making the best possible use of local studies to: (1) finding systematic reviews that address their many questions related to the policy issue at hand; (2) deriving insights from the reviews for a particular context (which may differ from where the studies included in the review were conducted); and (3) combining these insights with the insights from local data and studies and from local tacit (“how to”) knowledge and other forms of knowledge [1],[6][8]. Policymakers and stakeholders then need many types of systematic reviews, for these reviews to be packaged in different ways in order to facilitate their use in policymaking, and more generally for their use of the reviews to be supported in multiple complementary ways. The same holds true for health system managers, including those working in hospitals, nongovernmental organizations, and many others settings. In some countries these managers are counted as policymakers, and in others they are counted as stakeholders. In all countries they are decision makers in their own right. Much of what follows applies to health system managers as well.

Need for Many Types of Reviews

Policymakers and stakeholders need many types of reviews to inform any given policymaking process (Table 1). For example, reviews of observational studies can help to establish the magnitude of the problem (or the factors that contribute to it), characterize the harms and key elements of policy and program options for addressing the problem, and identify potential barriers to implementing a preferred option [9],[10]. Reviews of qualitative studies can help to identify alternative framings of the problem, understand how or why a policy or program option works (particularly if local adaptation is being considered), appreciate stakeholder's views about and experiences with particular options, and identify potential barriers to implementing a preferred option [11][13]. Reviews of effectiveness studies can help to characterize the benefits and sometimes the harms of each option being considered [14]. And finally, reviews of economic evaluations can help to characterize the cost-effectiveness of options [15]. Policymakers and stakeholders can find increasing numbers of all of these types of reviews.

thumbnail

Table 1. Examples of the types of systematic reviews needed in different steps in the policymaking process.

doi:10.1371/journal.pmed.1000141.t001

Of course the insights derived from these reviews must compete with many other factors in the policymaking process, such as institutional constraints, interest group pressure, citizens' values, and other types of information like policymakers' past experiences [16]. But some of these types of reviews can provide helpful ammunition in a fight over problem definition or helpful background for a discussion with stakeholders about a problem or about possible policy and program options to address it. Moreover, the strategic use of reviews during “windows of opportunity” created by political events, such as the election of a new government or an interest group pressure campaign, can help to push some problems or options higher or lower on the agenda [17]. Systematic reviews can also be drawn upon to develop a monitoring and evaluation plan when there is substantial uncertainty about what can be expected from a policy or program.

Growing Availability of Review-Derived Products

Policymakers and stakeholders now have access to at least three types of review-derived products: (1) summaries of systematic reviews that highlight decision-relevant information; (2) overviews of systematic reviews that provide a “map” of what policy questions have been addressed by systematic reviews and where additional reviews are needed and that derive insights from these reviews; and (3) policy briefs that draw on many systematic reviews to better understand a problem, policy or program options to address the problem, and possible implementation strategies for these options (Table 2). Some summaries add significant value to a review by highlighting key findings, evaluating the review's quality [18],[19], grading the strength of evidence contained in it [20], identifying local applicability and equity considerations [21][23], and/or providing commentaries by select users of the reviews. Identifying local applicability considerations is particularly important for reviews that address governance, financial, and delivery arrangements in health systems because the lessons learned from these reviews are likely to be context-sensitive [1],[24]. One key challenge lying ahead will be to ensure that summary-production processes keep up with the review-production/updating processes, which should include working through whether and when an apparent duplication of effort in the production of these summaries at the international level is problematic, and not simply the result of experimentation or appropriate targeting to particular audiences and contexts.

thumbnail

Table 2. Examples of review-derived products targeted at policymakers and stakeholders.

doi:10.1371/journal.pmed.1000141.t002

Overviews of systematic reviews can add value in some of the same ways as summaries of systematic reviews, albeit with a greater emphasis on breadth of coverage (e.g., the range of policy and program options examined) than depth of coverage (e.g., detail about what is known about any one option). Policy briefs, on the other hand, start with a policy issue, not with the reviews that researchers happen to have produced. The few current producers of policy briefs (as we have defined them in Table 2) have differed in their jurisdictional focus (e.g., whether the effort to contextualize the research evidence focuses on a country or a whole region) and in whether the briefs are seen as an end in themselves or as an input to one or more “deliberative dialogues.” Such dialogues typically involve convening one to two dozen policymakers and stakeholders to work through a policy issue, drawing on both the policy brief and their own and others' tacit knowledge about the issue.

Activities That Could Support the Use of Systematic Reviews

A range of activities are being piloted to support the use of reviews and review-derived products in policymaking (Table 3) [25],[26]. Few evaluations of the effectiveness of these activities have been undertaken; however, a group led by researchers at the Melbourne School of Population Health is registering a title for a systematic review on this topic with the Cochrane Collaboration. All that is available to inform decisions about the relative emphasis to give to these options are systematic reviews of the factors that influence the use of research evidence in policymaking [2],[27],[28]. A small number of factors emerged in these reviews with some frequency:

  • Interactions between researchers and policymakers increased (and a lack of interactions decreased) the prospects for using research evidence, particularly when the interactions were based on informal relationships;
  • Timeliness increased (and a lack of timeliness decreased) the prospects for research use; and
  • Accordance between research evidence and the beliefs, values, interests or political goals, and strategies of policymakers and stakeholders (or when political positions had not yet been taken) increased (and discordance decreased) the prospects for using research evidence.
thumbnail

Table 3. Examples of activities to support the use of systematic reviews and review-derived products by policymakers and stakeholders.

doi:10.1371/journal.pmed.1000141.t003

The importance of interactions underpins efforts by some organizations to engage both researchers and policymakers in priority-setting and/or production activities and in deliberative dialogues. The importance of timeliness underpins efforts to create and continuously update databases that provide “one stop shopping” for optimally packaged reviews and review-derived products, as well as efforts to develop capacity among policymakers to find and use research evidence efficiently (which includes improving their understanding of how many types of systematic reviews are needed to inform any given policymaking process and raising their awareness about the sources of these reviews and review-derived products). The importance of an accordance between research evidence and policymakers' beliefs, values, interests or political goals, and strategies underpins efforts to identify “windows of opportunity” in policymaking processes and use review-derived products as the basis for engaging policymakers and the stakeholders seeking to influence them, as well as efforts to support the “real time” identification of an accordance (e.g., through well-facilitated deliberative dialogues). However, all of these activities warrant evaluation and, if found to be effective, scaling-up in order to support the use of reviews and review-derived products by all policymakers and stakeholders.

Conclusion

Supporting the use of systematic reviews in policymaking has received growing attention in recent years. More of the types of reviews needed by policymakers and stakeholders are being produced, which reduces the burden placed on policymakers and stakeholders to find and synthesize the research evidence on their own. Similarly, more review-derived products targeted at policymakers and stakeholders are being produced, and these products increasingly help to address the unique challenges faced by these groups, namely assessing the local applicability of reviews and mobilizing the range of reviews that are needed in different steps in the policymaking process. Finally, many activities to support the use of reviews are being piloted. Future challenges include working through whether and when an apparent duplication of effort in the production of these summaries is problematic and scaling up activities that are found to be effective in supporting the use of reviews and review-derived products.

Author Contributions

ICMJE criteria for authorship read and met: JNL. Agree with the manuscript's results and conclusions: JNL. Wrote the first draft of the paper: JNL.

References

  1. 1. Lavis JN, Posada FB, Haines A, Osei E (2004) Use of research to inform public policymaking. Lancet 364: 1615–1621.
  2. 2. Lavis JN, Davies HTO, Oxman AD, Denis J-L, Golden-Biddle K, et al. (2005) Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Po 10: S1:35–S1:48.
  3. 3. Goodman C, Brieger W, Unwin A, Mills A, Meek S, et al. (2007) Medicine sellers and malaria treatment in sub-Saharan Africa: what do they do and how can their practice be improved? Am J Trop Med Hyg 77: 203–218.
  4. 4. Omari AAA, Gamble C, Garner P (2006) Artemether-lumefantrine (four-dose regimen) for treating uncomplicated falciparum malaria (Review). Cochrane Database Syst Rev CD005965.
  5. 5. Hopkins H, Talisuna A, Whitty CJM, Staedke SG (2007) Impact of home-based management of malaria on health outcomes in Africa: a systematic review of the evidence. Malar J. 6. 134 p. Available: http://www.malariajournal.com/content/6/​1/134. Accessed 15 July 2009.
  6. 6. Lomas J, Culyer T, McCutcheon C, McAuley L, Law S (2005) Conceptualizing and combining evidence for health system guidance. Ottawa, Canada: Canadian Health Services Research Foundation.
  7. 7. Culyer AJ, Lomas J (2006) Deliberative process and evidence-informed decisions-making in health care: Do they work and how might we know? Evidence & Policy 12: 357–371.
  8. 8. Lavis JN (2006) Moving forward on both systematic reviews and deliberative processes. Healthc Policy 1: 59–63.
  9. 9. Egger M, Davey Smith G, Schneider M (2001) Systematic reviews of observational studies. In: Egger M, Davey Smith G, Altman DG,, editors. Systematic reviews in health care: meta-analysis in context. London: BMJ Publishing Co. pp. 211–227.
  10. 10. Chou R, Helfand M (2005) Challenges in systematic reviews that assess treatment harms. Ann Intern Med 142: 1090–1099.
  11. 11. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A (2005) Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Po 10: 45–53.
  12. 12. Mays N, Pope C, Popay J (2005) Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Po 10: S1:6–S1:10.
  13. 13. Pope C, Mays N, Popay J (2007) Synthesizing qualitative and quantitative health evidence: a guide to methods. Maidenhead (United Kingdom): Open University Press.
  14. 14. Bravata DM, McDonald KM, Shojania KG, Sundaram V, Owens DK (2005) Challenges in systematic reviews: synthesis of topics related to the delivery, organization, and financing of health care. Ann Intern Med 142: 1056–1065.
  15. 15. Pignone M, Saha S, Hoerger T, Lohr KN, Teutsch S, et al. (2005) Challenges in systematic reviews of economic analyses. Ann Intern Med 142: 1073–1079.
  16. 16. Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, et al. (2002) Examining the role of health services research in public policymaking. Milbank Q 80: 125–154.
  17. 17. Kingdon JW (2003) Agendas, alternatives, and public policies. 2nd edition. New York: Longman.
  18. 18. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, et al. (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 7. Available: http://www.biomedcentral.com/1471-2288/7​/10. Accessed 15 July 2009.
  19. 19. Oxman AD, Schunemann HJ, Fretheim A (2006) Improving the use of research evidence in guideline development: 8. Synthesis and presentation of evidence. Health Research Policy and Systems 4. Available: http://www.health-policy-systems.com/con​tent/4/1/20. Accessed 15 July 2009.
  20. 20. Schunemann HJ, Fretheim A, Oxman AD (2006) Improving the use of research evidence in guideline development: 9. Grading evidence and recommendations. Health Research Policy and Systems 4. Available: http://www.health-policy-systems.com/con​tent/4/1/21. Accessed 15 July 2009.
  21. 21. Schunemann HJ, Fretheim A, Oxman AD (2006) Improving the use of research evidence in guideline development: 13. Applicability, transferability and adaptation. Health Research Policy and Systems. Available: http://www.health-policy-systems.com/con​tent/4/1/25. Accessed 15 July 2009.
  22. 22. Oxman AD, Schunemann HJ, Fretheim A (2006) Improving the use of research evidence in guideline development: 12. Incorporating considerations of equity. Health Research Policy and Systems. Available: http://www.health-policy-systems.com/con​tent/4/1/24. Accessed 15 July 2009.
  23. 23. Tsikata S, Robinson V, Petticrew M (2003) Do Cochrane systematic reviews contain useful information about health equity?
  24. 24. Gruen RL, Morris PS, McDonald EL, Bailie RS (2005) Making systematic reviews more useful for policy-makers. Bull World Health Organ 83: 480–481.
  25. 25. Lavis JN, Lomas J, Hamid M, Sewankambo NK (2006) Assessing country-level efforts to link research to action. Bull World Health Organ 84: 620–628.
  26. 26. Colby DC, Quinn BC, Williams CH, Bilheimer LT, Goodell S (2008) Research glut and information famine: making research evidence more useful for policymakers. Health Aff 27: 1177–1182.
  27. 27. Innvaer S, Vist G, Trommald M, Oxman AD (2002) Health policy-makers' perceptions of their use of evidence: a systematic review. J Health Serv Res Po 7: 239–244.
  28. 28. Lavis JN, Hammill AC, Gildiner A, McDonagh RJ, Wilson MG, et al. (2005) A systematic review of the factors that influence the use of research evidence by public policymakers. Final report submitted to the Canadian Population Health Initiative. Hamilton, Canada: McMaster University Program in Policy Decision-Making.
  29. 29. Misso M, Green S, Brennan S, Barnes H (2007) Policy relevant summaries: encouraging and supporting Australian policymakers to use Cochrane reviews.
  30. 30. Tharyan P, Clarke M, Green S (2005) How the Cochrane Collaboration is responding to the Asian tsunami. PLoS Med 2: e169. doi:10.1371/journal.pmed.0020169.
  31. 31. Lagarde M, Palmer N (2006) Evidence from systematic reviews to inform decision making regarding financing mechanisms that improve access to health services for poor people: a policy brief prepared for the International Dialogue on Evidence-Informed Action to Achieve Health Goals in Developing Countries (IDEAHealth). Geneva: World Health Organization.
  32. 32. Chopra M, Munro S, Lavis JN, Vist G, Bennett S (2006) Evidence from systematic reviews of effects to inform policy-making about optimizing the supply, improving the distribution, increasing the efficiency and enhancing the performance of health workers: a policy brief prepared for the International Dialogue on Evidence-Informed Action to Achieve Health Goals in Developing Countries (IDEAHealth). Geneva: World Health Organization.
  33. 33. Chopra M, Munro S, Lavis JN, Vist G, Bennett S (2008) Effects of policy options for human resources for health: an analysis of systematic reviews. Lancet 371: 668–674.
  34. 34. Kakad M, Oxman AD (2006) Evidence from systematic reviews to inform decision-making towards achieving the Millennium Development Goals for reducing maternal and child mortality: a policy brief prepared for the International Dialogue on Evidence-informed Action to Achieve Health Goals in Developing Countries (IDEAHealth). Geneva: World Health Organization.
  35. 35. Althabe F, Bergel E, Cafferata ML, Gibbons L, Ciapponi A, et al. (2008) Strategies for improving the quality of health care in maternal and child health in low- and middle-income countries: an overview of systematic reviews. Paediatr Perinat Epidemiol 22: 42–60.
  36. 36. Lewin S, Lavis JN, Oxman AD, Bastias G, Chopra M, et al. (2008) Supporting the delivery of cost-effective interventions in primary health-care systems in low-income and middle-income countries: an overview of systematic reviews. Lancet 372: 928–939.
  37. 37. Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, et al. (2001) Changing provider behavior: an overview of systematic reviews of interventions. Med Care 39: II:2–II:45.
  38. 38. Ryan R, Santesso N, Hill S, Kaufman C, Grimshaw J (2009) Consumer-oriented interventions for evidence-based prescribing and medicine use: an overview of Cochrane reviews (Protocol). Cochrane Database Syst Rev CD007768.
  39. 39. Pearson SD, Rawlins MD (2005) Quality, innovation, and value for money: NICE and the British National Health Service. JAMA 294: 2618–2622.
  40. 40. Tunis SR, Pearson SD (2006) Coverage options for promising technologies: Medicare's ‘coverage with evidence development’. Health Aff 25: 1218–1230.
  41. 41. Lomas J, Fulop N, Gagnon D, Allen P (2003) On being a good listener: setting priorities for applied health services research. Milbank Q 81: 363–388.
  42. 42. Lomas J (2000) Using ‘linkage and exchange’ to move research into policy at a Canadian foundation: encouraging partnerships between researchers and policymakers is the goal of a promising new Canadian initiative. Health Aff 19: 236–240.
  43. 43. Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J (2003) How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q 81: 221–248.
  44. 44. Lavis JN (2006) Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthc Policy 1: 21–33.
  45. 45. Denis J-L, Lomas J, Stipich N (2008) Creating receptor capacity for research in the health system: the Executive Training for Research Application (EXTRA) program in Canada. J Health Serv Res Po 13: 1–7.