Map Map


What is known about the impacts of the different policy options?

Finding systematic reviews

Systematic reviews are the ideal starting point for finding out what is known about the impacts of potential options, and the number of these that relate to health system arrangements and implementation strategies is increasing. When searching for systematic reviews that address the impacts of health system arrangements, Health Systems Evidence is a good place to start. Other sources that can be searched for systematic reviews of the impacts of health system arrangements and implementation strategies include The Cochrane Library and PubMed. A list of databases and recommended strategies for searching them are described in the ‘Additional resources’ of this guide.  The EVIPNet Virtual Health Library (www.evipnet.org) is being developed to provide quick and easy access to evidence for informed decisions about health systems in low- and middle-income countries. It will provide a meta-search engine to facilitate searching these and other databases.

SUPPORT has prepared concise summaries of the best available evidence of the effects of health systems interventions for low and middle-income countries. These summaries are provided in the Libraries section of these guides.

A search log, Workshop materials and a PowerPoint presentation on finding systematic reviews are available in the ‘Additional resources’ section of this guide. The SUPPORT Tool on finding systematic reviews offers further guidance.


Selecting systematic reviews

Once potentially relevant references have been found, it is then necessary to decide which systematic reviews to retrieve in full text form and examine in detail. Explicit selection criteria should be used to guide these judgements and to make the selection process as transparent as possible. These criteria might specify, for example:

It may sometimes be appropriate to have a selection criterion that relates to the actual setting of the review (e.g. in primary care). However only considering reviews that are restricted to specific countries or low- and middle-income countries runs the risk of excluding important evidence (which may sometimes be the best available). Including reviews of studies from diverse settings and subsequently assessing the applicability (see Box 4.2) of the results to your setting is likely to be more informative.


Box 4.2 Judgements about the applicability of the results of systematic reviews

Decisions about how to apply the results of systematic reviews are always located within specific contexts. This makes it necessary to make judgements about possible differences between where the research that is summarised in a particular systematic review was done and your own setting. For health system interventions this includes considerations of differences, for example, between:

  • The structural elements of health systems (such that an intervention could not work in the same way)

  • On-the-ground realities and constraints (that might substantially alter the potential benefits of the intervention)

  • Perspectives and influences of health system stakeholders (such that the intervention may not be accepted or taken up in the same way)

In addition, different baseline conditions may result in different absolute effects, even if the relative effectiveness is the same. While this may not lower your confidence in the evidence, it is important to keep this in mind when applying the results of studies from one setting to another.


More guidance on the applicability of the results of systematic reviews can be found in the SUPPORT tool provided in the ‘Additional resources’ section of this guide.


Making judgements about how much confidence to place in a systematic review

Once a relevant systematic review has been found, a decision must be taken about how much confidence to place in this review. This judgement is different to the judgement about how much confidence to place in the evidence, and this separate issue is addressed in the next section.

A number of checklists are available to guide assessments of the reliability of systematic reviews. The ‘Additional resources’ section of this guide contains a sample checklist adapted from the one used to prepare the SUPPORT summaries and other similar checklists.2,3,4 This checklist is designed to guide judgements about whether a review is likely to provide a reliable summary of the best available evidence of the impacts of these complex interventions.5,6 

The checklist is divided into two parts. The first assesses the methods used to identify, select, and critically appraise studies. The second assesses the methods used to analyse the results of the included studies. Summary assessments based on the questions in each section determine whether a review is rated as having minor, moderate, or major limitations. After this, an overall assessment is made based on the two summary assessments (and other potential limitations). Reviews are placed into one of the three categories listed below that then determine how the reviews are utilised in policy briefs:

If a systematic review without important limitations cannot be found, it may be necessary to search for individual studies instead; these can either supplement information in a review or take the place of a systematic review. In such instances, the same processes should be used as are used to select the studies in a systematic review. As far as possible, systematic and transparent (explicit) methods should be used to find, select, and critically appraise studies and to synthesize their results. Ideally the methods used to do this should be described in an appendix document of the policy brief. It should be remembered however, that such additional processes are likely to require substantial additional resources, and specific additional skills will be needed.

Workshop materials and a PowerPoint presentation on deciding how much confidence to place in a systematic review are available in the ‘Additional resources’ section of this guide. The SUPPORT tool on deciding how much confidence to place in a systematic review offers further guidance.


This page was last updated November 2011.