What is known about the impacts of the different policy options?

Finding systematic reviews

Systematic reviews are the ideal starting point for finding out what is known about the impacts of potential options, and the number of these that relate to health system arrangements and implementation strategies is increasing. When searching for systematic reviews that address the impacts of health system arrangements, Health Systems Evidence is a good place to start. Other sources that can be searched for systematic reviews of the impacts of health system arrangements and implementation strategies include The Cochrane Library and PubMed. A list of databases and recommended strategies for searching them are described in the ‘Additional resources’ of this guide.  The EVIPNet Virtual Health Library (www.evipnet.org) is being developed to provide quick and easy access to evidence for informed decisions about health systems in low- and middle-income countries. It will provide a meta-search engine to facilitate searching these and other databases.

A search log, Workshop materials and a PowerPoint presentation on finding systematic reviews are available in the ‘Additional resources’ section of this guide. The SUPPORT Tool on finding systematic reviews offers further guidance.

Selecting systematic reviews

Once potentially relevant references have been found, it is then necessary to decide which systematic reviews to retrieve in full text form and examine in detail. Explicit selection criteria should be used to guide these judgements and to make the selection process as transparent as possible. These criteria might specify, for example:

• Minimum methodological requirements (e.g. a methods section, explicit selection criteria, and a description of the search strategy used)
• The types of populations, patients or problems that the reviews need to address
• The types of interventions, options, health system arrangements, implementation strategies, outcomes or comparisons that the review needs to address

It may sometimes be appropriate to have a selection criterion that relates to the actual setting of the review (e.g. in primary care). However only considering reviews that are restricted to specific countries or low- and middle-income countries runs the risk of excluding important evidence (which may sometimes be the best available). Including reviews of studies from diverse settings and subsequently assessing the applicability (see Box 4.2) of the results to your setting is likely to be more informative.

More guidance on the applicability of the results of systematic reviews can be found in the SUPPORT tool provided in the ‘Additional resources’ section of this guide.

Making judgements about how much confidence to place in a systematic review

Once a relevant systematic review has been found, a decision must be taken about how much confidence to place in this review. This judgement is different to the judgement about how much confidence to place in the evidence, and this separate issue is addressed in the next section.

A number of checklists are available to guide assessments of the reliability of systematic reviews. The ‘Additional resources’ section of this guide contains a sample checklist adapted from the one used to prepare the SUPPORT summaries and other similar checklists.^2,3,4 This checklist is designed to guide judgements about whether a review is likely to provide a reliable summary of the best available evidence of the impacts of these complex interventions.^5,6 

The checklist is divided into two parts. The first assesses the methods used to identify, select, and critically appraise studies. The second assesses the methods used to analyse the results of the included studies. Summary assessments based on the questions in each section determine whether a review is rated as having minor, moderate, or major limitations. After this, an overall assessment is made based on the two summary assessments (and other potential limitations). Reviews are placed into one of the three categories listed below that then determine how the reviews are utilised in policy briefs:

• Fatal flaws – The review has limitations that are sufficiently important to render the results of the review unreliable. As such, the results should not be used in the policy brief (although it may still be possible to draw some key messages or useful information from the review, such as a framework for identifying potential options)
• Important limitations – The review has limitations important enough to make searching for another systematic review worthwhile. The results of this review should be interpreted cautiously if a better review cannot be found (however, the information provided in the review could potentially be supplemented with additional searches, or information from included studies may be included in the policy brief).
• Reliable – The review has only minor limitations and can be used as a reliable summary of the best available evidence.

If a systematic review without important limitations cannot be found, it may be necessary to search for individual studies instead; these can either supplement information in a review or take the place of a systematic review. In such instances, the same processes should be used as are used to select the studies in a systematic review. As far as possible, systematic and transparent (explicit) methods should be used to find, select, and critically appraise studies and to synthesize their results. Ideally the methods used to do this should be described in an appendix document of the policy brief. It should be remembered however, that such additional processes are likely to require substantial additional resources, and specific additional skills will be needed.

This page was last updated November 2011.