EBVM Toolkit 10: systematic review checklist

Introduction

Critical appraisal is a process which is used to help you identify the strengths and weakness of a research paper.  Understanding how appropriate the study design is for the question you are seeking to answer, how well the study was carried out, and how good the reporting in the paper is helps you to assess whether the paper is likely to provide reliable evidence.  

This page is designed to help you appraise a systematic review. Answering the questions will help you to reflect on how valid the results might be, how well reported they are and whether they are applicable to your local circumstances.

Systematic review checklist

For each question think about whether the answer is yes, no or not sure and what your reasoning is for that answer.

1. Did the review address a clearly focused question?

Is there a clear question, can the PICO be identified?

2. Did the authors select the right papers?

Did the papers address the question and have an appropriate study design?

3. Do you think the search would have found all the relevant important papers?

Look for search methods, databases used, reference list use, inclusion of unpublished studies etc.

4. Did the authors do enough to assess the quality of included studies?

Is there evidence of an assessment of potential bias? Is the process of assessment described?

5. If the results of the studies have been combined was it reasonable to do so?

Were the results sufficiently similar in design to combine? Are the results of the included studies clear? Are the reasons for any variations discussed?

6. What are the overall results of the review?

Are you clear about the ‘bottom line’ results? How are the results expressed (odds ratios, relative risk etc)?

7. How precise are the results?

Have confidence intervals been presented?

8. Can the results be applied to your practice?

Is the review relevant to your patient population? Can you gauge benefit and harm for your local situation?

9. Were all the important outcomes considered?

Are there any questions that you would consider important that were not addressed in the review?

10. Are the benefits described worth the harms and costs?

What are the possible adverse effects of the intervention? What are the costs?

Try it out yourself

You could use the following paper to try out the questions:

Nuttall, T. and Cole, L. (2007) Evidence based veterinary dermatology: a systematic review of interventions for the treatment of Pseudomonas otitis in dogs, Veterinary Dermatology, 18(2) pp 69-77. DOI: https://doi.org/10.1111/j.1365-3164.2007.00575.x