EBVM Toolkit 12: diagnostic test study checklist

Introduction

Critical appraisal is a process which is used to help you identify the strengths and weakness of a research paper.  Understanding how appropriate the study design is for the question you are seeking to answer, how well the study was carried out, and how good the reporting in the paper is helps you to assess whether the paper is likely to provide reliable evidence.  

This page is designed to help you appraise the report of a diagnostic test study. Answering the questions will help you to reflect on how valid the results might be, how well reported they are and whether they are applicable to your local circumstances.

Diagnostic tests may take many forms, including tests run on biological samples, such as blood or urine; imaging studies, such as radiographs or ultrasound; histopathology; and even aspects of the clinical examination. While there may be papers written about various aspects of all of these, for the purposes of this checklist we are assuming that you wish to critically appraise a paper that is reporting on the diagnostic accuracy of a test in order to assess whether it will be useful in guiding diagnosis in your practice.

Diagnostic test study checklist

For each question think about whether the answer is yes, no or not sure and what your reasoning is for that answer.

1. Does this study aim to validate a new test (against a reference standard) or compare two validated tests?

[text needed here]

2. Is this test potentially relevant to your practice?

Will it help you identify a treatable disorder, if so what benefit does it have over the test you currently use? (e.g. earlier detection, lower cost or less invasive) Will it change your treatment plan?

3. Was the methodology clearly described?

This should include information about the test, the population, the setting and the outcomes. Check that the test being validated (or a variant of it) is not being used to contribute to the definition of the reference standard.

4. Was there a comparison with an appropriate reference standard?

Is this reference test(s) the best available indicator in the circumstances? If not, have the authors justified the criteria against which the new test has been assessed?

5. Did this study include an appropriate spectrum of subjects that is clearly defined?

The study should state the animals included and the spectrum of disease, e.g. mild, severe, treated, untreated, comorbidities and commonly confused conditions. Has the disease prevalence been made clear?

6. Did all animals get the new diagnostic test and the reference standard?

[text needed here]

7. Could the results of the test have been influenced by the results of the reference standard?

Did the people who interpreted the tests know the result of the other test for that patient? i.e. were the assessments blinded?

8. Were the methods for performing the test described in sufficient detail to allow replication?

Has the test been repeated and what is the variability? Was the test shown to be reproducible both within and between observers? If this test will be done in practice will you be able to reproduce the results?

9. Are the results presented in a clear way?

What are the features of the test as derived from this study? Are the sensitivity and specificity and positive and negative predictive values presented? Are these acceptable given the disease condition and underlying prevalence?

10. How sure are you about the results?

Could they have occurred by chance? Were confidence intervals given for sensitivity, specificity and other features of the test?

11. Were all outcomes important to the individual or population considered?

For example, the costs and invasiveness of the procedure in comparison to other tests.

12. Can the results be applied to your practice?

Are the animals similar to your population? Does your setting differ significantly?

13. What would be the impact of using this test on your patients/population?

Will the knowledge of the test result lead to a change in patient management?

Try it out yourself

You could use the following paper to try out the questions:

Hall, J. et al. (2014) Comparison of serum concentrations of symmetric dimethylarginine and creatinine as kidney function biomarkers in cats with chronic kidney disease. Journal of Veterinary Internal Medicine 28(6) pp 1676-1683.  https://doi.org/10.1111/jvim.12445