Skip to main content

Critical Appraisal: Guide

What is Critical Appraisal?

 

   Definition

Critical appraisal can be defined as "the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context" (Burls 2009 p. 1).


Critical appraisal skills are often discussed in the context of evidence-based practice in the fields of nursing, midwifery, health sciences, pharmacy and medicine. This involves finding and critically appraising relevant research for its validity, relevance and reliability before it is combined with the expertise of the health practitioner (and the patients preferences) in treating patients.

This systematic approach can be applied in assessing the quality of research studies in other fields.

 

Why should we critically appraise literature?

It is often assumed that if an article is published in a journal it must be of quality; this is not always the case.

It has been argued that most, "if not in fact 99%, of published articles belong in the bin, and should certainly not be used to inform practice" (Greenhalgh 2014 p. 28). They may use "inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis, and faulty interpretation" (Altman 1994 p. 283).

When reading journal articles and studies it is important to ask yourself if the research is methodologically sound, relevant, is of value to the subject area and patients, has minimized the effects of potential biases and is reported in a clear way.

 

 

 

How Do I Start?

 

Consider some fundamental questions when you begin your appraisal (Greenhalgh 2014):

What was the research question and why was the study needed?

  • The question / hypothesis should be clearly stated in the study.
  • Does the study add something to knowledge in the area?
  • Consider the PICO method (MacInnes and Lamont 2014):
PROBLEM Is there a focused question?
INTERVENTION Is there a clear intervention being examined?
COMPARISON Is there a suitable control or alternative?
OUTCOMES Have results and outcomes been properly gathered and assessed?
What was the research design?
  • Quantitative or qualitative research? Combination?
  • Primary or secondary research?
  • What's the methodology?
  • This is crucial to your in-depth appraisal (will determine the appropriate checklist to apply)
  • The design used should be made clear. Unsure? Check out University of Minnesota's Understanding Research Study Designs guide, for a reminder of some of the types of health research.
Was the design appropriate to the question?
  • Different questions lend themselves to different sorts of research design.
Did the study meet expected standards of ethics and governance?
  • Were ethical issues given proper consideration?

You may also want to use the structure of the article as a framework for your appraisal, which will logically lead you to ask the right questions (Ingham-Broomfield 2014):

Abstracts and Introduction
  • Has the research question been clearly defined?
  • Does the abstract make clear the methods used, results and conclusions?
  • Is there a clear description of the problem being studied?
Literature Search
  • Has current and relevant literature been critically reviewed - rather than just described?
Methodology
  • Is the choice of research method justified?
  • Are there issues of bias?
  • Was any sample population appropriately selected to adequately represent the population being targeted by any particular treatment or drug?
Ethics
  • Were any issues of consent, confidentiality etc. properly addressed?
Results
  • Are the results set out clearly?
  • Have they been analysed properly?
  • Was the choice of particular statistical methods justified?
Conclusions
  • Are any conclusions set out clearly?
  • Do the results support the conclusion?

 

 

 

Key Tools

 

If you are an undergraduate student you will probably find these resources sufficient to allow you to successfully complete your critical appraisal assignment.  Postgraduate research students and staff working at a higher level may also wish to consult our additional resources page for further materials.

There is considerable overlap in the checklists for the same types of research from different organisations. The CASP checklists are possibly the most widely used but you may find you prefer the style of one organisation's lists to another. If in doubt you may wish to consult your lecturer.

Understanding Health Research: A Tool for Making Sense of Health Studies

This new tool takes you through a series of questions to assess the credibility of a piece of research. Designed to be accessible to all types of users, including members of the public seeking guidance on research on their own health circumstances, it is a practical way of grasping the basic concepts involved in critical appraisal.

The useful information section of the website also includes helpful explanations of concepts such as sampling methods, common sources of bias, correlation and causation.

   Go to Understanding Health Research

Critical Appraisal Skills Programme (CASP)

CASP provides a set of 8 checklists for qualitative studies, randomised controlled trials (RCTs), cohort studies, case control studies, diagnostic studies, systematic reviews and economic evaluations.

Each checklist contains 10 questions to facilitate your critical appraisal.

   Go to CASP

Scottish Intercollegiate Guidelines Network (SIGN)

These notes and checklists cover systematic reviews and meta-analyses, randomised controlled trials, cohort studies, case-control studies, diagnostic studies and economic studies.

   Go to Scottish Intercollegiate Guidelines Network (SIGN)

How to Read a Paper

The British Medical Journal's How to read a paper collection of free journal articles (mostly by Trish Greenhalgh) provides excellent advice on questions to ask when reading different types of research.

Although the papers do not contain checklists themselves the book, How to read a paper: the basics of evidence-based practice (Greenhalgh 2014), includes summaries of the points in the papers, together with additional information in similarly titled chapters and additional chapters, as checklists.

   Go To How To Read a Paper Medical Journal

   Go To How To Read a Paper: The Basics of Evidence-based Medicine

Centre for Evidence-based Medicine (CEBM)

Based in Oxford, the Centre' website provides critical appraisal worksheets covering randomised controlled trials, systematic reviews, diagnostics and prognosis.

   Go to Centre for Evidence-based Medicine (CEBM): Critical Appraisal Tools

BestBETS (Best Evidence Topics): Critical Appraisal Worksheets

Coverage includes checklists for randomised controlled trials and qualitative research.

   Go to BestBETS (Best Evidence Topics): Critical Appraisal Worksheets

'Risk of Bias' Tools

Although not strictly a checklist, the 'risk of bias' tool provided by the Cochrane Collaboration in the Cochrane Handbook for Systematic Reviews of Interventions (2011) provides highly useful advice on checking for bias in research.

   Go To 'Risk of Bias' Tool

Joanna Briggs Institute Critical Appraisal Tools

The tools comprise checklists of questions for use in assessing 13 types of research study (including qualitative research and randomised controlled trials).

   Go To Joanna Briggs Institute Critical Appraisal Tool

Additional Resources

The following may facilitate the appraisal of more complicated studies and assist postgraduate research students and members of staff working in greater depth and/or seeking to critically appraise studies for possible inclusion or exclusion from systematic reviews.

International Centre for Allied Health Evidence: Critical Appraisal Tools

An extensive lists of checklists and tools for all types of research design.

   Go To International Centre for Allied Health Evidence

Duke University Medical Centre University Library & Archives: Evidence-Based Practice: Appraise

Some internally produced checklists, plus links to additional sources, including coverage of research studies focusing on therapy, diagnosis, prognosis and harm.

   Go To Duke University Medical Centre University Library & Archives

OT Seeker

Critical appraisal tutorial from a team of occupational therapists from two major Australian universities.

   Go To OT Seeker

PEDro Scale

The Physiotherapy Evidence Database (PEDro) contains details of over 33,000 randomised trials, systematic reviews and clinical practice guidelines in physiotherapy. The PEDro scale provides a structure for critically appraising trials only.

   Go To PEDro Scale

AMSTAR Checklist

AMSTAR stands for A MeaSurement Tool to Assess systematic Reviews. The checklist allows you to logically appraise the quality of a systematic review.

   Go To AMSTAR

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)

While the PRISMA statement gives guidance on writing up and reporting systematic reviews it can be useful to use the checklist to appraise a particular completed review against the standards set by PRISMA.

   Go To CONSORT

CONSORT

The Consort statement provides a set of recommendations for reporting randomized trials. The checklist provided for authors of studies writing up trials also provides a handy list of points to use when judging the quality of trials.

   Go To CONSORT

AGREE II

The Agree II instrument provides guidance on assessing the quality of guidelines.

   Go To AGREE II

 

 

 

  library@rgu.ac.uk       01224 263450        RGU Library       @RGULibrary        Library Matters: RGU Library Blog