Skip to main content

Systematic Reviews

Category: Working with sources

Purpose: General Interest

Audience(s): Faculty, Graduate student

Systematic Reviews - Definition

  • A systematic review brings together all empirical evidence that fits specific eligibility criteria to answer a very direct research question. Key characteristics of a systematic review are: a clear and concise question, specific inclusion and exclusion criteria, a rigorous and systematic search of the literature, critical appraisal of included studies, mechanisms to manage the citations and data, analysis and interpretation of the results and a final report for publication.
  • Difference between Literature Reviews and Systematic Reviews
  • Navigating Video
  • Cochrane Video

Guides and Manuals

  • Finding What Works in Health Care: Standards for Systematic Reviews (IOM)

    IOM's (2011) standards address the entire systematic review process, from locating, screening, and selecting studies for the review, to synthesizing the findings and assessing the overall quality of the body of evidence, to producing the final review report. Includes a link to the IOM Standards for Systematic Reviews.

  • Cochrane Collaboration Handbook

    The Cochrane Handbook for Systematic Reviews of Interventions is the official document that describes in detail the process of preparing and maintaining Cochrane systematic reviews on the effects of healthcare interventions. This is Version 5.1.0 of the Handbook; last edited March 20, 2011.

  • Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA Statement)

    The aim of the PRISMA Statement is to help authors improve the reporting of systematic reviews and meta-analyses. The focus of PRISMA is randomized trials, but it can also be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions.

  • Methods Guide for Effectiveness and Comparative Effectiveness Reviews (AHRQ)

    This guide was developed to improve the transparency, consistency, and scientific rigor of those working on Comparative Effectiveness Reviews.

  • Campbell Collaboration

    Guidelines for producing a Campbell Systematic Review. The Campbell Collaboration is an international research network that produces systematic reviews of the effects of social interventions.

  • Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care

    Provides practical guidance for undertaking evidence synthesis based on a thorough understanding of systematic review methodology. Presents core principles of systematic reviews and highlights issues that are specific to reviews of clinical tests, public health interventions, adverse effects, and economic evaluations. The final chapter discusses incorporation of qualitative research in or alongside effectiveness reviews.

  • Qualitative Research Guide

    Online resources to aid in conducting, finding, using, synthesizing, and teaching qualitative research in health sciences.

  • 3IE Impact

    International Initiative for Impact Evaluation offers a database of systematic reviews on impact evaluations and has methods information for conducting your own.

  • Health Research Policy and Systems - Confidence in Systematic Reviews. This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers.
  • Health Research Policy and Systems - Preparing and using Policy. This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers.
  • Support Summaries Review - Evidence of the effects of health system interventions for low- and middle-income countries

Searching

  • Bandolier, an independent journal about evidence-based healthcare, written by Oxford scientists, (RAM AND HJM) was printed in February 1994. It has appeared monthly ever since and has become the premier source of evidence based healthcare information in the UK and worldwide for both healthcare professionals and consumers.
  • The Campbell Library is the peer-reviewed online monograph series of systematic reviews prepared under the editorial control of the Campbell Collaboration. Campbell systematic reviews follow structured guidelines and standards for summarizing the international research evidence on the effects of interventions in crime and justice, education, international development, and social welfare.
  • Cochrane Library is regularly updated collection of evidence-based medicine databases, including the Cochrane Database of Systematic Reviews: evidence based systematic reviews prepared by the Cochrane Collaboration. These reviews provide high quality information to people providing and receiving care and to those responsible for research, teaching, funding and administration at all levels
  • NREPP is a searchable online registry of more than 350 sustance abuse and mental health interventions. NREPP was developed to help the public learn more about evidence-based interventions that are available for implementation.
  • University of York Centre for Reviews and Dissemination National Institute of Health Research Database.

    Systematic reviews are widely recognised as reliable sources of information about the effects of health and social care interventions. But as with individual research studies, they can be hard to find, may have flaws and can be difficult to interpret. Between 1994 and March 2015, CRD produced and maintained DARE, a database uniquely providing access to over 13,000 abstracts of quality assessed and critically appraised systematic reviews.

  • Rx for Change searchable database containing current research evidence about intervention strategies used to alter behaviours of health technology prescribing, practice, and use. Developed by CADTH.

What does it take to do a Systematic Review?

  • Time - Average time for a review is 18 months. Cochrane Reviews take approximately 2 years.
  • Team - Cochrane Reviews require teams to complete a review. Most reviews include subject specialists as reviewers to screen abstracts, statisticians to help with data analysis, librarians to develop comprehensive search strategies and a project leader to coordinate.
  • Defined Question - Clear and concise research question is the key to a good systematic review. Use PICO framework to identify key concepts and determine inclusion and exclusion criteria.
    • P=patient, population or problem
    • I=intervention, prognostic factor or exposure
    • C=comparison or intervention
    • O=outcome you would like to measure or achieve
  • Written protocol - To outline the study methodology.
  • Registered protocol - Register your protocol with PROSPERO. Registration is free and open to anyone undertaking a systematic review. You are required to register a protocol with Cochrane.
  • Comprehensive Literature Search - Identify key databases, develop a search strategy, conduct a comprehensive and detailed search and keep documented evidence of citations.
  • Citation Management - Best to work with a citation management system.
  • Follow reporting guidelines - Use appropriate guidelines for review of your publication.

Appraisal and Analysis

  • IOM Standard: 3.6 Critically appraise each study

    3.6.1 Systematically assess the risk of bias, using predefined criteria3.6.2 Assess the relevance of the study’s populations, interventions, and outcome measures3.6.3 Assess the fidelity of the implementation of interventions

  • Cochrane Manual

    See Chapter 8: Assessing risk of bias in included studies

  • JAMAevidence - Users' Guides

    Chapters on evaluating studies.

  • Jadad Scale

    The Jadad scale, sometimes known as Jadad scoring or the Oxford quality scoring system, is a procedure to independently assess the methodological quality of a clinical trial. Jadad et al. published a three-point questionnaire that formed the basis for a Jadad score. Jadad, A.R.; Moore R.A., Carroll D., Jenkinson C., Reynolds D.J.M., Gavaghan D.J., McQuay H.J. (1996). "Assessing the quality of reports of randomized clinical trials: Is blinding necessary?". Controlled Clinical Trials 1996 17 (1): 1–12. PMID 8721797

  • GRADE Working Group

    The working group has developed a common, sensible and transparent approach to grading quality of evidence and strength of recommendations.

  • Avoiding Bias in Selecting Studies (AHRQ)

    This is a chapter from "Methods Guide for Effectiveness and Comparative Effectiveness Reviews."

  • The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses

    Nonrandomised studies, including case-control and cohort studies, can be challenging to implement and conduct. Assessment of the quality of such studies is essential for a proper understanding of nonrandomised studies. The Newcastle-Ottawa Scale (NOS) is an ongoing collaboration between the Universities of Newcastle, Australia and Ottawa, Canada. It was developed to assess the quality of nonrandomised studies with its design, content and ease of use directed to the task of incorporating the quality assessments in the interpretation of meta-analytic results.

  • Additional tools for assessing the quality of a study
  • AMSTAR checklist

Data Management

  • Abstrackr

    From Brown University. Open and free to all. It’s set up to automatically pull in abstracts from NLM using PMIDs. One can also transfer abstracts over from Reference Manager or EndNote. Allows for collaborative screening of abstracts.

  • Covidence

    Covidence is a not-for-profit service dedicated to improving the use of evidence in healthcare decision-making. The service partners are The Alfred Hospital, Monash University, National ICT Australia and the University of London.

  • EPPI-Reviewer 4

    EPPI-Reviewer 4 is software for all types of literature review, including systematic reviews, meta-analyses, 'narrative' reviews and meta-ethnographies.

  • The PRISMA Flow Diagram depicts the flow of information through the different phases of a systematic review. It maps out the number of records identified, included and excluded, and the reasons for exclusions.

Reporting

  • Reporting Guidelines

    The EQUATOR Network gathers guidelines for reporting on different types of trials and research studies. Includes CONSORT, STROBE, PRISMA, MOOSE, RAMESES, and more.

  • Standards for the Reporting of new Cochrane Intervention Reviews

    The Methodological Expectations of Cochrane Intervention Reviews (MECIR) project aims to specify methodological expectations for Cochrane Protocols, Reviews, and updates of reviews on the effects of interventions, and to ensure that these methodological expectations are supported and implemented across The Cochrane Collaboration.

  • Review Manager (RevMan) is the software used for preparing and maintaining Cochrane Reviews. You can use RevManfor protocols and full reviews. It is most useful when you have formulated the question for the review, and allows you to prepare the text, build the tables showing the characteristics of studies and the comparisons in the review, and add study data. It can perform meta-analyses and present the results graphically.

​Types of Reviews

  • A typology of Reviews
  • Selected Review Types
    • Narrative Reviews
      • Useful in tracing concept development
      • Scope is broad and comprehensive
      • Methodology is not standardized
      • Journal requirements vary - check the journal's instruction for authorsCollins JA, Fauser BC. Balancing the strengths of systematic and narrative reviews. Hum Reprod Update. 2005 Mar-Apr;11(2):103-4.
    • Reviews of Reviews (Umbrella)
      • Systematic review using only systematic reviews as subjects
      • Synthesizes systematic reviews of same topic
      • Assesses scope and quality of individual systematic reviews
      • Smith V, Devane D, Begley CM, Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol. 2011 Feb 3;11(1):15. doi: 10.1186/1471-2288-11-15.
    • Realist Reviews
      • Focuses on context and process
      • Uses an iterative protocol
      • Useful for complex policy interventions
      • Rycroft-Malone J, McCormack B, Hutchinson AM, DeCorby K, Bucknall TK, Kent B, Schultz A, Snelgrove-Clarke E, Stetler CB, Titler M, Wallin L, Wilson V. Realist synthesis: illustrating the method for implementation research.Implement Sci.2012 Apr 19;7:33. doi: 10.1186/1748-5908-7-33.
    • Rapid Reviews
      • Used on emerging issues needing quick answers
      • Use systematic review methods
      • Time constraints (often ≤3 months)
      • Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Syst Rev. 2012 Feb 10;1:10. doi:10.1186/2046-4053-1-10.
    • Scoping Reviews
    • Systematic Reviews
      • Addresses a specific question
      • Uses specified methodology
      • Requires a team and long term commitment
      • Lodge, M. (2011). Conducting a systematic review: Finding the evidence: Conducting a systematic review. Journal of Evidence-Based Medicine, 4(2), 135–139. https://doi.org/10.1111/j.1756-5391.2011.01130.x

Librarians can help you with systematic reviews.

Contact your Liaison Librarian.