Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Systematic Reviews and Evidence Syntheses: Automation

A guide to the methodology of and resources for Systematic Reviews

Introduction

Evidence synthesis automation is fast-moving research area.

Given the significant time investment required by evidence syntheses, there has been extensive research into methods and tools for automating or semi-automating parts of the process. This page serves as a resource where you can find links to automation tools and scholarly conversations around systematic review automation.

Automation can be considered in two ways. One: automating tasks so as to assist humans in completing stages of the systematic review process (e.g. a tool which extracts keywords from titles and abstracts to aid the librarian in creating a search string). Two: automating tasks so as to replace humans in stages of the systematic review process (e.g. a tool which automatically assesses and assigns a risk of bias rating to a study).

There are currently no tools which are reliable enough to replace humans entirely at any stage of the systematic review process. However, there are many tools (with a variety of learning curves) which allow for a semi-automation of tasks so as to assist human reviewers.

Automation tools

Systematic Review Toolbox is the best source for tools that support tasks throughout all stages of systematic review projects.

Below you will find a small selection of tools, organized by the stage of the systematic review process for which they are most relevant.


Searching

Litsearchr is an R package which facilitates semi-automatic search strategy development.

Polyglot Search translates the syntax of database search strings from one database to another (e.g. PubMed to Web of Science).

Yale MeSH Analyzer examines MeSH terms attached to groups of PubMed citations.

Screening

Rayyan is a tool which aids with screening and deduplication.

Revtools is an R package can support articles screening.

Critical appraisal

RobotReviewer is a machine learning system which allows users to upload RCT articles and see automatically determined information concerning the trial conduct (the 'PICO', study design, and whether there is a risk of bias).

What do the standards say about the use of automation tools?

While the quality of automation tools has increased significantly over the years, there are no tools which are recommended as a replacement for a human reviewer.

The Cochrane Handbook provides clarification on the acceptability and feasibility of automation at several points:

Combining human and machine intelligence

Learning about automation

Clark J, et al. (2021). The impact of systematic review automation tools on methodological quality and time taken to complete systematic review tasks: case study. JMIR Medical Education. 7(2), 10.2196/24418. https://mededu.jmir.org/2021/2/e24418

Clark, et al. (2020). A full systematic review was completed in 2 weeks using automation tools: A case study. Journal of Clinical Epidemiology, 121, 81–90. https://www.jclinepi.com/article/S0895-4356(19)30719-X/fulltext

Lau, J. (2019). Automation in systematic reviews. BMC. https://www.biomedcentral.com/collections/systrevautomation

Methods Symposium: Advanced Methods and Innovative Technologies for Evidence Synthesis. (2021). Automation for systematic reviews. https://medschool.cuanschutz.edu/ophthalmology/education/methodology/methods-symposium-2021#ac-session-2-automation-for-systematic-reviews-1

Pham, et al. (2021). Text mining to support abstract screening for knowledge syntheses: A semi-automated workflow. Systematic Reviews, 10(1), 156. https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-021-01700-x

UNC Chapel Hill. (2020). Learn about SR automation. https://guides.lib.unc.edu/automation

van Altena, A. J, et al. (2019). Usage of automation tools in systematic reviews. Research Synthesis Methods, 10(1), 72–82. https://doi-org.ezproxy.neu.edu/10.1002/jrsm.1335