Introduction
Evidence synthesis automation is a fast-moving research area.
Given the significant time investment required by evidence syntheses, there has been extensive research into methods and tools for automating or semi-automating parts of the process. This page serves as a resource where you can find links to automation tools and scholarly conversations around systematic review automation.
Automation can be considered in two ways.
- Automating tasks so as to assist humans in completing stages of the systematic review process (e.g. a tool which extracts keywords from titles and abstracts to aid the librarian in creating a search string).
- Automating tasks so as to replace humans in stages of the systematic review process (e.g. a tool which automatically assesses and assigns a risk of bias rating to a study).
There are currently no tools which are reliable enough to replace humans entirely at any stage of the systematic review process. However, there are many tools (with a variety of learning curves) which allow for a semi-automation of tasks so as to assist human reviewers.
What do the standards say about the use of automation tools?
While the quality of automation tools has increased significantly over the years, there are no tools which are recommended as a replacement for a human reviewer.
The Cochrane Handbook provides clarification on the acceptability and feasibility of automation at several points:
Automation tools
Systematic Review Toolbox (temporarily unavailable) is the best source for tools that support tasks throughout all stages of systematic review projects.
Below you will find a small selection of tools, organized by the stage of the systematic review process for which they are most relevant.
Searching
Litsearchr is an R package which facilitates semi-automatic search strategy development.
Polyglot Search translates the syntax of database search strings from one database to another (e.g. PubMed to Web of Science).
Yale MeSH Analyzer examines MeSH terms attached to groups of PubMed citations.
Screening
Rayyan is a tool which aids with screening and deduplication.
Revtools is an R package can support articles screening.
Critical appraisal
RobotReviewer is a machine learning system which allows users to upload RCT articles and see automatically determined information concerning the trial conduct (the 'PICO', study design, and whether there is a risk of bias).
Combining human and machine intelligence
Selected resources for learning about SR automation
Clark J, et al. (2021). The impact of systematic review automation tools on methodological quality and time taken to complete systematic review tasks: case study. JMIR Medical Education. 7(2), 10.2196/24418. https://mededu.jmir.org/2021/2/e24418
Clark, et al. (2020). A full systematic review was completed in 2 weeks using automation tools: A case study. Journal of Clinical Epidemiology, 121, 81–90. https://www.jclinepi.com/article/S0895-4356(19)30719-X/fulltext
Lau, J. (2019). Automation in systematic reviews. BMC. https://www.biomedcentral.com/collections/systrevautomation
Marshall, I. J., Trikalinos, T. A., Soboczenski, F., Yun, H. S., Kell, G., Marshall, R., & Wallace, B. C. (2023). In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving. Journal of clinical epidemiology, 153, 26–33. https://doi.org/10.1016/j.jclinepi.2022.08.013
Methods Symposium: Advanced Methods and Innovative Technologies for Evidence Synthesis. (2021). Automation for systematic reviews. https://medschool.cuanschutz.edu/ophthalmology/education/methodology/methods-symposium-2021#ac-session-2-automation-for-systematic-reviews-1
UNC Chapel Hill. (2024). Learn about SR automation. https://guides.lib.unc.edu/automation