If you are considering doing a systematic review or meta-analysis, this step-by-step guide aims to support you along the way. It explains the background to these methodologies, what is involved, and how to get started, keep going, and finish!
A Systematic review answers a defined research question by collecting and summarising all empirical evidence that fits pre-specified eligibility criteria.
A meta-analysis is the use of statistical methods to summarise the results of these studies. Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months) and to be useful to other researchers and practitioners they should have:
It is essential that each review is approached rigorously and with careful attention to detai. Plan carefully and document everything. The consensus reporting guidelines for different study deigns proposed by EQUATOR are useful starting point. PRISMA provides guidance on what you should include when reporting a systematic review.
The massive expansion of research output, both in peer-reviewed publications and unpublished, e.g.in conference presentations or symposia, mean it is difficult to establish what work has been done in your area already and to ensure that clinical practice keeps up to date with the best research evidence. See this presentation by Susan Shenkin for an Introduction to Systematic Reviews
A systematic review is often required as part of undergraduate or postgraduate theses, grant proposals and establishing research agendas. It will be most useful where:
Systematic reviews can be of interventions (i.e. randomised controlled trails) or observations (i.e. case control or cohort studies). The type of study to be included will depend on your research question. Although sociology and psychology have been performing systematic reviews of observational studies for decase, many of the recent resources have been developed within a medical framework using randomised controlled trails (RCTs) to assess whether a treatment is effective or not. In psychology and related disciplines, observational studies are more common (as RCTs may not be feasible or ethical, e.g. it would not be possible to randomise children to poor or enriched social environments to assess impact on cognitions), and systematic reviews have a very important role to play.
It is very difficult to perform a systematic review alone. The highest quality reviews will have input from experts in
(see presentation on systematic Literature Searching Sheila Fisken)
>
Clearly establish what your question is: consider using PICO:
This may need several revisions, and some scoping of the literature, to find something that is specific and answerable. A. clear, specific and answerable question is essential to a successful review.
To avoid wasting your time and energy, establish whether this question has already been answered in the published literature, or is registered as an on-going review (e.g. search in DARE, which contains abstracts of quality assessed systematic reviews and details of all Cochrane reviews and protocols, or NHS EED which contains abstracts of quality assessed economic evaluations)
This will ensure that others know yours is on-going (and many journals will now look for registration to ensure high quality reviews which do not deviate from pre-defined criteria are published)
This can be complex and requires searching in different databases (which each require different search strategies) as well as locating non-published studies e.g. by contacting experts in the field, or hand-searching conference proceedings (see presentation on Systematic Literature searching Sheila Fisken). Seek advice directly from your librarian (for the University of Edinburgh, contact Sheila Fisken)
You will also need to become familiar with reference management software (e.g. RefMan, EndNote, Mendeley Wikipedia compares many of the available product. To allow you to manage your searches and also to make writing the final paper easier.
By screening the title and/or abstract you will be able to reject many of the papers you identified as not fulfilling your inclusion criteria. You should then generate a long-list of all the papers you need to read in more detail (err on the side of over-inclusion). Retrieve these from the library from e-journals, by copying the paper original, or request an inter-library loan. Keep a record of why you reject each one (to allow you to fill in the PRISMA flowchart, see later). Ideally, this should be done by two researchers independently (it is very easy to miss one or two articles when screening large numbers) and disputes settled between you, or with discussion with a third person.
Devise a form tailored to your research question to ensure you obtain all relevant information from each of the included studies. You will need to pilot and refine this form before your final search. Ideally this form should be electronic to minimise transcription errors.
There is no consensus on the best way to assess study quality, but most methods encompass issues such as:
The consensus reporting guidelines for different study designs proposed by EQUATOR are the use full starting point, but note these are guidelines for reporting of original studies, NOT for assessment of study quality.
STROBE also provides useful guidelines for good reporting of observational research, including checklists of items that should be included in this type of research.
Useful resources for assessing quality of different study designs can be found here and some specific examples are QUADAS for studies of diagnostic accuracy
This article by Sanderson, Tatt and Higgins (2007) provides a review of the wide range of tools used to assess study quality. It does not recommend any specific tool for general use, but lists the domains which should be included [1) appropriate selection of participants 2) appropriate measurement of variables and 3) appropriate control of confounding, as well as considering design specific biases]. You may need to develop your own quality assessment tool, but do seek advice on the best method of quality assessment for your review.
This article by the Cochrane Collaboration describes a tool they developed for assessing risk of bias in random trials
You can present the data from the studies narratively and/or statistically (a meta-analysis). If studies are very heterogenous it may be most appropriate to summarise the data narratively and not attempt a statistical (meta-analytic) summary.
There are guidelines as to how to present a narrative synthesis here (see section 1.3.5.1 –Narrative Synthesis). A statistical synthesis should include numerical and graphical presentations of the data, and also look at the strength and consistency of the evidence and investigate reasons for any inconsistencies.
For guidance on how to review a diagnostic study see this powerpoint presentation by Francesca Chappell
It is essential that your review is presented clearly and in accordance with current best practice. For general guidance seethe Equator network site here
The PRISMA statement and this related article by Liberati et al. (2009) provies very clear guidance on reporting of systematic reviews, including a flow chart studies, and there is useful advice on reporting meta-analysis of observational studies (MOOSE) at JAMA, 2000 PMID: 10789670
General guidance on scientific writing
A very useful guide on how to write a systematic review by Prof Joanna Wardlaw (2010)
Examples of published systematic reviews and meta-analyses by resesarchers from CCACE
Ensure it is sunitted and published, and registered on the relevant database if appropriate. Remember that research will progress, and your review may need to be updated. It is essential that you keep clear (paper and electronic) records of your search, decisions and data extraction so this can be repeated.
Much of this advice is based on the excellent (and extensive) guidance from the Cochrane Collaboration and the Centre for Reviews and Dissemination at York. If you are proposing to perform a systematic review these provide invaluable detailed advice, and useful examples.