The Trials and Tribulations of Doing a Systematic Review

Science is incremental, with every study – positive or negative – meant to contribute to a body of knowledge. Reviewing this body of knowledge is a key part of the process, to know what was done before and what still needs to be done. This is done as part of a ‘literature review’; to review the studies on a topic, find out how to advance our understanding, and design studies to address this.

This is what I did at the end of the first year of my PhD. I completed a project proposal which included a narrative literature review on the biopsychosocial factors of how social support affects mental health. Whilst I have technically already reviewed the literature, I’ve decided to re-do it in a ‘systematic’ way to ensure that I included all relevant research and to minimise bias. This is important for a topic that is multidisciplinary and incorporates both social and biological sciences, as different research fields use different terms for ‘social support’, such as ‘social connection’ or ‘social network’. To ensure that the remainder of my PhD research is based on an accurate view of the research that’s been conducted to date, I broadened my search terms and the databases used in my review of the literature.

1

WHY AM I DOING THIS TO MYSELF?

  • NARRATIVE REVIEWS GIVE A BIASED VIEW OF THE LITERATURE.

Narrative reviews don’t always give the full picture of a topic. Often, someone will read a few influential papers in an area and use these as a base for finding other papers. This ultimately results in an echo chamber where the same or similar ideas are communicated. This also affects a person’s judgement of what they expect to find, making it more likely for them to dismiss ideas that do not fit. Although scientists try to be objective, we are ultimately human and are susceptive to our own cognitive biases to try and make sense of the world.

Another problem is publication bias, whereby positive findings tend to be published more than null or negative findings. Surveying the literature may suggest that an effect is robust and replicable, but this may be because the negative findings are buried away and never published, overestimating the likelihood that an effect is true in the world.

  • SYSTEMATIC REVIEWS INCREASE TRANSPARENCY, OBJECTIVITY AND REPLICABILITY.

Systematic reviews address the pitfalls of standard reviews, which is why they are regarded as the gold standard of evidence. Systematic reviews require the researcher to pre-specify what they will search for, where they will search, how they will determine whether a study is suitable for inclusion in the review, and more. This process reduces the chance that a study will be ignored because it does not fit a particular narrative. This also increases accountability, as all stages of the review are documented, and another person can try to replicate the search if needed.

… But it isn’t as easy as it sounds.

WHAT’S THE BIG DEAL?

  • IT’S DIFFICULT TO KNOW WHERE TO START WITH A SYSTEMATIC REVIEW.

The phrase ‘systematic review’ conjures up ideas of a rigid process of following strict guidelines. In reality, a systematic review has to be tailored to your research question and how much time you have to dedicate to it. Whilst there are excellent guidelines and tools available – such as resources from PRISMA and the Cochrane Collaboration – it takes a while to trawl through what is available to see what fits your project.

Luckily the University of Manchester library has online resources to help with knowing where to start, which provided a solid starting point for my review.

  • A LOT OF TIME IS DEDICATED TO PREPARATION.

Before you can even think about searching for studies, you have to write a protocol stating what you will do and how you will do it, to prevent you from veering off track. I underestimated how long it would take to write the protocol and how much effort would go into constructing it, but ultimately it protects against questionable methods and biases. This planning also set me up nicely to pre-register my protocol on the online register PROSPERO.

  • EVERY DATABASE WORKS DIFFERENTLY.

Literature databases are all unique; some contain studies from one research discipline, others contain studies from multiple disciplines. Some databases use quotation marks around phrases, some have limits to the amount of terms you can enter, and almost all have different key terms that are tagged onto each study. The point is, it’s not as simple as copying and pasting your search strategy from one database to another, as you will most likely throw up an error in the process. You need to invest time in learning the intricacies of each database and adapt your terms accordingly.

This was made all the easier by the ‘Searching systematically’ course by the University of Manchester library, which shows how different databases work and gives an example of how to replicate search strategies across databases. I appreciated someone showing me how to navigate different databases step-by-step and helping me solve cryptic error messages that popped up.

  • YOU HAVE TO BALANCE THE BREADTH OF YOUR SEARCH WITH THE TIME YOU HAVE TO DO IT.

When I was planning my review, I had the noble idea of wanting to find every study ever conducted on the topic. Whilst it is possible to find unpublished research, there is a trade-off between finding relevant studies and progressing with your review. You can search obscure journals and conference abstracts all day, but you also have to decide whether it is worth it just to find one potentially relevant study. It may be that you don’t find everything that’s ever been done but that’s okay – as long as you are systematic within your means, as defined by your protocol.

2

WHERE AM I NOW?

I have now conquered the behemoth of the systematic review protocol and registered this on PROSPERO – hooray! But this is probably just the beginning of the protocol, as it will evolve as the review unfolds and new challenges are presented (with updates being publicly available too, of course).

Systematic reviews are challenging. There is no one-size-fits-all guideline for it. I will probably run into many more problems during the process, but part of doing research is learning by doing and adapting to difficulties. As scientists, we do not have to be perfect, but we have to do the best we can with the resources that are available to us now.

PHOTO CREDITS

Photo by saeed mhmdi on Unsplash

Photo by AbsolutVision on Pixabay.

Originally published on the Research Hive at https://manchesterresearchhive.wordpress.com/2019/11/19/the-trials-and-tribulations-of-the-systematic-review/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s