Abstract
Background Since the emergence of COVID-19, the academic and scientific community has reacted at pace to understand its epidemiology, diagnosis, treatment, prevention and impact. There are concerns that in the panic to get answers to help manage the pandemic, many of the cornerstones of robust methods are being omitted. The aim of this review was to map the nature, scope and quality of evidence syntheses on COVID-19 and to explore the relationship between review quality and the extent of researcher, policy and media interest.
Methods We conducted a systematic review of systematic reviews, rapid reviews, overviews and qualitative evidence syntheses addressing a research question relating to COVID-19. Searches were conducted in PubMed, Epistemonikos COVID-19 evidence, the Cochrane Library of Systematic Reviews, The Cochrane COVID-19 Study Register, EMBASE, CINAHL, Web of Science Core Collection, and the WHO COVID-19 database in June 2020. Abstract and full text screening were undertaken by two independent reviewers. Descriptive information on review type, purpose, population, size, citation and attention metrics were extracted along with whether the review met six key methodological criteria. For reviews meeting all six methodological criteria, additional data were extracted on methods and publication metrics and AMSTAR-2 was used to assess the quality of the reported methods. Registration: PROSPERO CRD42020188822
Results Searches returned 2334 unique records. After applying eligibility criteria we included 280 reviews. Less than half reported undertaking critical appraisal and a third had no reproducible search strategy. There was considerable overlap in topics, with discordant findings. Eighty-eight of the 280 reviews met all six methodological criteria. Of these, 3 were rated as of moderate or high quality on AMSTAR-2, with the majority having critical flaws: only a third reported registering a protocol, and less than one in five searched named COVID-19 databases. Review conduct and publication was rapid, with 56 of the 88 systematic review reported as being conducted within three weeks, and half published within three weeks of submission. Despite being of low quality and many lacking robust methods, the reviews received substantial attention across both academic and public platforms, and the attention was not related to the quality of review methods.
Conclusion Methodological flaws limit the validity of systematic reviews and the generalisability of their findings. Yet by being reported as ‘systematic reviews’, many readers may well regard them as high quality evidence, irrespective of the methods undertaken. To maintain trustworthiness, researchers, peer-reviewers and journal editors need to ensure systematic reviews adhere to guidelines of best practice.