Takes more effort than conventional database searching and may cost more but may improve retrieval of studies
Ten years ago, researchers such as Andrew Booth were discussing the evidence on ‘supplementary search’ techniques, in addition to database searching, with the aim of finding the significant evidence on a topic in a realistic searching timescale. Systematic searchers were being encouraged to move away from blanket assumptions that exhaustive searching of many databases was always the best approach.
Instead, research was showing the value of more varied techniques including citation tracking, reference harvesting from key reviews, author contact, searching trial registries, hand searching of journals, and web searching of key organisations and search engines. A more diverse approach was felt to be suitable for broader questions than straightforward clinical effectiveness; this matched what Lesley Grayson, Alan Gomersall and Annette Boaz had been saying several years earlier about systematic review techniques in the social sciences and social policy field.
Time and cost – trade-offs
Commercial clients in particular are unlikely to want to wait for exhaustive searches of multiple databases (which could give outputs of many thousands of search hits), so ‘supplementary search’ could be attractive, IF it takes less time.
The snag is that most supplementary approaches are actually more labour intensive than database searching, and clients may want to know why you are departing from exhaustive searching of multiple databases. It’s hard to give a clear-cut answer for every situation, except to say that individual studies have demonstrated better recall from a diversity of search approaches.
Supplementary searching in practice – aligned with guidance
Research on searching has expanded a lot in the several years since Andrew Booth’s ‘triple plus’ approach (searching three key databases plus specialist sources plus supplementary searching) was explained at an information specialist’s CPD meeting. Another search researcher who has been very active during this time is Chris Cooper, of UCL.
Their team’s 2017 study on supplementary search is a good place to start looking at the evidence. This study took four of the most respected systematic searching handbooks (Cochrane, Campbell, CRD, NICE) and compared their recommendations with what the literature was saying about the value of and resource required for supplementary searching in practice.
On the whole, practice as reported in research was aligned with guidance recommendations. Usefully, the study attempted to evaluate the time cost and value of any extra studies found by supplementary searching. Relatively little data were available on these factors – prior researchers assumed that finding extra studies was a good thing, which may not be the case for some topics. Back in 2011 at the EBLIP6 conference, I mentioned that there was a lack of research on the time taken to do search tasks, and was asked about doing that research.
Searching efficiency – more to be discovered
Chris and his research colleagues have published further work on research efficiency – a couple of examples are on the inefficiency of handsearching conference proceedings and trial registries, and the cost of contacting authors directly to discover further and unpublished studies.
In conclusion, there is much more to read and be curious about when it comes to searching efficiency: these notes make a start.
Booth, A. Going beyond the database list: a creative approach to exhaustive/exhausting searching.(presentation). University of Sheffield; 2013.
Cooper C, Booth A, Britten N, et al. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev. 2017;6:234. https://doi.org/10.1186/s13643-017-0625-1
Cooper C, Snowsill T, Worsley C, et al. Handsearching had best recall but poor efficiency when exporting to a bibliographic tool: case study. J Clin Epidemiol. 2020 Jul 1;123:39-48.
Cooper C, Bou JT, Varley-Campbell J. Evaluating the effectiveness, efficiency, cost and value of contacting study authors in a systematic review: a case study and worked example. BMC Med Res Methodol. 2019 Dec;19(1):1-8.
EBLIP6 Valuing knowledge and expertise. 27-30 June 2011, Salford, UK.
Grayson L, Gomersall A. A difficult business: finding the evidence for social science reviews. London: ESRC UK Centre for Evidence Based Policy and Practice, Queen Mary University of London. 2003.