It’s big, and it’s easy and free to use – but what are the limitations of Google Scholar? Part 2 of this blog series looks at what research says about its coverage and performance.
Google Scholar is a very useful search tool, but as searchers we’d be falling short if we didn’t educate ourselves on its performance and coverage limitations.
Performance for systematic reviews: Gusenbauer and Haddaway recently put 26 search systems, including PubMed, Google Scholar and Web of Science, through a series of rigorous tests of search capabilities. Open source databases and grey literature were included, as well as subscription bibliographic databases and published peer reviewed items.
Their detailed study is essential reading for all systematic reviewers (tip: print it to read), but here are some brief highlights:
- Questions the researchers thought systematic reviewers should ask about their sources:
“1. What is the coverage of the search system, to ensure I access a database suitable for my review?
2. How effectively can I articulate my search via queries, filters, or citation searches so I can retrieve results with high recall and precision?
3. Can I reproduce my search, so that repeated queries will retrieve the same results?
4. How efficiently can I search the system, so I can perform the review within my resource limits?”
- Google Scholar is suited to exploratory searches where the searcher is interested mostly in the first page of results, and that don’t depend on formal search queries.
- Despite Google Scholar’s exceptionally large coverage of academic literature, the search functions are so basic that it has limited value for systematic searching.
- We should treat Google Scholar as a supplementary source for systematic reviews, on account of lack of expert search functions and/or low reproducibility of searches.
Citation tracking: Google Scholar came out top for recall when compared with Microsoft Academic, Web of Science, Scopus, Dimensions and COCI by Martín-Martín et al in their April 2020 publication. Google Scholar retrieved 88% of over 3 million citations of 2,515 highly cited articles published in 2006.
But coverage/recall under experimental conditions may not be the same as what can actually be retrieved. Boeker et al (2013) concluded that “Google Scholar is not ready as a professional searching tool for tasks where structured retrieval methodology is necessary”, based on their study testing retrieval of Cochrane review references.
What else is out there?
The research literature contains examples of Google Scholar use for bibliometrics (see Google Scholar Digest) or to disseminate research (Zientek et al). Few researchers have explicitly relied on Google Scholar exclusively, to conduct a formal scoping review. Labana (2020) has done so for reasons of access during lockdown, although exporting 1,220 records to EndNote must have been laborious.
What seems to be missing from the published literature is evaluation of supplementary uses of Google Scholar for searching, e.g. as a starting point to explore a topic, to help confirm saturation of search outputs from databases, or simply as an informal, cost-effective and rapid way to access some interesting references. Please share relevant research that you are aware of.
Martín-Martín A, Thelwall M, Orduna-Malea E, López-Cózar ED. Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations’ COCI: a multidisciplinary comparison of coverage via citations. arXiv preprint arXiv:2004.14329. 2020 Apr 29.
Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods. 2020 Mar;11(2):181-217.
Boeker M, Vach W, Motschall E. Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC Medical Research methodology. 2013;13:131.
Google Scholar Digest (last updated 2018). Blogspot, available at: http://googlescholardigest.blogspot.com/ (accessed 03 Jan 2021).
Labana, R. The public mental health while in a community quarantine due to COVID-19 pandemic: a scoping review of literature using Google Scholar. Preprints. 2020:2020050050 (doi: 10.20944/preprints202005.0050.v1).
Zientek LR, Werner JM, Campuzano MV, Nimon K. The use of Google Scholar for research and research dissemination. New Horizons in Adult Education and Human Resource Development. 2018;30(1):39-46.