The research team conducting the review must be capable of undertaking online searches using a fit‐for‐purpose set of search systems, which will enable the researchers to search for and identify all available relevant research in a procedurally unbiased manner. It is the way in which evidence synthesis is undertaken that “may enhance or undermine the trustworthiness of its conclusion or, in common social science parlance, can create threats to the validity of its conclusions.” ( 5, p33) By following strict rules and guidance, a systematic review provides a comprehensive synthesis of a well‐defined area of research. In particular, high‐quality evidence synthesis, in the form of systematic reviews, systematic maps, and meta‐analyses (which themselves should be based on systematic searches and critical appraisal), aims “to produce an unbiased description of the cumulative state of evidence on a research problem or hypothesis” ( 5, p32) and syntheses are thus “viewed as the most reliable sources of evidence for practice and policy.” 6 In this way, evidence synthesis is capable of highlighting important developments in a particular field of research.
3 In academia, it is true that “we are drowning in information, but starving for knowledge” ( 4, p12).Įvidence syntheses-the collective term for robust summaries of evidence-aim to mitigate the issues of decoupled empirical evidence amidst ever‐growing research output. As a consequence, the relevance and impact of their research is reduced. Scientific discourse and cumulative knowledge are threatened if researchers fail to connect their empirical or theoretical analyses with past knowledge. As a result, research agendas neither build on nor advance previous findings but exist in isolation from the greater body of evidence.
The growing volume of research makes it ever harder for practitioners and researchers to keep track of past and current findings in a specific discipline and across disciplines. Research output, as measured by the number of academic publications, continues to grow exponentially, 1, 2 placing scientists in danger of becoming decoupled from the discourse with which they are engaged. Our findings aim to support researchers in conducting better searches for better evidence synthesis. We call for database owners to recognize the requirements of evidence synthesis and for academic journals to reassess quality requirements for systematic reviews. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system. Indeed, only half of the search systems analyzed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall, and reproducibility. A novel, query‐based method tests how well users are able to interact and retrieve records with each system. This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed, and Web of Science. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort.
Rigorous evidence identification is essential for systematic reviews and meta‐analyses (evidence syntheses) because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power.
0 kommentar(er)
