Bibliometrics to assess the productivity and impact of medical research
Abstract
Background: Bibliometrics is the use of statistical and mathematical analysis to assess research production and quality. These metrics provide important insights into the quality and impact of research by applying standardised metrics. However, there are inherent limitations in their application.
Objective: We aimed to review existing bibliometric indices and assess their comparative utility in the assessment of medical researchers. We specifically aimed to evaluate the utility of the h-index in identifying young or developing medical researchers with future research potential.
Method: We conducted a focussed literature review on commonly used bibliometrics. To explore the utility of these metrics we then used them to evaluate a sample of researchers from a South African medical school faculty. Researchers were ranked with the following metrics: number of publications; h-index; citations per paper; citations per paper per year; and m-index. The h-index, citations and publication counts were drawn from ResearchGate and, if not available, from Google Scholar. The top 20 researchers, based on publication count, were then analysed further.
Results: We identified 145 researchers for analysis of which 37 were excluded due to an inability to obtain additional information. Higher time-dependent metrics (publication count, citation count, h-index) were directly proportional to years since first publication. Indices that corrected for time, such as the m-index, provided more insight and better discrimination in identifying younger researchers with greater research potential.
Conclusion: Bibliometrics have utility as part of the assessment of academic output but may be subject to time-dependent bias. Research quality is best measured using the h-index, g-index and m-index. The h-index is limited by being time dependent and field specific and overlooks highly cited papers. Bibliometrics that account for time, such as the m-index, should be considered in the early identification of young researchers, ideally accompanied by critical peer review.
References
Amano, Tatsuya, Juan P. González-Varo, and William J. Sutherland. 2016. “Languages are still a major barrier to global science.” PLoS Biology 14(12): 1‒8.
Biagioli, M. 2016. “Watch out for cheats in citation game.” Nature 535(7611): 201. doi: 10.1038/ 535201a.
Biccard, B. M., R. A. Dyer, J. L. Swanevelder, J. F. Coetzee, and S. L. Shafer. 2017. “Is the HPCSA requirement for a research dissertation for specialist registration the best option?” Southern African Journal of Anaesthesia and Analgesia 23(4): 4‒6.
Biccard, B. M., T. E. Madiba, H. L. Kluyts, D. M. Munlemvo, F. D. Madzimbamuto, A. Basenero, C. S. Gordon, C. Youssouf, S. R. Rakotoarison, V. Gobin, et al. 2018. “Perioperative patient outcomes in the African Surgical Outcomes Study: A 7-day prospective observational cohort study.” Lancet 391(10130):1 589‒1598. doi: 10.1016/S0140-6736(18)30001-1.
Carpenter, Christopher R., David C. Cone, and Cathy C. Sarli. 2014. “Using publication metrics to highlight academic productivity and research impact.” Academic Emergency Medicine 21(10): 1160‒1172.
Department of Higher Education and Training. 2017. “Ministerial Statement on the Implementation of the University Capacity Development Programme through Effective Management and Utilisation of the University Capacity Development Grant 2018‒2020.” South Africa. https://www.dhet.gov.za/UCD%20Policies/Ministerial%20Statement%20on%20the%20Implementation%20of%20the%20University%20Capacity%20Development%20Programme.pdf.
Department of Higher Education and Training. 2019. “Strategic Plan 2020‒2025”: 1‒56. South Africa. https://www.dhet.gov.za/SiteAssets/Strategic%20Plans/DHET%20Amended%20SP%202020-2025.pdf
Egghe, Leo. 2006. “Theory and practise of the g-index.” Scientometrics 69(1): 131‒152. doi: 10.1007/s11192-006-0144-7.
Hicks, Diana, Paul Wouters, Ludo Waltman, Sarah De Rijcke, and Ismael Rafols. 2015. “Bibliometrics: The Leiden Manifesto for research metrics.” Nature 520(7548): 429‒431.
Hirsch, J. E. 2005. “An index to quantify an individual’s scientific research output.” Proceedings of the National Academy of Sciences of the United States of America 102(46): 16569‒16572. doi: 10.1073/pnas.0507655102.
Lee, Alan T. K. and Carol A. Simon. 2018. “Publication incentives based on journal rankings disadvantage local publications.” South African Journal of Science 114(9‒10): 1‒3.
MacMaster, F. P., R. Swansburg, and K. Rittenbach. 2017. “Academic Productivity in Psychiatry: Benchmarks for the H-Index.” Academic Psychiatry 41(4): 452‒454. doi: 10.1007/s40596-016-0656-2.
Pagel, P. S. and J. A. Hudetz. 2011. “H-index is a sensitive indicator of academic activity in highly productive anaesthesiologists: Results of a bibliometric analysis.” Acta Anaesthesiol Scand 55(9): 1085‒1089. doi: 10.1111/j.1399-6576.2011.02508.x.
Rad, A. E., W. Brinjikji, H. J. Cloft, and D. F. Kallmes. 2010. “The H-index in academic radiology.” Academic Radiology 17(7): 817‒21. doi: 10.1016/j.acra.2010.03.011.
Rodseth, R. N., R. D. Wise, and D. Bishop. 2017. “Polluting the well.” Southern African Journal of Anaesthesia and Analgesia 23(6).
Svider, P. F., A. A. Pashkova, Z. Choudhry, N. Agarwal, O. Kovalerchik, S. Baredes, J. K. Liu, and J. A. Eloy. 2013. “Comparison of scholarly impact among surgical specialties: an examination of 2429 academic surgeons.” Laryngoscope 123(4): 884‒889. doi: 10.1002/lary.23951.
Tomaselli, Keyan G. 2018. “Perverse incentives and the political economy of South African academic journal publishing.” South African Journal of Science 114(11‒12): 1‒6.
Tschudy, M. M., T. L. Rowe, G. J. Dover, and T. L. Cheng. 2016. “Pediatric Academic Productivity: Pediatric Benchmarks for the h- and g-Indices.” The Journal of Pediatrics 169: 272‒276. doi: 10.1016/j.jpeds.2015.10.030.
Van den Besselaar, Peter and Ulf Sandström. 2019. “Measuring researcher independence using bibliometric data: A proposal for a new performance indicator.” PloS one 14(3).
Yong, Alexander. 2014. “Critique of Hirsch’s citation index: A combinatorial Fermi problem.” Notices of the AMS 61(9): 1040‒1050.
Copyright (c) 2021 Mbali Ndwandwe, David Gray Bishop, Robert Wise, Reitze Rodseth
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
This journal is an open access journal, and the authors and journal should be properly acknowledged, when works are cited.
Authors, copyright holders, may use the publishers version for teaching purposes, in books, theses, dissertations, conferences and conference papers.
A copy of the authors' publishers version may also be hosted on the following websites:
- Non-commercial personal homepage or blog.
- Institutional webpage.
- Authors Institutional Repository.
The following notice should accompany such a posting on the website: This is an electronic version of an article published in SAJHE, Volume XXX, number XXX, pages XXX “XXX", DOI. Authors should also supply a hyperlink to the original paper or indicate where the original paper (http://www.journals.ac.za/index.php/SAJHE) may be found.
Authors publishers version, affiliated with the Stellenbosch University will be automatically deposited in the University Institutional Repository SUNScholar.
Articles as a whole, may not be re-published with another journal.
The following license applies:
Attribution CC BY-NC-ND 4.0
