Volume 25, Issue 4 (3-2023)
jha 2023, 25(4): 125-143 |
Back to browse issues page
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Asaadi Shally A, Sotoudeh H, Abbaspour J. The effectiveness of methodological elements in ranking results by relevance. jha 2023; 25 (4) :125-143
URL: http://jha.iums.ac.ir/article-1-4222-en.html
URL: http://jha.iums.ac.ir/article-1-4222-en.html
1- PhD, Knowledge and Information Sciences, Faculty of Education and Psychology, Shiraz University, Shiraz
2- Professor, Faculty of Education and Psychology, Shiraz University, Shiraz ,sotudeh@shirazu.ac.ir
3- Assistant Professor, Faculty of Education and Psychology, Shiraz University, Shiraz
2- Professor, Faculty of Education and Psychology, Shiraz University, Shiraz ,
3- Assistant Professor, Faculty of Education and Psychology, Shiraz University, Shiraz
Abstract: (933 Views)
Introduction: Methodological validity is one of the aspects of quality. Methodological elements are parts of the text of articles that deal with research methodology. The purpose of this study was to determine the contribution and role of methodological elements in explaining the relationship between evidence and questions.
Methods: This semi-experimental study employed a one-group pretest–posttest design. The research population consisted of clinical trial articles included in the meta-analysis of Cochrane systematic review articles. The sampling method employed was purposeful, whereby systematic review articles containing at least 50 related clinical trial articles retrieved by the retrieval system were selected as the research sample.
Results: The results of the paired t-test showed that the difference in the average nDCG score across all four groups was negative at all points of accuracy. The highest average difference (−0.064) was observed for the basic and standard methodological elements in the abstract at accuracy point 10 (the tenth document in the retrieved results), while the lowest average difference (−0.021) was observed for the basic methodological elements in the abstract at the 50th and 70th accuracy points.
Conclusion: The findings of this research showed that methodological elements, whether independently or to expand the abstract, do not affect the ranking of relevance results or may even have a negative effect. In other words, the occurrence of methodological elements in the text or their weighting can reduce relevant results.
Methods: This semi-experimental study employed a one-group pretest–posttest design. The research population consisted of clinical trial articles included in the meta-analysis of Cochrane systematic review articles. The sampling method employed was purposeful, whereby systematic review articles containing at least 50 related clinical trial articles retrieved by the retrieval system were selected as the research sample.
Results: The results of the paired t-test showed that the difference in the average nDCG score across all four groups was negative at all points of accuracy. The highest average difference (−0.064) was observed for the basic and standard methodological elements in the abstract at accuracy point 10 (the tenth document in the retrieved results), while the lowest average difference (−0.021) was observed for the basic methodological elements in the abstract at the 50th and 70th accuracy points.
Conclusion: The findings of this research showed that methodological elements, whether independently or to expand the abstract, do not affect the ranking of relevance results or may even have a negative effect. In other words, the occurrence of methodological elements in the text or their weighting can reduce relevant results.
Keywords: Methodological elements, Cochrane system review articles, Relevance, Information retrieval, Methodological validity Reviewers' opinions, Cochrane Database, Clinical Trial
Type of Study: Research |
Subject:
Medical Librarianship and Information Science
Received: 2022/10/2 | Accepted: 2022/12/21 | Published: 2023/06/11
Received: 2022/10/2 | Accepted: 2022/12/21 | Published: 2023/06/11
References
1. Fatahi R. Analysis of factors affecting the relativity of relevance in information retrieval systems. Informatics. 2004;2(1):7-22. [In Persian].
2. Khalowi M. Its relevance and meaning in information retrieval. Iranian Journal of Information Processing & Management. 2008;23(3):105-18. [In Persian].
3. Goffman W, Newill VA. Communication and epidemic processes. Proc R Soc Lond A Math Phys Sci. 1967;298(1454): 316-34. [DOI:10.1098/rspa.1967.0106]
4. Schutze H, Pedersen J. Information retrieval based on word sense. Proceedings of the Fourth Annual Symposium on Document Analysis and Information Retrieval; 1995 Apr 24-26; Las Vegas , Nevada. 1995. p. 161-176.
5. Van Nieuwenhoven CA, Buskens E, Van Tiel FH, Bonten MJ. Relationship between methodological trial quality and the effects of selective digestive decontamination on pneumonia and mortality in critically ill patients. JAMA. 2001,18;286(3):335-40. [DOI:10.1001/jama.286.3.335]
6. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Control Clin Trials. 1996;17(1):1-12. [DOI:10.1016/0197-2456(95)00134-4]
7. De Vries AC, Besselink MGH, Buskens E, Ridwan BU, Schipper M, Van Erpecum KJ, et al. Randomized controlled trials of antibiotic prophylaxis in severe acute pancreatitis: Relationship between methodological quality and outcome. Pancreatology. 2007;7(5-6):531-8. [DOI:10.1159/000108971]
8. Sargeant JM, Elgie R, Valcour J, Saint-Onge J, Thompson A, Marcynuk P, et al. Methodological quality and completeness of reporting in clinical trials conducted in livestock species. Prev Vet Med. 2009;91(2-4):107-15. [DOI:10.1016/j.prevetmed.2009.06.002]
9. Xu Y, Chen Z. Relevance judgment: What do information users consider beyond topicality? J Am Soc Inf Sci Technol. 2006;57(7):961-73. [DOI:10.1002/asi.20361]
10. Zhao J, Kan MY, Procter PM, Zubaidah S, Yip WK, Li GM. Improving search for evidence-based practice using information extraction. AMIA Annu Symp Proc.2010;2010:937-41.
11. De Bruijn B, Carini S, Kiritchenko S, Martin J, Sim I. Automated information extraction of key trial design elements from clinical trial publications. AMIA Annu Symp Proc. 2008;2008:141-5.
12. Huang X, Lin J, Demner-Fushman D. Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annu Symp Proc. 2006;2006:359-63.
13. Boudin F, Nie JY, Bartlett JC, Grad R, Pluye P, Dawes M. Combining classifiers for robust PICO element detection. BMC Med Inform Decis Mak. 2010;10(1):1-6. [DOI:10.1186/1472-6947-10-29]
14. Hassanzadeh H, Groza T, Hunter J. Identifying scientific artefacts in biomedical literature: The evidence based medicine use case. J Biomed Inform. 2014;49:159-70. [DOI:10.1016/j.jbi.2014.02.006]
15. Wallace BC, Kuiper J, Sharma A, Zhu MB, Marshall IJ. Extracting PICO sentences from clinical trial reports using supervised distant supervision. J Mach Learn Res. 2016;17:1-26.
16. Cohen AM, Smalheiser NR, McDonagh MS, Yu C, Adams CE, Davis JM, et al. Automated confidence ranked classification of randomized controlled trial articles: An aid to evidence-based medicine. J Am Med Inform Assoc. 2015;22(3):707-17. [DOI:10.1093/jamia/ocu025]
17. Bekhuis T, Demner-Fushman D. Towards automating the initial screening phase of a systematic review. In: Safran C, Reti S, Marin HF, editors. MEDINFO 2010. IOS Press; 2010. p. 146-50.
18. Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics. 2010;11:1-11. [DOI:10.1186/1471-2105-11-55]
19. Wallace BC, Noel-Storr A, Marshall IJ, Cohen AM, Smalheiser NR, Thomas J. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc. 2017;24(6):1165-8. [DOI:10.1093/jamia/ocx053]
20. Hsu W, Speier W, Taira RK. Automated extraction of reported statistical analyses: Towards a logical representation of clinical trial literature. AMIA Annu Symp Proc. 2012;2012:350-9.
21. Alizadeh A, Rashidi H. Intelligent extraction of important words from important parts of scientific articles. Proceedings of the Conference on Computer Engineering and Sustainable Development with a focus on computer networking, modeling and systems security; 2013 Sep 19; Mashhad: Khavaran Institute of Higher Education. 2013. p. 1-7. [In Persian]
22. Rashidi Sharifabad K, Sotoudeh H, Mirzabigi M, Fakhrahamd M. Measuring the similarity of opinions of free referees and the content of scientific articles using natural language processing. Librarianship and Information Organization Studies. 2020;31(2):86-103. [In Persian]
23. Paek H, Kogan Y, Thomas P, Codish S, Krauthammer M. Shallow semantic parsing of randomized controlled trial reports. AMIA Annu Symp Proc. 2006(2006):604-8.
24. Kiritchenko S, De Bruijn B, Carini S, Martin J, Sim I. ExaCT: Automatic extraction of clinical trial characteristics from journal publications. BMC Med Inform Decis Mak. 2010;10:1-17. [DOI:10.1186/1472-6947-10-56]
25. Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, et al. Mayo clinical text analysis and knowledge extraction system (cTAKES): Architecture, component evaluation and applications. J Am Med Inform Assoc. 2010;17(5):507-13. [DOI:10.1136/jamia.2009.001560]
26. Jain S, Peng X, Wallace BC. Detecting twitter posts with adverse drug reactions using convolutional neural networks. In: Sarker A, Gonzalez G, editors. Proceedings of the 2nd Social Media Mining for Health Research and Applications Workshop (SMM4H 2017); 2017 Nov 4; Washington, DC, United States. 2017. p. 72-75.
27. Chung GY, Coiera E. A study of structured clinical abstracts and the semantic classification of sentences. Proceedings of the Workshop on BioNLP 2007: Biological, Translational, and Clinical Language Processing; 2007 Jun 29 ; Prague, Czech Republic. United States: Association for Computational Linguistics; 2007. p. 121-8. [DOI:10.3115/1572392.1572415]
28. Hatami Naghani B, Abessi M. Content analysis of science paper's by using text mining. Business Intelligence Management Studies. 2017;5(18):137-67. [In Persian]
29. Summerscales RL, Argamon S, Bai S, Hupert J, Schwartz A. Automatic summarization of results from clinical trials. Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine. 2011 Nov 12-15; Atlanta, GA, USA. IEEE; 2012. p. 372-7. [DOI:10.1109/BIBM.2011.72]
30. Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014;3:1-15. [DOI:10.1186/2046-4053-3-74]
31. Yoon SH, Kim SW, Park S. C-Rank: A link-based similarity measure for scientific literature databases. Inf Sci. 2016;326:25-40. [DOI:10.1016/j.ins.2015.07.036]
32. Heneghan C, Badenoch D. Evidence-based medicine toolkit. 2nd ed. John Wiley & Sons; 2013.
33. Dehghan M. The feasibility of presenting a quality control model for scientific articles in the publication process based on predictive indicators of scientific impact [Master's thesis]. Shiraz: Shiraz university; 2016. [In Persian]
34. Marshall IJ, Kuiper J, Wallace BC. Automating risk of bias assessment for clinical trials. IEEE J Biomed Health Inform. 2014;19(4):1406-12. [DOI:10.1109/JBHI.2015.2431314]
35. Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: Evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc. 2016;23(1):193-201. [DOI:10.1093/jamia/ocv044]
36. Marshall IJ, Kuiper J, Banner E, Wallace BC. Automating biomedical evidence synthesis: RobotReviewer. Proc Conf Assoc Comput Linguist Meet. 2017;2017:7-12. [DOI:10.18653/v1/P17-4002]
37. Millard LAC, Flach PA, Higgins JPT. Machine learning to assist risk-of-bias assessments in systematic reviews. Int J Epidemiol. 2016;45(1):266-77. [DOI:10.1093/ije/dyv306]
38. Asadi A, Sotudeh H, Abbaspour J, Fakhr-Ahmad M. The potentials of cochrane reviewers' comments and citation contexts in the recognition of randomized controlled trials' texts and their main sections. Health Information Management. 2020;17(4):181-8. [In Persian]
39. Times Higher Education. World University Rankings [Internet]. 2018 [cited 2018 Sep 26]. Available from: https://www.timeshighereducation.com/world-university-rankings
40. Sarker A, Molla D, Paris C. Automatic evidence quality prediction to support evidence-based decision making. Artif Intell Med. 2015;64(2):89-103. [DOI:10.1016/j.artmed.2015.04.001]
41. Achananuparp P, Hu X, Shen X. The evaluation of sentence similarity measures. In: Song I-Y, Eder J, Nguyen TM, editors. Data Warehousing and Knowledge Discovery. DaWaK 2008. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer; 2008. p. 305-16. (LNISA,volume 5182) [DOI:10.1007/978-3-540-85836-2_29]
42. Haynes RB, Wilczynski N, McKibbon KA, Walker CJ, Sinclair JC. Developing optimal search strategies for detecting clinically sound studies in MEDLINE. J Am Med Inform Assoc. 1994;1(6):447-58. [DOI:10.1136/jamia.1994.95153434]
43. Ruthven I, Lalmas M. A survey on the use of relevance feedback for information access systems. Knowl Eng Rev. 2003;18(2):1-55. [DOI:10.1017/S0269888903000638]
44. Cool C, Belkin N, Frieder O, Kantor P. Characteristics of text affecting relevance judgments. National online meeting. 1993;14:77-77.
45. Dhammi IK, Kumar S. Medical subject headings (MeSH) terms. Indian J Orthop. 2014;48(5):443-4. [DOI:10.4103/0019-5413.139827]
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
Site Keywords
Health, Administration, Scientific Article, Iran University of Medical Sciences,
