Artículo

Assessing Citation Integrity in Biomedical Publications: Corpus Annotation and NLP Models

Resumen

Motivation: Citations have a fundamental role in scholarly communication and assessment. Citation accuracy and transparency is crucial for the integrity of scientific evidence. In this work, we focus on quotation errors, errors in citation content that can distort the scientific evidence and that are hard to detect for humans. We construct a corpus and propose natural language processing (NLP) methods to identify such errors in biomedical publications. Results: We manually annotated 100 highly-cited biomedical publications (reference articles) and citations to them. The annotation involved labeling citation context in the citing article, relevant evidence sentences in the reference article, and the accuracy of the citation. A total of 3063 citation instances were annotated (39.18% with accuracy errors). For NLP, we combined a sentence retriever with a fine-tuned claim verification model to label citations as ACCURATE, NOT_ACCURATE, or IRRELEVANT. We also explored few-shot in-context learning with generative large language models. The best performing model—which uses citation sentences as citation context, the BM25 model with MonoT5 reranker for retrieving top-20 sentences, and a fine-tuned MultiVerS model for accuracy label classification—yielded 0.59 micro-F1 and 0.52 macro-F1 score. GPT-4 in-context learning performed better in identifying accurate citations, but it lagged for erroneous citations (0.65 micro-F1, 0.45 macro-F1). Citation quotation errors are often subtle, and it is currently challenging for NLP models to identify erroneous citations. With further improvements, the models could serve to improve citation quality and accuracy.
https://doi.org/10.1093/bioinformatics/btae420
Assessing Citation Integrity in Biomedical Publications: Corpus Annotation and NLP Models
2024
gold
https://academic.oup.com/bioinformatics/advance-article-pdf/doi/10.1093/bioinformatics/btae420/58342072/btae420.pdf
Maria Janina Sarol; Shufan Ming; Shruthan Radhakrishna; Jodi Schneider; Halil Kilicoglu
Informatics Programs, University of Illinois Urbana-Champaign , Champaign, IL 61820, United States; School of Information Sciences, University of Illinois Urbana-Champaign , Champaign, IL 61820, United States; Department of Computer Science, University of Illinois Urbana-Champaign , Champaign, IL 61801, United States
Artículo obtenido de:
OpenAlex
Fuente
0 0 votos
Califica el artículo
Subscribirse
Notificación de
Campus Samborondón
Vía Samborondón Km. 13.5
Call Center: 5150400
metricas@ecotec.edu.ec
Facebook Instagram Youtube

Mapa del sitio