Artículo

Assessing Citation Integrity in Biomedical Publications: Corpus Annotation and NLP Models

Resumen

Motivation: Citations have a fundamental role in scholarly communication and assessment. Citation accuracy and transparency is crucial for the integrity of scientific evidence. In this work, we focus on quotation errors, errors in citation content that can distort the scientific evidence and that are hard to detect for humans. We construct a corpus and propose natural language processing (NLP) methods to identify such errors in biomedical publications. Results: We manually annotated 100 highly-cited biomedical publications (reference articles) and citations to them. The annotation involved labeling citation context in the citing article, relevant evidence sentences in the reference article, and the accuracy of the citation. A total of 3063 citation instances were annotated (39.18% with accuracy errors). For NLP, we combined a sentence retriever with a fine-tuned claim verification model to label citations as ACCURATE, NOT_ACCURATE, or IRRELEVANT. We also explored few-shot in-context learning with generative large language models. The best performing model—which uses citation sentences as citation context, the BM25 model with MonoT5 reranker for retrieving top-20 sentences, and a fine-tuned MultiVerS model for accuracy label classification—yielded 0.59 micro-F1 and 0.52 macro-F1 score. GPT-4 in-context learning performed better in identifying accurate citations, but it lagged for erroneous citations (0.65 micro-F1, 0.45 macro-F1). Citation quotation errors are often subtle, and it is currently challenging for NLP models to identify erroneous citations. With further improvements, the models could serve to improve citation quality and accuracy.
Chavez, Henry (57213581521); Albornoz, María Belén (57192277503); Martín, Fernando (57746138000)
‘Big data’ Research: A Bibliometric Analysis of the Scopus Database, 2009-2019
2022
10.5530/jscires.11.1.7
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85132106342&doi=10.5530%2fjscires.11.1.7&partnerID=40&md5=94cfb7404692c0b2ce9373f1b3b2d931
All Open Access; Hybrid Gold Open Access
Scopus
Artículo obtenido de:
Scopus
Fuente
0 0 votos
Califica el artículo
Campus Samborondón
Vía Samborondón Km. 13.5
Call Center: 5150400
metricas@ecotec.edu.ec
Facebook Instagram Youtube

Mapa del sitio