Artículo

Assessing Citation Integrity in Biomedical Publications: Corpus Annotation and NLP Models

Resumen

Motivation: Citations have a fundamental role in scholarly communication and assessment. Citation accuracy and transparency is crucial for the integrity of scientific evidence. In this work, we focus on quotation errors, errors in citation content that can distort the scientific evidence and that are hard to detect for humans. We construct a corpus and propose natural language processing (NLP) methods to identify such errors in biomedical publications. Results: We manually annotated 100 highly-cited biomedical publications (reference articles) and citations to them. The annotation involved labeling citation context in the citing article, relevant evidence sentences in the reference article, and the accuracy of the citation. A total of 3063 citation instances were annotated (39.18% with accuracy errors). For NLP, we combined a sentence retriever with a fine-tuned claim verification model to label citations as ACCURATE, NOT_ACCURATE, or IRRELEVANT. We also explored few-shot in-context learning with generative large language models. The best performing model—which uses citation sentences as citation context, the BM25 model with MonoT5 reranker for retrieving top-20 sentences, and a fine-tuned MultiVerS model for accuracy label classification—yielded 0.59 micro-F1 and 0.52 macro-F1 score. GPT-4 in-context learning performed better in identifying accurate citations, but it lagged for erroneous citations (0.65 micro-F1, 0.45 macro-F1). Citation quotation errors are often subtle, and it is currently challenging for NLP models to identify erroneous citations. With further improvements, the models could serve to improve citation quality and accuracy.
Autores
Pugas, MAR; Lopes, EL; Lopes, EH; Ferreira, HL
Título
From One End to The Other: A Bibliometric Study of Publications on Omission Neglect Based on The Journals Between 1988 and 2016
Afiliaciones
Universidade Nove de Julho
Año
2020
DOI
10.23925/2178-0080.2020v22i1.44752
Tipo de acceso abierto
gold, Green Submitted
Referencia
WOS:000543784400006
Artículo obtenido de:
WOS
Fuente
0 0 votos
Califica el artículo
Campus Samborondón
Vía Samborondón Km. 13.5
Call Center: 5150400
metricas@ecotec.edu.ec
Facebook Instagram Youtube

Mapa del sitio