Adapter Based on Pre-Trained Language Models for Classification of Medical Text

© 2024 by the Authors. Licensee Whioce Publishing, Singapore. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution 4.0 International License ( https://creativecommons.org/licenses/by/4.0/ )

Download PDF

Cite

XML

HTML

Abstract

We present an approach to classify medical text at a sentence level automatically. Given the inherent complexity of medical text classification, we employ adapters based on pre-trained language models to extract information from medical text, facilitating more accurate classification while minimizing the number of trainable parameters. Extensive experiments conducted on various datasets demonstrate the effectiveness of our approach.

References

Alishahi A, Chrupa?a G, Linzen T, 2019, Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshop. Natural Language Engineering, 25(4): 543–557.

Mujtaba G, Shuib L, Idris N, et al., 2019, Clinical Text Classification Research Trends: Systematic Literature Review and Open Issues. Expert Systems with Applications, 116: 494–520.

Kaurova O, Alexandrov M, Blanco X, 2011, Classification of Free Text Clinical Narratives (Short Review). Business and Engineering Applications of Intelligent and Information Systems, 2011: 124–135.

Hoang N, Patrick J, 2016, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 13–17, 2016: Text Mining in Clinical Domain: Dealing with Noise. San Francisco, 549–558.

Brown TB, Mann B, Ryder N, et al., 2020, Language Models are Few-Shot Learners. arXiv. https://doi.org/10.48550/arXiv.2005.14165

Radford A, Wu J, Child R, et al., 2019, Language Models are Unsupervised Multitask Learners. OpenAI blog, 1(8): 9.

Devlin J, Chang M-W, Lee K, et al., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2–7, 2019: BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Minneapolis, 4171–4186. Association for Computational Linguistics.

Houlsby N, Giurgiu A, Jastrzebski S, et al., 2019, Parameter-Efficient Transfer Learning for NLP. International Conference on Machine Learning, 2790–2799.

Hu E J, Shen Y, Wallis P, et al., 2021, LoRA: Low-rank Adaptation of Large Language Models. arXiv. https://doi.org/10.48550/arXiv.2106.09685

Zhang Q, Chen M, Bukharin A, et al., 2022, AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. arXiv. https://doi.org/10.48550/arXiv.2303.10512

Dettmers T, Pagnoni A, Holtzman A, et al., 2024, QLoRA: Efficient Finetuning of Quantized LLMs. arXiv. https://doi.org/10.48550/arXiv.2305.14314

Previous article in this issue

Next article in this issue