Logo PTI Logo FedCSIS

Proceedings of the 19th Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 39

LSTM-based Deep Neural Network With A Focus on Sentence Representation for Sequential Sentence Classification in Medical Scientific Abstracts

, , , , , ,

DOI: http://dx.doi.org/10.15439/2024F5872

Citation: Proceedings of the 19th Conference on Computer Science and Intelligence Systems (FedCSIS), M. Bolanowski, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 39, pages 219–224 ()

Full text

Abstract. The Sequential Sentence Classification task within the domain of medical abstracts, termed as SSC, involves the categorization of sentences into pre-defined headings based on their roles in conveying critical information in the abstract. In the SSC task, sentences are sequentially related to each other. For this reason, the role of sentence embeddings is crucial for capturing both the semantic information between words in the sentence and the contextual relationship of sentences within the abstract, which then enhances the SSC system performance. In this paper, we propose a LSTM-based deep learning network with a focus on creating comprehensive sentence representation at the sentence level. To demonstrate the efficacy of the created sentence representation, a system utilizing these sentence embeddings is also developed, which consists of a Convolutional-Recurrent neural network (C-RNN) at the abstract level and a multi-layer perception network (MLP) at the segment level. Our proposed system yields highly competitive results compared to state-of-the-art systems and further enhances the F1 scores of the baseline by 1.0\%, 2.8\%, and 2.6\% on the benchmark datasets PudMed 200K RCT, PudMed 20K RCT and NICTA-PIBOSO, respectively. This indicates the significant impact of improving sentence representation on boosting model performance.

References

  1. A. Rai et al., “Query specific focused summarization of biomedical journal articles,” in 2021 16th Conference on Computer Science and Intelligence Systems (FedCSIS), 2021, pp. 91–100, http://dx.doi.org/10.15439/2021F128.
  2. H. S. Nguyen et al., “Semantic explorative evaluation of document clustering algorithms,” in 2013 Federated Conference on Computer Science and Information Systems, 2013, pp. 115–122.
  3. F. Dernoncourt and J. Y. Lee, “PubMed 200k RCT: a dataset for sequential sentence classification in medical abstracts,” in Proceedings of the Eighth International Joint Conference on Natural Language Processing, Nov. 2017, pp. 308–313.
  4. I. Amini, D. Martinez, and D. Molla, “Overview of the ALTA shared task,” in Proceedings of the Australasian Language Technology Association Workshop 2012, 2012, pp. 124–129.
  5. X. Shang, Q. Ma, Z. Lin, J. Yan, and Z. Chen, “A span-based dynamic local attention model for sequential sentence classification,” in Proc. ACL-IJCNLP, 2021, pp. 198–203, http://dx.doi.org/10.18653/v1/2021.acl-short.26.
  6. J. Lin, D. Karakos, D. Demner-Fushman, and S. Khudanpur, “Generative content models for structural analysis of medical abstracts,” in Proceedings of the HLT-NAACL BioNLP Workshop on Linking Natural Language and Biology, 2006, pp. 65–72.
  7. P. Ruch, C. Boyer et al., “Using argumentation to extract key sentences from biomedical abstracts,” International Journal of Medical Informatics, vol. 76, no. 2, pp. 195–200, 2007, http://dx.doi.org/https://doi.org/10.1016/j.ijmedinf.2006.05.002.
  8. S. N. Kim et al., “Automatic classification of sentences to support evidence based medicine,” in BMC bioinformatics, vol. 12, no. 2, 2011, pp. 1–10, http://dx.doi.org/https://doi.org/10.1186/1471-2105-12-S2-S5.
  9. F. Dernoncourt, J. Y. Lee, and P. Szolovits, “Neural networks for joint sentence classification in medical paper abstracts,” in Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2017, pp. 694–700.
  10. D. Jin and P. Szolovits, “Hierarchical neural networks for sequential sentence classification in medical scientific abstracts,” in Proc. EMNLP, 2018, pp. 3100–3109, http://dx.doi.org/10.18653/v1/D18-1349.
  11. K. Yamada, T. Hirao, R. Sasano, K. Takeda, and M. Nagata, “Sequential span classification with neural semi-Markov CRFs for biomedical abstracts,” in Findings of the Association for Computational Linguistics: EMNLP, 2020, pp. 871–877, http://dx.doi.org/10.18653/v1/2020.findings-emnlp.77.
  12. Y. Peng, S. Yan, and Z. Lu, “Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten bench-marking datasets,” in Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 58–65, http://dx.doi.org/10.18653/v1/W19-5006.
  13. Devlin et al., “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint https://arxiv.org/abs/1810.04805, 2018, http://dx.doi.org/10.18653/V1/N19-1423.
  14. J. Pennington, R. Socher, and C. Manning, “GloVe: Global vectors for word representation,” in in Proc. EMNLP, 2014, pp. 1532–1543, doi: 10.3115/v1/D14-1162.
  15. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023.
  16. Y. Gu et al., “Domain-specific language model pretraining for biomedical natural language processing,” ACM Transactions on Computing for Healthcare (HEALTH), vol. 3, no. 1, pp. 1–23, 2021, doi: 10.1145/3458754.
  17. P. K. Diederik and B. Jimmy, “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2015.
  18. A. Brack et al., “Sequential sentence classification in research papers using cross-domain multi-task learning,” International Journal on Digital Libraries, pp. 1–24, 2024, http://dx.doi.org/https://doi.org/10.1007/s00799-023-00392-z.
  19. Cohan et al., “Pretrained language models for sequential sentence classification,” in Proc. EMNLP-IJCNLP, 2019, pp. 3693–3699, http://dx.doi.org/10.18653/v1/D19-1383.
  20. G. Yu, Z. Zhang, H. Liu, and L. Ding, “Masked sentence model based on bert for move recognition in medical scientific abstracts,” Journal of Data and Information Science, vol. 4, no. 4, pp. 42–55, 2019, http://dx.doi.org/10.2478/jdis-2019-0020.
  21. S. Gonçalves, P. Cortez, and S. Moro, “A deep learning classifier for sentence classification in biomedical and computer science abstracts,” Neural Comput. Appl., vol. 32, no. 11, p. 6793–6807, 2020, http://dx.doi.org/10.1007/s00521-019-04334-2.
  22. A. Agibetov, K. Blagec, H. Xu, and M. Samwald, “Fast and scalable neural embedding models for biomedical sentence classification,” BMC bioinformatics, vol. 19, pp. 1–9, 2018, http://dx.doi.org/https://doi.org/10.1186/s12859-018-2496-4.
  23. A. Sarker, D. Mollá, and C. Paris, “An approach for automatic multi-label classification of medical sentences,” in Proceedings of the 4th International Louhi Workshop on Health Document Text Mining and Information Analysis. Sydney, NSW, Australia, 2013.
  24. M. Lui, “Feature stacking for sentence classification in evidence-based medicine,” in Proceedings of the Australasian Language Technology Association Workshop, 2012, pp. 134–138.
  25. D. Mollá, “Overview of the alta shared task: Piboso sentence classification, 10 years later,” in Proceedings of the The 20th Annual Workshop of the Australasian Language Technology Association, 2022, pp. 178–182