Logo PTI Logo FedCSIS

Proceedings of the 17th Conference on Computer Science and Intelligence Systems

Annals of Computer Science and Information Systems, Volume 30

A Comparative Study of Short Text Classification with Spiking Neural Networks

, , ,

DOI: http://dx.doi.org/10.15439/2022F184

Citation: Proceedings of the 17th Conference on Computer Science and Intelligence Systems, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 30, pages 79–88 ()

Full text

Abstract. Short text classification is an important task widely used in many applications. However, few works investigated applying Spiking Neural Networks (SNNs) for text classification. To the best of our knowledge, there were no attempts to apply SNNs as classifiers of short texts. In this paper, we offer a comparative study of short text classification using SNNs. To this end, we selected and evaluated three popular implementations of SNNs: evolving Spiking Neural Networks (eSNN), the NeuCube implementation of SNNs, as well as the SNNTorch implementation that is available as the Python language package. In order to test the selected classifiers, we selected and preprocessed three publicly available datasets: 20-newsgroup dataset as well as imbalanced and balanced PubMed datasets of medical publications. The preprocessed 20-newsgroup dataset consists of first 100 words of each text, while for the classification of PubMed datasets we use only a title of each publication. As a text representation of documents, we applied the TF-IDF encoding. In this work, we also offered a new encoding method for eSNN networks, that can effectively encode values of input features having non-uniform distributions. The designed method works especially effectively with the TF-IDF encoding. The results of our study suggest that SNN networks may provide the classification quality is some cases matching or outperforming other types of classifiers.

References

  1. J. Weissbock, A. A. Esmin, and D. Inkpen, “Using external information for classifying tweets,” in 2013 Brazilian Conference on Intelligent Systems, 2013, pp. 1–5.
  2. M. Kozlowski and H. Rybinski, “Clustering of semantically enriched short texts,” Journal of Intelligent Information Systems, vol. 53, no. 1, pp. 69–92, 2019.
  3. I. Laña, J. L. Lobo, E. Capecci, J. Del Ser, and N. Kasabov, “Adaptive long-term traffic state estimation with evolving spiking neural networks,” Transportation Research Part C: Emerging Technologies, vol. 101, pp. 126 – 144, 2019.
  4. P. S. Maciąg, N. Kasabov, M. Kryszkiewicz, and R. Bembenik, “Air pollution prediction with clustering-based ensemble of evolving spiking neural networks and a case study for london area,” Environmental Modelling & Software, vol. 118, pp. 262 – 280, 2019.
  5. P. S. Maciąg, M. Kryszkiewicz, and R. Bembenik, “Online evolving spiking neural networks for incremental air pollution prediction,” in 2020 International Joint Conference on Neural Networks (IJCNN), 2020, pp. 1–8.
  6. H. Liu, G. Lu, Y. Wang, and N. Kasabov, “Evolving spiking neural network model for PM2.5 hourly concentration prediction based on seasonal differences: A case study on data from beijing and shanghai,” Areosol and Air Quality Research, vol. 21, no. 2, p. 200247, 2021.
  7. L. Paulun, A. Wendt, and N. Kasabov, “A retinotopic spiking neural network system for accurate recognition of moving objects using neucube and dynamic vision sensors,” Frontiers in Computational Neuroscience, vol. 12, p. 42, 2018.
  8. P. S. Maciąg, M. Kryszkiewicz, R. Bembenik, J. L. Lobo, and J. Del Ser, “Unsupervised anomaly detection in stream data with online evolving spiking neural networks,” Neural Networks, vol. 139, pp. 118–139, 2021.
  9. K. Demertzis and L. Iliadis, “A hybrid network anomaly and intrusion detection approach based on evolving spiking neural network classification,” in E-Democracy, Security, Privacy and Trust in a Digital World, A. B. Sideridis, Z. Kardasiadou, C. P. Yialouris, and V. Zorkadis, Eds. Cham: Springer International Publishing, 2014, pp. 11–23.
  10. N. K. Kasabov, “Neucube: A spiking neural network architecture for mapping, learning and understanding of spatio-temporal brain data,” Neural Networks, vol. 52, pp. 62–76, Apr. 2014.
  11. N. Kasabov and E. Capecci, “Spiking neural network methodology for modelling, classification and understanding of eeg spatio-temporal data measuring cognitive processes,” Information Sciences, vol. 294, pp. 565 – 575, 2015, innovative Applications of Artificial Neural Networks in Engineering.
  12. J. K. Eshraghian, M. Ward, E. Neftci, X. Wang, G. Lenz, G. Dwivedi, M. Bennamoun, D. S. Jeong, and W. D. Lu, “Training spiking neural networks using lessons from deep learning,” arXiv preprint https://arxiv.org/abs/2109.12894, 2021.
  13. J. L. Lobo, I. Oregi, A. Bifet, and J. Del Ser, “Exploiting the stimuli encoding scheme of evolving spiking neural networks for stream learning,” Neural Networks, vol. 123, pp. 118 – 133, 2020.
  14. Q. Li, H. Peng, J. Li, C. Xia, R. Yang, L. Sun, P. S. Yu, and L. He, “A survey on text classification: From traditional to deep learning,” vol. 13, no. 2, apr 2022. [Online]. Available: https://doi.org/10.1145/3495162
  15. J. Y. Lee and F. Dernoncourt, “Sequential short-text classification with recurrent and convolutional neural networks,” arXiv preprint https://arxiv.org/abs/1603.03827, 2016.
  16. Chen, Yahui, “Convolutional neural network for sentence classification,” Master’s thesis, 2015. [Online]. Available: http://hdl.handle.net/10012/9592
  17. Y. Hu, Y. Li, T. Yang, and Q. Pan, “Short text classification with a convolutional neural networks based method,” in 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV). IEEE, 2018, pp. 1432–1435.
  18. L. Yao, C. Mao, and Y. Luo, “Graph convolutional networks for text classification,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 7370–7377, Jul. 2019.
  19. K. Kowsari, K. Jafari Meimandi, M. Heidarysafa, S. Mendu, L. Barnes, and D. Brown, “Text classification algorithms: A survey,” Information, vol. 10, no. 4, p. 150, 2019.
  20. M. Białas, M. M. Mirończuk, and J. Mańdziuk, “Biologically plausible learning of text representation with spiking neural networks,” in Parallel Problem Solving from Nature – PPSN XVI, T. Bäck, M. Preuss, A. Deutz, H. Wang, C. Doerr, M. Emmerich, and H. Trautmann, Eds. Cham: Springer International Publishing, 2020, pp. 433–447.
  21. Y. Wang, Y. Zeng, J. Tang, and B. Xu, “Biological neuron coding inspired binary word embeddings,” Cognitive Computation, vol. 11, no. 5, pp. 676–684, 2019.
  22. J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543.
  23. Q. Le and T. Mikolov, “Distributed representations of sentences and documents,” in International conference on machine learning, 2014, pp. 1188–1196.
  24. J. L. Lobo, I. Laña, J. Del Ser, M. N. Bilbao, and N. Kasabov, “Evolving spiking neural networks for online learning over drifting data streams,” Neural Networks, vol. 108, pp. 1 – 19, 2018.
  25. B. Petro, N. Kasabov, and R. M. Kiss, “Selection and optimization of temporal spike encoding methods for spiking neural networks,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–13, 2019.
  26. E. M. Izhikevich and G. M. Edelman, “Large-scale model of mammalian thalamocortical systems,” Proceedings of the National Academy of Sciences, vol. 105, no. 9, pp. 3593–3598, 2008.
  27. F. Ponulak and A. Kasiński, “Supervised learning in spiking neural networks with resume: sequence learning, classification, and spike shifting,” Neural computation, vol. 22, no. 2, pp. 467–510, 2010.
  28. F. Ponulak and A. Kasinski, “Introduction to spiking neural networks: Information processing, learning and applications.” Acta neurobiologiae experimentalis, vol. 71, no. 4, pp. 409–433, 2011.
  29. I. Feinerer, “Introduction to the tm package text mining in R,” Avail. on line: http://cran.r-project.org/web/packages/tm/vignettes/tm.pdf, 2013.
  30. R. Řehůřek and P. Sojka, “Software Framework for Topic Modelling with Large Corpora,” in Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. Valletta, Malta: ELRA, May 2010, pp. 45–50, http://is.muni.cz/publication/884893/en.
  31. BioASQ Team. (2021) A challenge in large-scale biomedical semantic indexing and question answering. [Online]. Available: http://www.bioasq.org/participate/challenges