Annals of Computer Science and Information Systems, Volume 21

DOI: http://dx.doi.org/10.15439/2020F78

Citation: Proceedings of the 2020 Federated Conference on Computer Science and Information Systems, M. Ganzha, L. Maciaszek, M. Paprzycki (eds). ACSIS, Vol. 21, pages 199–204 (2020)

Full text

Abstract. Natural language inference (NLI) is a well established part of natural language understanding (NLU). This task is usually stated as a 3-way classification of sentence pairs with respect to entailment relation (entailment, neutral, contradiction). In this work, we focus on a derived task of relation inference: we propose a method of transforming a general NLI corpus to an annotated corpus for relation inference that utilizes existing NLI annotations. We subsequently introduce a novel relation inference corpus obtained from a well known SNLI corpus and provide its brief characterization. We investigate several DNN siamese architectures for this task and this particular corresponding corpus. We set several baselines including hypothesis only baseline. Our best architecture achieved 96.92 \% accuracy.

References

1. A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes, “Supervised learning of universal sentence representations from natural language inference data,” arXiv preprint https://arxiv.org/abs/1705.02364, 2017.
2. A. Obamuyide and A. Vlachos, “Zero-shot relation classification as textual entailment,” in Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), 2018, pp. 72–78.
3. I. Dagan, O. Glickman, and B. Magnini, “The pascal recognising textual entailment challenge,” in Machine Learning Challenges Workshop. Springer, 2005, pp. 177–190.
4. L. Bentivogli, I. Dagan, and B. Magnini, “The recognizing textual entailment challenges: Datasets and methodologies,” in Handbook of Linguistic Annotation. Springer, 2017, pp. 1119–1147.
5. S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large annotated corpus for learning natural language inference,” arXiv preprint https://arxiv.org/abs/1508.05326, 2015.
6. A. Williams, N. Nangia, and S. R. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” arXiv preprint https://arxiv.org/abs/1704.05426, 2017.
7. A. Lai, Y. Bisk, and J. Hockenmaier, “Natural language inference from multiple premises,” arXiv preprint https://arxiv.org/abs/1710.02925, 2017.
8. O. Levy, T. Zesch, I. Dagan, and I. Gurevych, “Recognizing partial textual entailment,” in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2013, pp. 451–455.
9. O. Levy and I. Dagan, “Annotating relation inference in context via question answering,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2016, pp. 249–255.
10. A. B. Abacha and D. Demner-Fushman, “Recognizing question entailment for medical question answering,” in AMIA Annual Symposium Proceedings, vol. 2016. American Medical Informatics Association, 2016, p. 310.
11. L. Galárraga, G. Heitz, K. Murphy, and F. M. Suchanek, “Canonicalizing open knowledge bases,” in Proceedings of the 23rd acm international conference on conference on information and knowledge management. ACM, 2014, pp. 1679–1688.
12. G. Stanovsky, J. Michael, L. Zettlemoyer, and I. Dagan, “Supervised open information extraction,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018, pp. 885–895.
13. M. Banko, M. J. Cafarella, S. Soderland, M. Broadhead, and O. Etzioni, “Open information extraction from the web.” in Ijcai, vol. 7, 2007, pp. 2670–2676.
14. T.-H. Wu, Z. Wu, B. Kao, and P. Yin, “Towards practical open knowledge base canonicalization,” in Proceedings of the 27th ACM International Conference on Information and Knowledge Management. ACM, 2018, pp. 883–892.
15. A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme, “Hypothesis only baselines in natural language inference,” in Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, 2018, pp. 180–191.
16. T. Khot, A. Sabharwal, and P. Clark, “SciTail: A textual entailment dataset from science question answering,” in AAAI, 2018.
17. M. Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, and R. Zamparelli, “A SICK cure for the evaluation of compositional distributional semantic models,” in Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014, 2014, pp. 216–223.
18. S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. Bowman, and N. A. Smith, “Annotation artifacts in natural language inference data,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 2018, pp. 107–112.
19. C. Biemann, “Creating a system for lexical substitutions from scratch using crowdsourcing,” Language Resources and Evaluation, vol. 47, no. 1, pp. 97–122, 2013.
20. J. Berant, I. Dagan, and J. Goldberger, “Global learning of typed entailment rules,” in Proceedings of the 49th Annual Meeting of the Asso- ciation for Computational Linguistics: Human Language Technologies—Volume 1. Association for Computational Linguistics, 2011, pp. 610– 619.
21. O. Levy, I. Dagan, and J. Goldberger, “Focused entailment graphs for open ie propositions,” in Proceedings of the Eighteenth Conference on Computational Natural Language Learning, 2014, pp. 87–97.
22. N. Zeichner, J. Berant, and I. Dagan, “Crowdsourcing inference-rule evaluation,” in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2. Association for Computational Linguistics, 2012, pp. 156–160.
23. M. Víta, “From building corpora for recognizing faceted entailment to recognizing relational entailment,” in Position Papers of the 2018 Federated Conference on Computer Science and Information Systems, 2018, p. 33.
24. M. Víta and J. Klímek, “First steps in recognizing relational entailment – experimental corpus and baselines,” in Human Language Technologies as a Challenge for Computer Science and Linguistics - 2019, P. P. Zygmunt Vetulani, Ed. Wydawnictwo Nauka i Innowacje, 2019, pp. 143–147.
25. M. Víta and J. Klímek, “Exploiting open ie for deriving multiple premises entailment corpus,” in Proceedings of Recent Advances in Natural Language Processing, 2019, pp. 1257–1264.
26. Y. Lin, Z. Liu, M. Sun, Y. Liu, and X. Zhu, “Learning entity and relation embeddings for knowledge graph completion,” in Twenty-ninth AAAI conference on artificial intelligence, 2015.
27. D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar et al., “Universal sentence encoder,” arXiv preprint https://arxiv.org/abs/1803.11175, 2018.
28. J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543.
29. Q. Du, C. Zong, and K.-Y. Su, “Adopting the word-pair-dependency-triplets with individual comparison for natural language inference,” in Proceedings of the 27th International Conference on Computational Linguistics, 2018, pp. 414–425.
30. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in Proceedings of NAACL-HLT, 2018, pp. 2227–2237.
31. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186.
32. V. Žitkus, R. Butkienė, R. Butleris, R. Maskeliūnas, R. Damaševičius, and M. Woźniak, “Minimalistic approach to coreference resolution in lithuanian medical records,” Computational and mathematical methods in medicine, vol. 2019, 2019.