Logo PTI
Polish Information Processing Society
Logo FedCSIS

Annals of Computer Science and Information Systems, Volume 8

Proceedings of the 2016 Federated Conference on Computer Science and Information Systems

Modeling Co-Verbal Gesture Perception in Type Theory with Records

DOI: http://dx.doi.org/10.15439/2016F83

Citation: Proceedings of the 2016 Federated Conference on Computer Science and Information Systems, M. Ganzha, L. Maciaszek, M. Paprzycki (eds). ACSIS, Vol. 8, pages 383–392 ()

Full text

Abstract. In natural language face to face communication interlocutors exploit manifold non-verbal information resources, most notably hand and arm movements, i.e. gestures. In this paper, a type-theoretical approach using Type Theory with Records is introduced which accounts for iconic gestures within an information state update semantics. Iconic gesture are semantically exploited in two steps: firstly, their kinetic representations are mapped onto vector sequence representations from vector space semantics, modeling a perceptual gesture classification; secondly, these vectorial representation are linked to linguistic predicates giving rise to a computational account to semantic-kinematic interfaces. Each of the steps involves reasoning processes, which are made explicit. The resulting framework shows how various resources have to be integrated in the update mechanism in order to deal with apparently simple multimodal utterances.

References

  1. A. Kendon, Gesture: Visible Action as Utterance. Cambridge, MA: Cambridge University Press, 2004.
  2. D. McNeill, Hand and Mind–What Gestures Reveal about Thought. Chicago: Chicago University Press, 1992.
  3. P. Ekman and W. V. Friesen, “The repertoire of nonverbal behavior: Categories, origins, usage, and coding,” Semiotica, vol. 1, no. 1, pp. 49–98, 1969.
  4. M. Refice, M. Savino, M. Caccia, and M. Adduci, “Automatic classification of gestures: a context-dependent approach,” in Proceedings of the 2011 Federated Conference on Computer Science and Information System, 2011, pp. 743–750.
  5. T. Hachaj, M. R. Ogiela, and M. Piekarczyk, “Dependence of Kinect sensors number and position on gestures recognition with gesture description language semantic classifier,” in Proceedings of the 2013 Federated Conference on Computer Science and Information Systems, M. P. M. Ganzha, L. Maciaszek, Ed. IEEE, 2013, pp. 571–575.
  6. M. W. Alibali, “Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information,” Spatial Cognition and Computation, vol. 5, pp. 307–331, 2005.
  7. R. Cooper and A. Ranta, “Natural languages as collections of resources,” in Language in Flux: Relating Dialogue Coordination to Language Variation, Change and Evolution, ser. Communication, Mind and Language, R. Cooper and R. Kempson, Eds. London: College Publications, 2008.
  8. R. Cooper, “Type theory and semantics in flux,” in Philosophy of Linguistics, ser. Handbook of Philosophy of Science, R. Kempson, T. Fernando, and N. Asher, Eds. Oxford and Amsterdam: Elsevier, 2012, vol. 14, pp. 271–323.
  9. A. Lücking, K. Bergmann, F. Hahn, S. Kopp, and H. Rieser, “The Bielefeld speech and gesture alignment corpus (SaGA),” in Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality, ser. LREC 2010. Malta: 7th International Conference for Language Resources and Evaluation, 2010. http://dx.doi.org/10.13140/2.1.4216.1922 pp. 92–98.
  10. J. Zwarts and Y. Winter, “Vector space semantics: A model-theoretic analysis of locative prepositions,” Journal of Logic, Language, and Information, vol. 9, no. 2, pp. 169–211, 2000.
  11. S. Larsson, “Formal semantics for perceptual classification,” Journal of Logic and Computation, 2013. http://dx.doi.org/10.1093/logcom/ext059
  12. A. Lücking, Ikonische Gesten. Grundzüge einer linguistischen Theorie. Berlin and Boston: De Gruyter, 2013, zugl. Diss. Univ. Bielefeld (2011).
  13. S. Dobnik, R. Cooper, and S. Larsson, “Modelling language, action and perception in Type Theory with Records,” in Proceedings of the 7th International Workshop on Constraint Solving and Language Processing, ser. CSLP’12, 2012, pp. 51–62.
  14. A. Kendon, “Gesticulation and speech: Two aspects of the process of utterance,” in The Relationship of Verbal and Nonverbal Communication, ser. Contributions to the Sociology of Language, M. R. Key, Ed. The Hague: Mouton, 1980, vol. 25, pp. 207–227.
  15. D. Loehr, “Apects of rhythm in gesture in speech,” Gesture, vol. 7, no. 2, pp. 179–214, 2007.
  16. H. Rieser, “Aligned iconic gesture in different strata of mm route- description,” in LonDial 2008: The 12th Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL), King’s College London, 2008, pp. 167–174.
  17. M. Johnston, P. R. Cohen, D. McGee, S. L. Oviatt, J. A. Pittman, and I. Smith, “Unification-based multimodal integration,” in Proceedings of the Eighth Conference on European Chapter of the Association for Computational Linguistics, European Chapter Meeting of the ACL. Madrid, Spain: Association for Computational Linguistics, 1997, pp. 281–288.
  18. M. Johnston, “Deixis and conjunction in multimodal systems,” in Pro- ceedings of the 18th Conference on Computational Linguistics – Volume I, International Conference On Computational Linguistics. Saarbrücken, Germany: Association for Computational Linguistics, 2000, pp. 362–368.
  19. B. Bringert, R. Cooper, P. Ljunglöf, and A. Ranta, “Multimodal dialogue system grammars,” in Proceedings of the 9th Workshop on the Semantics and Pragmatics of Dialogue, ser. Dialor’05, 2005.
  20. A. Lücking, H. Rieser, and M. Staudacher, “Multi-modal integration for gesture and speech,” in Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, ser. Brandial’06, D. Schlangen and R. Fernández, Eds. Potsdam: Universitätsverlag Potsdam, 2006, pp. 106–113.
  21. J. Barwise and J. Perry, Situations and Attitudes, ser. The David Hume Series of Philosophy and Cognitive Science Reissues. Stanford: CSLI Publications, 1983.
  22. H. Kamp and U. Reyle, From Discourse to Logic. Dordrecht: Kluwer Academic Publishers, 1993.
  23. R. Montague, Formal Philosophy: Selected Papers. New Haven: Yale University Press, 1974.
  24. T. Fernando, “Observing events and situations in time,” Linguistics and Philosophy, vol. 30, pp. 527–550, 2007. http://dx.doi.org/10.1007/s10988-008-9026-1
  25. S. Kopp, P. Tepper, and J. Cassell, “Towards integrated microplanning of language and iconic gesture for multimodal output,” in Proceedings of the International Conference on Multimodal Interfaces (ICMI’04). ACM Press, 2004, pp. 97–104.
  26. G. Johansson, “Visual perception of biological motion and a model for its analysis,” Perception & Psychophysics, vol. 14, no. 2, pp. 201–211, 1973.
  27. L. B. Lombard, Events: A Metaphysical Study. London: Routledge & Kegan Paul, 1986.
  28. J. Zwarts, “Vectors as relative positions: A compositional semantics of modified PPs,” Journal of Semantics, vol. 14, no. 1, pp. 57–86, 1997.
  29. H. Rieser, “On factoring out a gesture typology from the Bielefeld speech-gesture-alignment corpus,” in Proceedings of GW 2009: Gesture in Embodied Communication and Human-Computer Interaction, S. Kopp and I. Wachsmuth, Eds. Berlin and Heidelberg: Springer, 2010, pp. 47–60.
  30. J. Zwarts, “Vectors across spatial domains: From place to size, orientation, shape, and parts,” in Representing Direction in Language and Space, ser. Explorations in Language and Space. Oxford, NY: Oxford University Press, 2003, vol. 1, ch. 3, pp. 39–68.
  31. M. Weisgerber, “Decomposing path shapes: About an interplay of manner of motion and ‘the path’,” in Proceedings of the Annual meeting of the Gesellschaft für Semantik, ser. Sinn und Bedeutung 10, C. Ebert and C. Endriss, Eds. Berlin: Zentrum für allgemeine Sprachwissenschaft, 2006, pp. 405–419.
  32. A. Lücking, “The display situation,” Towards a formal description of gesture and the speech-gesture interface: Panel the 6th conference of the International Society for Gesture Studies (ISGS) at the University of California, San Diego.
  33. N. Goodman, Languages of Art. An Approach to a Theory of Symbols, 2nd ed. Idianapolis: Hackett Publishing Company, Inc., 1976.
  34. M. Weisgerber, “Where lexical semantics meets spatial description: A framework for “Klettern” and “Steigen”,” in Proceedings of Sinn und Bedeutung 2005, ser. SuB9, E. Maier, C. Bary, and J. Huitink, Eds., 2005, pp. 507–521.
  35. T. Sowa, Understanding Coverbal Iconic Gestures in Shape Descriptions. Berlin: Akademische Verlagsgesellschaft, 2006, zugl. Diss. Univ. Bielefeld.
  36. K. Barczewska and A. Drozd, “Comparison of methods for hand gesture recognition based on dynamic time warping algorithm,” in Proceedings of the 2013 Federated Conference on Computer Science and Information Systems, M. P. M. Ganzha, L. Maciaszek, Ed. IEEE, 2013, pp. 207–210.
  37. H. Rieser, “When hands talk to mouth. gesture and speech as autonomous communicating processes,” in Proceedings of the 19th Workshop on the Semantics and Pragmatics of Dialogue, ser. SEMDIAL 2015 goDIAL, C. Howes and S. Larsson, Eds., Gothenburg, Sweden, 2015, pp. 122–130.
  38. K. Alahverdzhieva and A. Lascarides, “Analysing language and co-verbal gesture in constraint-based grammars,” in Proceedings of the 17th International Conference on Head-Driven Phase Structure Grammar (HPSG), S. Müller, Ed., Paris, 2010, pp. 5–25.
  39. J. Ginzburg, The Interactive Stance: Meaning for Conversation. Oxford, UK: Oxford University Press, 2012.
  40. R. Cooper and J. Ginzburg, “TTR for natural language semantics,” in The Handbook of Contemporary Semantic Theory, 2nd ed., S. Lappin and C. Fox, Eds. Oxford, UK: Wiley-Blackwell, 2015, ch. 12, pp. 375–407.