Logo PTI Logo FedCSIS

Proceedings of the 18th Conference on Computer Science and Intelligence Systems

Annals of Computer Science and Information Systems, Volume 35

Analysis of a GPT-3 chatbot with respect to its input in a sales dialogue

, , ,

DOI: http://dx.doi.org/10.15439/2023F3513

Citation: Proceedings of the 18th Conference on Computer Science and Intelligence Systems, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 35, pages 637–647 ()

Full text

Abstract. The aim of the study at hand is to configure and evaluate a GPT-3 chatbot which is resistant to faulty input prompts and sensitive to the emotional setting of a sales dialogue. Design Science Research Methodology by Peffers et al. [46] was applied and evaluated with qualitative interviews in two conditions, that is, short and long language input. Results show that the chatbot was overall able to mimic human-like sales conversations. Some deviant behavior could be observed, especially in the short input condition, revealing more verbiage and insistent questions for purchase by the chatbot.

References

  1. Adamopoulou, E., and Moussiades, L. 2020a. "An Overview of Chatbot Technology," in Artificial Intelligence Applications and Innovations, I. Maglogiannis, L. Iliadis, and E. Pimenidis (eds.), Springer: Cham, pp. 373-383.
  2. Adamopoulou, E., and Moussiades, L. 2020b. "Chatbots: History, technology, and applications," Machine Learning with Applications (2).
  3. Balakrishnan, J., and Dwivedi, Y. 2021. "Conversational commerce: entering the next stage of AI-powered digital assistants," Annals of Operations Research. 10.1007/s10479-021-04049-5.
  4. Barnett, A., Savic, M., Pienaar, K., Carter, A., Warren, N., Sandral, E., Manning, V., and Lubamn D. I. 2021. "Enacting ‘more-than-human’ care: Clients’ and counsellors’ view on the multiple affordances of chatbots in alcohol and other drug counselling," Int J Drug Policy (94).
  5. Bechtold, A. 2017. Foto zum Thema flaches Fokusfoto des Frauengesichtes – Kostenloses Bild zu Porträt auf Unsplash (https://unsplash.com/de/fotos/3402kvtHhOo?utm_source=unsplash& utm_medium=referral&utm_content=creditCopyText; accessed February 27, 2023)
  6. Cavedon, L., Kroos, C., Herath, D., Burnham, D., Bishop, L., Leung, Y., and Stevens, C. 2015. "C’Mon dude!: Users adapt their behaviour to a robotic agent with an attention model," International Journal of Human-Computer Studies (80), pp. 14-23.
  7. Cerezo, J., Kubelka, J., Robbes, R., and Bergel, A. 2019. "Building an Expert Recommender Chatbot," 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE), pp. 59-63.
  8. Clawson, V. K., Bostrom, R. P., and Anson, R. 2016. "The Role of the Facilitator in Computer-Supported Meetings," Small Group Research (24:4), pp. 547-565.
  9. Crolic, C., Thomaz, F., Hadi, R., and Stephen, A. T. 2022. "Blame the Bot: Anthropomorphism and Anger in Customer–Chatbot Interactions," Journal of Marketing (86:1), pp. 132-148.
  10. Döring, D., and Zeller, M. 2022. "Das strukturierte Verkaufsgespräch: Die wichtigsten Werkzeuge für den Vertrieb und ihre Anwendung in der Praxis," Wiesbaden: Springer.
  11. Festinger, L. 1957. "A theory of cognitive dissonance," Combined Academic Publ., Anniversary Edition.
  12. Følstad, A., Nordheim, C. B., and Bjørkli, C. A. 2018. "What Makes Users Trust a Chatbot for Customer Service? An Exploratory Interview Study," Internet Science. INSCI 2018. Lecture Notes in Computer Science, pp. 194-208.
  13. Gibson, J.J. 1977. "The theory of affordances," Perceiving, Acting and Knowing, pp. 67–82. Lawrence Erlbaum Associates, Mahwah.
  14. Go, E., and Sundar, S. 2019. "The effects of visual, identity and conversational cues on humanness perceptions," Computers in Human Behavior (97), pp. 304-316.
  15. Gregor, S., Kruse, L. C., and Seidel, S. 2020. "The Anatomy of a Design Principle," Journal of the Association for Information Systems (21), pp. 1622-1652.
  16. Harmon-Jones, E., and Mills, J. 2019. "An introduction to cognitive dissonance theory and an overview of current perspectives on the theory," Cognitive dissonance: Reexamining a pivotal theory in psychology, pp. 3-24.
  17. Holland, H. 2014. "Dialogmarketing - Offline und Online," in Digitales Dialogmarketing, H. Holland (eds). Wiesbaden: Springer Gabler, pp. 3-28. https://doi.org/10.1007/978-3-658-02541-0_1.
  18. IBM n.d. What is a chatbot? IBM. (https://www.ibm.com/topics/chatbots; accessed February 18, 2023).
  19. Kahnemann, D. 2012. "Thinking, fast and slow". Penguin.
  20. Kim, Y., Kwak, S. S., and Kim, M.-S. 2013. "Am I Acceptable to You? Effect of a Robot’s Verbal Language Forms on People’s Social Distance from Robots," Computers in Human Behavior (29:3), pp. 1091-1101.
  21. Kucherbaev, P., Bozzon, A., and Houben, G.-J. 2018. "Human Aided Bots," IEEE Internet Computing (22), pp. 36-43.
  22. Kroeber-Riel, W., and Gröppel-Klein, A. 2019. "Konsumentenverhalten". Vahlen.
  23. Lebovitz, S, Levina, N, and Lifshitz-Assaf, H. 2021. "Is AI Ground Truth Really True? The Dangers of Training and Evaluating AI Tools Based on Experts’ Know-What," MIS Quarterly, (45: 3) pp. 1501-1526.
  24. Lee, Sae. 2020. "Chatbots and Communication: The Growing Role of Artificial Intelligence in Addressing and Shaping Customer Needs," Business Communication Research and Practice (3), pp. 103-111. 10.22682/bcrp.2020.3.2.103.
  25. Lewis, M., Yarats, D., Dauphin, Y., Parikh, D., and Batra, D. 2017. "Deal or No Deal? End-to-End Learning for Negotiation Dialogues," in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 2433–2443.
  26. Lindquist, K. A. 2021. "Language and Emotion: Introduction to the Special Issue," Affective science (2:2), pp. 91-98. https://doi.org/10.1007/s42761-021-00049-7.
  27. Lippert, A., Gatewood, J., Cai, Z., and Graesser, A. 2019. "Using an Adaptive Intelligent Tutoring System to Promote Learning Affordances for Adults with Low Literacy Skills" in Adaptive Instructional Systems, Sottilare, R., Schwarz, J. (eds). HCII 2019. Lecture Notes in Computer Science(), vol 11597. Springer, Cham. https://doi.org/10.1007/978-3-030-22341-0_26
  28. Lokman, A. S., and Ameedeen, M. A. 2018. "Modern Chatbot Systems: A technical review," in Proceedings of the Future Technologies Conference (FTC) 2018, Vancouver, Canada, pp. 1012-1023.
  29. Lunberry, D, and Liebenau, J. 2021. "Human or Machine? A study of anthropomorphism through an affordance lens," in Digital transformation and human behavior: innovation for people and organisations, C. Metallo, M. Ferrara, A. Lazazzara and S. Za (eds.), Berlin: Springer, pp. 201-215.
  30. Lutkevich, B., and Schmelzer, R. 2023. What is GPT-3? Everything you need to know, Enterprise AI, TechTarget, January 26. (https://www.techtarget.com/searchenterpriseai/definition/GPT-3; accessed February 27, 2023).
  31. Mangold, R. 2014. "Werbepsychologie," in Digitales Dialogmarketing, H. Holland (eds), Wiesbaden: Springer Gabler, pp. 29-50. https://doi.org/10.1007/978-3-658-02541-0_2.
  32. Markus, M.L. and Silver, M.S. 2008. "A foundation for the study of IT effects: a new look at DeSanctis and Poole’s concepts of structural features and spirit," J. Assoc. Inf. Syst. (9:10).
  33. Mayer, R. D., and Harrison, N. 2019. As customers begin to shop through voice assistants, what can brands do to stand out? (2https://hbr.org/2019/08/as-customers-begin-to-shop-through-voice-assistants-what-can-brands-do-to-stand-out2019, accessed 15th February 2022).
  34. Meske, C., Amojo, I., and Thapa, D. 2020. "Understanding the Affordances of Conversational Agents in Mental Mobile Health Services," ICIS 2020 Proceedings (9).
  35. Messina, C. 2016. 2016 will be the year of conversational commerce. (https://medium.com/chris-messina/2016-will-be-the-year-of-conversational-commerce-1586e85e3991#.bsdskkyji, accessed 21 Mai 2023)
  36. Mishra, S., Khashabi, D., Baral, C., Yejin, C., and Hajishirzi, H. 2022. "Reframing Instructional Prompts to GPTk’s Language".
  37. Moussawi, S. 2018. "User Experiences with Personal Intelligent Agents: A Sensory, Physical, Functional and Cognitive Affordances View," in Proceedings of the 2018 ACM Conference on Computers and People Research (SIGMIS-CPR’18), New York, pp. 86-92.
  38. Morrissey, K., and Kirakowski, J. 2013. "Realness’ in Chatbots: Establishing Quantifiable Criteria. Human-Computer Interaction," in Interaction Modalities and Techniques. HCI 2013. Lecture Notes in Computer Science, pp. 87-96.
  39. Mygland, M.J., Schibbye, M., Pappas, I.O. and Vassilakopoulou, P. 2021. "Affordances in Human-Chatbot Interaction: A Review of the Literature," in Responsible AI and Analytics for an Ethical and Inclusive Digitized Society. I3E 2021. Lecture Notes in Computer Science (12896), D. Dennehy, A. Griva, N. Pouloudi, Y.K. Dwivedi, I. Pappas, M. Mäntymäki (eds), Cham: Springer. https://doi.org/10.1007/978-3-030-85447-8_1.
  40. Newzoo n.d. Top Countries/Markets by Smartphone Penetration & Users, Newzoo (https://newzoo.com/insights/rankings/top-countries-by-smartphone-penetration-and-users; accessed February 27, 2023).
  41. OpenAI 2015-2023. OpenAI (https://openai.com/; accessed April 29, 2023).
  42. OpenAI. 2015-2023. Introducing ChatGPT (https://openai.com/blog/chatgpt/; accessed April 29, 2023).
  43. OpenAI. 2015-2023. GPT-4 (https://openai.com/product/gpt-4; accessed April 29, 2023).
  44. Pamungkas, E.W. 2019. Emotionally-Aware Chatbots: A Survey. ArXiv, abs/1906.09774.
  45. Pauletto, S., Balentine, B., Pidcock, C., Jones, K., Bottaci, L., Aretoulaki, M., and Balentine, J. 2013. "Exploring expressivity and emotion with artificial voice and speech technologies," Logopedics Phoniatrics Vocology (38:3), pp. 115-125.
  46. Peffers, K., Tuunanen, T., Rothenberger, M. A., and Chatterjee, S. 2007. "A design science research methodology for information systems research", Journal of Management Information Systems (24), pp. 45-77.
  47. Rackham, N. 1996. "The SPIN selling fieldbook practical tools, methods, exercises, and resources", McGraw-Hill.
  48. Radzwill, N., and Benton, M. 2017. "Evaluating Quality of Chatbots and Intelligent Conversational Agents," Software Quality Professional, vol. 19, (3), pp. 25-36.
  49. Rajaobelina, L., and Ricard, L. 2021. "Classifying potential users of live chat services and chatbots," in J Financ Serv Mark, pp. 81-94.
  50. Ramesh, K., Ravishankaran, S., Joshi, A., and Chandrasekaran, K., 2017. "A Survey of Design Techniques for Conversational Agents," in Information, Communication and Computing Technology, Kaushik, S., Gupta, D., Kharb, L., and Chahal, D. (eds.). Singapore: Springer Singapore, pp. 336-350.
  51. Richard, T. 1980. "Toward a positive theory of Consumer Choice," Journal of Economic Behavior and Organization (1:1), pp. 39-60.
  52. Richardson Sales Performance 2021. DNA of a sales dialogue, November 29 (https://www.richardson.com/sales-resources/dialogues/#:~:text=A\%20sales\%20dialogue\%20describes\%20the,solution\%20will\%20meet\%20those\%20needs.; accessed February 27, 2023).
  53. Schiewer, G. L., Altarriba, J., Chin N. B. 2022. "Handbook of Linguistics and Communication Science (HSK) Volume 1," Berlin, Boston: De Gruyter Mouton. https://doi.org/10.1515/9783110347524.
  54. Shanahan, D. 2008. "A new view of language, emotion and the brain," Integrative Psychological & Behavioral Science (42:1), pp. 6–19. https://doi.org/10.1007/s12124-008-9052-0.
  55. Shawar, B. A., and Atwell, E. 2007. "Chatbots: Are they Really Useful?," LDV Forum 22, pp. 29-49.
  56. Scherer, K. R. 1989. "Psychologie der Emotionen".
  57. Scherer, K. R. 1994. "Emotion serves to decouple stimulus and response," in The nature of emotions, Fundamental questions, P. Ekman, and R. J. Davidson (eds), New York, p. 127-130.
  58. Seyffert, R. (1966). "Werbelehre. Theorie und Praxis der Werbung," Stuttgart.
  59. Shiv, B., and Fedorikhin, A. 1999. "Heart and mind in conflict: The interplay of affect and cognition in consumer decision making," Journal of Consumer Research (26), p. 278-292.
  60. Solomon, M. 2017. "If Chatbots win, customers lose, says Zappos Customer Service expert," Forbes, Forbes Magazine, March 24 (https://www.forbes.com/sites/micahsolomon/2017/03/23/customers-lose-if-chatbots-win-says-zappos-customer-service-expert/?sh=3f54c8ed6087; accessed February 27, 2023).
  61. Stoeckli, E., Dremel, C., Uebernickel, F., and Brenner, W. 2020. "How affordances of chatbots cross the chasm between social and traditional enterprise systems," Electron Markets (30), pp. 369-403.
  62. Thaler, R. 1980. "Toward a positive theory of consumer choice," Journal of Economic Behavior and Organization (1), pp. 39-60.
  63. Thaler, R. 1985. "Mental Accounting and Consumer Choice," Marketing Science (4:3), pp. 199-214.
  64. Thieltges, A., Schmidt, F., and Hegelich, S. 2016. "The Devil’s Triangle: Ethical Considerations on Developing Bot Detection Methods," 2016 AAAI Spring Symposium Series.
  65. Tiffert, A. 2021. "Konkrete Anleitung für ein Mitarbeitercoaching im persönlichen Verkauf," in Coaching von Mitarbeitern im persönlichen Verkauf, Wiesbaden: Springer Gabler.
  66. Tversky, A., and Kahneman, D. 1981. "The framing of decisions and the psychology of choice," Behavioral Decision Making, pp. 25-41.
  67. Varghese, E., and Pillai, R. 2017. "A Review on generative conversational model," International Journal of Advance Research in Science and Engineering (6:12), pp. 148-152.
  68. Waizenegger, L., Seeber, I., Dawson, G., and Desouza, K. 2020. "Conversational Agents – Exploring Generative Mechanisms and Second-hand Effects of Actualized Technology Affordances," in Hawaii International Conference on System Sciences.
  69. Wakefield, J. 2016. Microsoft chatbot is taught to swear on Twitter, BBC News, BBC, March 24 (https://www.bbc.com/news/technology-35890188; accessed February 27, 2023).
  70. Wang, J., Hwang, G.-H., and Chang, C.-Y. 2021. "Directions of the 100 most cited chatbot-related human behavior research: A review of academic publications," Computers and Education: Artificial Intelligence, 2. p. 100023.