Logo PTI Logo FedCSIS

Proceedings of the 19th Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 39

Automated feedback generation in an intelligent tutoring system for counselor education

, , ,

DOI: http://dx.doi.org/10.15439/2024F1649

Citation: Proceedings of the 19th Conference on Computer Science and Intelligence Systems (FedCSIS), M. Bolanowski, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 39, pages 501–512 ()

Full text

Abstract. This paper investigates the implementation of AI- driven feedback in an intelligent tutoring system (ITS) developed for training of counselors. By using LLMs, the study explores the automatic generation of feedback for communication-intensive tasks such as online counseling. The evaluation compares dif- ferent feedback methods, including the sandwich, WWW and STATE methods, and assesses their emotional and objective impact. The results show that AI-generated feedback fulfills objective criteria better than emotional ones. Fine-tuning an open source LLM can improve both the emotional and objective quality of feedback. Furthermore, the study examines the accep- tance of AI feedback among aspiring counselors, highlighting the influence of familiarity with AI on acceptance levels. Ethical con- siderations, including bias and hallucination, are addressed, with recommendations for risk mitigation through multi-feedback options and expert supervision. This research contributes to the understanding of the role of AI in improving digital counseling practices and highlights the need for continuous evaluation and ethical considerations.

References

  1. J. Hattie and H. Timperley, “The Power of Feedback,” Review of Educational Research, vol. 77, no. 1, pp. 81–112, Mar. 2007. doi: 10.3102/003465430298487
  2. S. Narciss, S. Sosnovsky, L. Schnaubert, E. Andrès, A. Eichelmann, G. Goguadze et al., “Exploring feedback and student characteristics relevant for personalizing feedback strategies,” Computers & Education, vol. 71, pp. 56–76, Feb. 2014. http://dx.doi.org/10.1016/j.compedu.2013.09.011
  3. W. Dai, J. Lin, H. Jin, T. Li, Y.-S. Tsai, D. Gašević et al., “Can Large Language Models Provide Feedback to Students? A Case Study on Chat-GPT,” in 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), Jul. 2023. http://dx.doi.org/10.1109/ICALT58122.2023.00100 pp. 323–325, iSSN: 2161-377X.
  4. R. E. Wang, Q. Zhang, C. Robinson, S. Loeb, and D. Demszky, “Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes,” Apr. 2024, https://arxiv.org/abs/2310.10648 [cs].
  5. E. M. Engelhardt, “Onlineberatung – Digitales Beratungsangebot für Alle?” in Digital Diversity, H. Angenent, B. Heidkamp, and D. Kergel, Eds. Wiesbaden: Springer Fachmedien Wiesbaden, 2019, pp. 161–173. ISBN 978-3-658-26752-0 978-3-658-26753-7
  6. M. Stieler, S. Lipot, and R. Lehmann, “Zum Stand der Onlineberatung in Zeiten der Corona Krise. Entwicklungs- und Veränderungsprozesse der Onlineberatungslandschaft,” e-beratungsjournal.net – Zeitschrift für Online-Beratung und computervermittelte Kommunikation, vol. 18no. 1, pp. 50–65, 2022. http://dx.doi.org/10.48341/262P-7T64 Publisher: Universität für Weiterbildung Krems & e-beratungsjournal.net.
  7. A. Chaszczewicz, R. S. Shah, R. Louie, B. A. Arnow, R. Kraut, and D. Yang, “Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors,” Mar. 2024, https://arxiv.org/abs/2403.15482 [cs].
  8. S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain et al., “Large Language Models: A Survey,” Feb. 2024, https://arxiv.org/abs/2402.06196 [cs].
  9. L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang et al., “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,” Nov. 2023, https://arxiv.org/abs/2311.05232 [cs].
  10. E. Mousavinasab, N. Zarifsanaiey, S. R. Niakan Kalhori, M. Rakhshan, L. Keikha, and M. Ghazi Saeedi, “Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods,” Interactive Learning Environments, vol. 29, no. 1, pp. 142–163, Jan. 2021. http://dx.doi.org/10.1080/10494820.2018.1558257
  11. E. Rudolph, N. Engert, and J. Albrecht, “An AI-Based Virtual Client for Educational Role-Playing in the Training of Online Counselors,” in Proceedings of the 16th International Conference on Computer Supported Education - Volume 2: CSEDU, vol. 2. SCITEPRESS, May 2024. http://dx.doi.org/10.5220/0012690700003693. ISBN 978-989-758-697-2 pp. 108–117.
  12. J. Carbonell, “AI in CAI: An Artificial-Intelligence Approach to Computer-Assisted Instruction,” IEEE Transactions on Man- Machine Systems, vol. 4, no. 11, pp. 190–202, 1970. doi: 10.1109/TMMS.1970.299942
  13. A. T. Corbett, K. R. Koedinger, and J. R. Anderson, “Chapter 37 - Intelligent Tutoring Systems,” in Handbook of Human-Computer Interaction (Second Edition), M. G. Helander, T. K. Landauer, and P. V. Prabhu, Eds. Amsterdam: North-Holland, Jan. 1997, pp. 849–874. ISBN 978-0-444-81862-1
  14. W. Ma, O. O. Adesope, J. C. Nesbit, and Q. Liu, “Intelligent tutoring systems and learning outcomes: A meta-analysis,” Journal of Educational Psychology, vol. 106, no. 4, pp. 901–918, 2014. http://dx.doi.org/10.1037/a0037123 Place: US Publisher: American Psychological Association.
  15. C. Cao, “Scaffolding CS1 Courses with a Large Language Model-Powered Intelligent Tutoring System,” in Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, ser. IUI ’23 Companion. New York, NY, USA: Association for Computing Machinery, 2023. http://dx.doi.org/10.1145/3581754.3584111. ISBN 9798400701078 pp. 229–232.
  16. J.-Y. Kuo, H.-C. Lin, P.-F. Wang, and Z.-G. Nie, “A Feedback System Supporting Students Approaching a High-Level Programming Course,” Applied Sciences, vol. 12, no. 14, p. 7064, Jan. 2022. http://dx.doi.org/10.3390/app12147064 Number: 14 Publisher: Multidisciplinary Digital Publishing Institute.
  17. Z. Marafie, K.-J. Lin, D. Wang, H. Lyu, Y. Liu, Y. Meng et al., “AutoCoach: An Intelligent Driver Behavior Feedback Agent with Personality-Based Driver Models,” Electronics, vol. 10, no. 11, p. 1361, Jan. 2021. http://dx.doi.org/10.3390/electronics10111361 Number: 11 Publisher: Multidisciplinary Digital Publishing Institute.
  18. A. Botelho, S. Baral, J. A. Erickson, P. Benachamardi, and N. T. Heffernan, “Leveraging natural language processing to support automated assessment and feedback for student open responses in mathematics,” Journal of Computer Assisted Learning, vol. 39, no. 3, pp. 823–840, 2023. http://dx.doi.org/10.1111/jcal.12793
  19. B. Grawemeyer, M. Mavrikis, W. Holmes, S. Gutierrez-Santos, M. Wiedmann, and N. Rummel, “Affecting off-task behaviour: how affect-aware feedback can improve student learning,” in Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, ser. LAK ’16. New York, NY, USA: Association for Computing Machinery, Apr. 2016. http://dx.doi.org/10.1145/2883851.2883936. ISBN 978-1-4503-4190-5 pp. 104–113.
  20. M. Dzikovska, N. Steinhauser, E. Farrow, J. Moore, and G. Campbell, “BEETLE II: Deep Natural Language Understanding and Automatic Feedback Generation for Intelligent Tutoring in Basic Electricity and Electronics,” International Journal of Artificial Intelligence in Education, vol. 24, no. 3, pp. 284–332, Sep. 2014. http://dx.doi.org/10.1007/s40593-014-0017-9
  21. J. McDonald, A. Knott, S. Stein, and R. Zeng, “An empirically-based, tutorial dialogue system: design, implementation and evaluation in a first year health sciences course.” in Proceedings of Electric Dreams. Proceedings ascilite 2013. Australasian Society for Computers in Learning in Tertiary Education, 2013. ISBN 978-1-74138-403-1 pp. 562–572.
  22. F. St-Hilaire, D. D. Vu, A. Frau, N. Burns, F. Faraji, J. Potochny et al., “A New Era: Intelligent Tutoring Systems Will Transform Online Learning for Millions,” Mar. 2022.
  23. C. Cao, “Leveraging Large Language Model and Story-Based Gamification in Intelligent Tutoring System to Scaffold Introductory Programming Courses: A Design-Based Research Study,” Feb. 2023, https://arxiv.org/abs/2302.12834 [cs].
  24. Y. Y. Chiu, A. Sharma, I. W. Lin, and T. Althoff, “A Computational Framework for Behavioral Assessment of LLM Therapists,” Jan. 2024, https://arxiv.org/abs/2401.00820 [cs].
  25. M. Glickman and T. Sharot, “How human-AI feedback loops alter human perceptual, emotional and social judgements,” Nov. 2022.
  26. J. Lin, L. Sha, Y. Li, D. Gasevic, and G. Chen, “Establishing Trustworthy Artificial Intelligence in Automated Feedback,” Jul. 2022.
  27. D. Kaur, S. Uslu, K. J. Rittichier, and A. Durresi, “Trustworthy Artificial Intelligence: A Review,” ACM Computing Surveys, vol. 55, no. 2, pp. 39:1–39:38, Jan. 2022. http://dx.doi.org/10.1145/3491209
  28. D. Gursoy, O. H. Chi, L. Lu, and R. Nunkoo, “Consumers acceptance of artificially intelligent (AI) device use in service delivery,” International Journal of Information Management, vol. 49, pp. 157–169, Dec. 2019. http://dx.doi.org/10.1016/j.ijinfomgt.2019.03.008
  29. V. Venkatesh, M. G. Morris, G. B. Davis, and F. D. Davis, “User Acceptance of Information Technology: Toward a Unified View,” MIS Quarterly, vol. 27, no. 3, pp. 425–478, 2003. http://dx.doi.org/10.2307/30036540 Publisher: Management Information Systems Research Center, University of Minnesota.
  30. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models,” Jul. 2023, https://arxiv.org/abs/2307.09288 [cs].
  31. H. Luo and L. Specia, “From Understanding to Utilization: A Survey on Explainability for Large Language Models,” Feb. 2024, https://arxiv.org/abs/2401.12874 [cs].
  32. I. O. Gallegos, R. A. Rossi, J. Barrow, M. M. Tanjim, S. Kim, F. Dernoncourt et al., “Bias and Fairness in Large Language Models: A Survey,” Mar. 2024, https://arxiv.org/abs/2309.00770 [cs].
  33. N. Dainton, “1 Bedeutung und Wert von Feedback,” in Feedback in der Hochschullehre, ser. utb-Titel ohne Reihe. Haupt, Dec. 2020, pp. 11–22. ISBN 978-3-8252-4891-8
  34. J. Fengler, Feedback geben: Strategien und Übungen; ... mit über 100 Übungen, 4th ed., ser. Beltz Weiterbildung. Weinheim Basel: Beltz, 2009. ISBN 978-3-407-36471-5
  35. R. de Villiers, “7 Principles of highly effective managerial feedback: Theory and practice in managerial development interventions,” The International Journal of Management Education, vol. 11, no. 2, pp. 66–74, Jul. 2013. http://dx.doi.org/10.1016/j.ijme.2013.01.002
  36. J. Ade and U. Gläßer, “Lehrmodul 12: Feedback in der Mediation,” Zeitschrift für Konfliktmanagement, vol. 12, no. 2, Jan. 2009. http://dx.doi.org/10.9785/ovs-zkm-2009-60
  37. N. Dainton, “2 Wo klemmt es,” in Feedback in der Hochschullehre, ser. utb-Titel ohne Reihe. Haupt, Dec. 2020, pp. 23–32. ISBN 978-3-8252-4891-8
  38. A. Seidl, “Dein Wunsch geht in Erfüllung,” in Freundlich, aber bestimmt – Die richtigen Worte finden in Gesundheitsberufen. Berlin, Heidelberg: Springer Berlin Heidelberg, 2014, pp. 51–70. ISBN 978-3-642-41803-7 978-3-642-41804-4
  39. A. Dohrenwend, “Serving Up the Feedback Sandwich,” Family Practice Management, vol. 9, no. 10, pp. 43–46, Nov. 2002.
  40. A. J. Henley and F. D. DiGennaro Reed, “Should You Order the Feedback Sandwich? Efficacy of Feedback Sequence and Timing,” Journal of Organizational Behavior Management, vol. 35, no. 3-4, pp. 321–335, Oct. 2015. http://dx.doi.org/10.1080/01608061.2015.1093057
  41. J. Sammet and J. Wolf, “Präsenztraining im Blended Learning,” in Vom Trainer zum agilen Lernbegleiter: So funktioniert Lehren und Lernen in digitalen Zeiten, J. Sammet and J. Wolf, Eds. Berlin, Heidelberg: Springer, 2019, pp. 55–65. ISBN 978-3-662-58510-8
  42. J. Grenny, K. Patterson, R. McMillan, A. Switzler, and E. Gregory, Crucial Conversations, 3rd ed. New York: McGraw Hill, 2022. ISBN 978-1-260-47419-0
  43. F. Eidenbenz, “Standards in der Online-Beratung,” in Handbuch Online-Beratung. Vandenhoeck & Ruprecht GmbH & Co. KG, Nov. 2009, pp. 213–228. ISBN 978-3-525-40154-5