Logo PTI Logo FedCSIS

Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 43

Cognitive-Aware Peer Assessment: Design Implications from a Classroom Deployment

,

DOI: http://dx.doi.org/10.15439/2025F8584

Citation: Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS), M. Bolanowski, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 43, pages 641–646 ()

Full text

Abstract. Peer assessment is widely used in higher education, yet the cognitive demands placed on student assessors, particularly under conditions of overload and repetition, remain poorly understood. We examine how two cognitive factors, information overload and what we term assessment fatigue, influence evaluation behavior and user experience. Assessment fatigue is defined as cognitive strain resulting from repeated evaluative tasks. The study draws on data from a university-level deployment of a structured peer evaluation system. Behavioral metrics and self-report questionnaires were analyzed using Structural Equation Modeling (SEM). Results reveal a significant indirect pathway from information overload to system satisfaction, mediated by fatigue. Based on these findings, we propose design recommendations for cognitively-aware assessment systems that adapt to students' cognitive constraints, contributing to the development of AI-supported educational tools that are more robust and human-centered.

References

  1. Roy F. Baumeister, Ellen Bratslavsky, Mark Muraven, and Dianne M. Tice. Ego depletion: Is the active self a limited resource? Journal of Personality and Social Psychology, 74(5):1252–1265, 1998. https://doi.org/10.1037/0022-3514.74.5.1252.
  2. David Bawden and Lyn Robinson. The dark side of information: overload, anxiety and other paradoxes and pathologies. Journal of information science, 35(2):180–191, 2009. https://doi.org/10.1177/0165551508095781.
  3. Gee-Woo Bock, Mimrah Mahmood, Sanjeev Sharma, and Youn Jung Kang. The impact of information overload and contribution overload on continued usage of electronic knowledge repositories. Journal of Organizational Computing and Electronic Commerce, 20(3):257–278, 2010. https://doi.org/10.1080/10919392.2010.494530.
  4. Chao-hsiu Chen. The implementation and evaluation of a mobile self-and peer-assessment system. Computers & Education, 55(1):229–236, 2010. https://doi.org/10.1016/j.compedu.2010.01.008.
  5. Olga Chernikova, Daniel Sommerhoff, Matthias Stadler, Doris Holzberger, Michael Nickl, Tina Seidel, Enkelejda Kasneci, Stefan Küchemann, Jochen Kuhn, Frank Fischer, et al. Personalization through adaptivity or adaptability? a meta-analysis on simulation-based learning in higher education. Educational Research Review, page 100662, 2024. https://doi.org/10.1016/j.edurev.2024.100662.
  6. Shai Danziger, Jonathan Levav, and Liora Avnaim-Pesso. Extraneous factors in judicial decisions. Proceedings of the National Academy of Sciences, 108(17):6889–6892, 2011. https://doi.org/10.1073/pnas.1018033108.
  7. Amir Darvishi, Hadi Khosravi, Shazia Sadiq, and Dragan Gašević. Incorporating ai and learning analytics to build trustworthy peer assessment systems. British Journal of Educational Technology, 2022. https://doi.org/10.1111/bjet.13233.
  8. Lihi Dery. Interactive and iterative peer assessment. In Proceedings of the European Conference on Artificial Intelligence (ECAI), https://doi.org/10.3233/FAIA240656, 2024.
  9. Martin J. Eppler and Jeanne Mengis. The concept of information overload: A review of literature from organization science, accounting, marketing, mis, and related disciplines. The Information Society, 20(5):325–344, 2004. https://doi.org/10.1080/01972240490507974.
  10. Frank Fischer, Elisabeth Bauer, Tina Seidel, Ralf Schmidmaier, Anika Radkowitsch, Birgit J Neuhaus, Sarah I Hofer, Daniel Sommerhoff, Stefan Ufer, Jochen Kuhn, et al. Representational scaffolding in digital simulations–learning professional practices in higher education. Information and Learning Sciences, 123(11/12):645–665, 2022. https://doi.org/10.1108/ILS-06-2022-0076.
  11. Janusz A Hołyst, Philipp Mayr, Michael Thelwall, Ingo Frommholz, Shlomo Havlin, Alon Sela, Yoed N Kenett, Denis Helic, Aljoša Rehar, Sebastijan R Maček, et al. Protect our environment from information overload. Nature Human Behaviour, 8(3):402–403, 2024. https://doi.org/10.1038/s41562-024-01833-8.
  12. Felix Krieglstein, Maik Beege, Günter Daniel Rey, Christina Sanchez-Stockhammer, and Sascha Schneider. Development and validation of a theory-based questionnaire to measure different types of cognitive load. Educational Psychology Review, 35(1):9, 2023. https://doi.org/10.1007/s10648-023-09738-0.
  13. Chinmay Kulkarni, Kristine P. Wei, Harini Le, Daniel Chia, Katharine Papadopoulos, Justin Cheng, Daphne Koller, and Scott R. Klemmer. Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction, 20(6):1–31, 2013. https://doi.org/10.1145/2505057.
  14. H. Li, Y. Xiong, C. V. Hunter, X. Guo, and R. Tywoniw. Does peer assessment promote student learning? a meta-analysis. Assessment & Evaluation in Higher Education, 45(2):193–211, 2020. https://doi.org/10.1080/02602938.2019.1620679.
  15. Ernesto Panadero, M. Romero, and J.-W. Strijbos. The impact of a rubric and friendship on peer assessment: Effects on construct validity, performance, and perceptions of fairness and comfort. Studies in Educational Evaluation, 39(4):195–203, 2013. https://doi.org/10.1016/j.stueduc.2013.10.005.
  16. Kumar Raman and Thorsten Joachims. Methods for ordinal peer grading. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1037–1046, 2014. https://doi.org/10.1145/2623330.2623654.
  17. Nihar B. Shah, James K. Bradley, Ameet Parekh, Martin J. Wainwright, and Kannan Ramchandran. A case for ordinal peer evaluation in moocs. In NIPS Workshop on Data Driven Education, 2013.
  18. Ivan Stelmakh, Nihar B. Shah, and Aarti Singh. Catch me if i can: Detecting strategic behaviour in peer assessment. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):4794–4802, 2021. https://doi.org/10.1609/aaai.v35i6.16611.
  19. Keith Topping. Peer assessment between students in colleges and universities. Review of Educational Research, 68(3):249–276, 1998.
  20. Keith J. Topping. Peer assessment. Theory Into Practice, 48(1):20–27, 2009.
  21. Toby Walsh. The peerrank method for peer assessment. In ECAI, pages 909–914, 2014. https://doi.org/10.3233/978-1-61499-419-0-909.
  22. Bahman Zohuri and Farhang Mossavar-Rahmani. Revolutionizing education: The dynamic synergy of personalized learning and artificial intelligence. International Journal of Advanced Engineering and Management Research, 9(1):143–153, 2024. https://doi.org/10.51505/ijaemr.2024.9111.