Logo PTI
Polish Information Processing Society
Logo FedCSIS

Annals of Computer Science and Information Systems, Volume 20

Communication Papers of the 2019 Federated Conference on Computer Science and Information Systems

A mixed-methods measurement and evaluation methodology for mobile application usability studies

DOI: http://dx.doi.org/10.15439/2019F299

Citation: Communication Papers of the 2019 Federated Conference on Computer Science and Information Systems, M. Ganzha, L. Maciaszek, M. Paprzycki (eds). ACSIS, Vol. 20, pages 101–106 ()

Full text

Abstract. Low usability of mobile application is thought to diminish the perceived level of the quality by a user whose experiences substantially determines its market success or failure. However, while a single method of measurement employed to study usability may produce an unreliable or incomplete evaluation outcome, in this paper, contrarily we propose to take advantage of both qualitative and quantitative methods adequate to collect data, that would describe all usability attributes. In particular, in the scope of mobile application usability studies, this paper (i) depicts the main assumptions of elaborated M4MAUME methodology, (ii) describes the self-developed software tool (RVDA) for retrospective video data analysis, (iii) specifies the experimental setup, and (iv) discusses the preliminary results obtained from 20 experiments performed on four different groups of mobile applications. Eventually our findings lead us to the conclusion that mixing different methods have produced reliable and valuable outcomes which may be used to improve and manage usability in current and future projects, as well as to enhance existing software quality assurance (SQA) programs.

References

  1. McGee, M., Rich, A., & Dumas, J. (2004). Understanding the usability construct: User-perceived usability. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 907–911.
  2. Sung, J., & Yun, Y. (2010). Toward a more robust usability concept with perceived enjoyment in the context of mobile multimedia service. International Journal of Human Computer Interaction, 1(2), 12–32.
  3. Koohang, A. (2004). Expanding the concept of usability. Informing Science: International Journal of an Emerging Transdiscipline, 7, 129–141.
  4. Jeffries, R., & Desurvire, H. (1992). Usability testing vs. heuristic evaluation: was there a contest? ACM SIGCHI Bulletin, 24(4), 39–41.
  5. Zhang, Z., Basili, V., & Shneiderman, B. (1999). Perspective-based usability inspection: An empirical validation of efficacy. Empirical Software Engineering, 4(1), 43–69.
  6. Goldberg, J. H., & Wichansky, A. M. (2003). Eye tracking in usability evaluation: A practitioner's guide. The Mind's Eye, 493–516.
  7. Norman, D. A., & Draper, S. W. (1986). User centered system design: New perspectives on human-computer interaction. CRC Press.
  8. Desmet, P. M., & Hekkert, P. (2007). Framework of product experience. International journal of design, 1(1), 57–66.
  9. Plattner, H., Meinel, C., & Leifer, L. (Eds.). (2010). Design thinking: understand–improve–apply. Springer Science & Business Media.
  10. Instone, K. (1997). Usability heuristics for the Web. Web Review, October.
  11. Nielsen, J. (2005). Ten usability heuristics.
  12. Paz, F., Paz, F. A., Pow-Sang, J. A., & Collantes, L. (2014). Usability heuristics for transactional web sites. 11th International Conference on Information Technology: New Generations, 627–628. IEEE.
  13. Weichbroth, P. (2018). Delivering Usability in IT Products: Empirical Lessons from the Field. International Journal of Software Engineering and Knowledge Engineering, 28(07), 1027–1045.
  14. Manchester, J. (2013). Software Revolution, Part IV: Computing in a Mobile World. https://www.forbes.com/sites/truebridge/2013/10/22/software-revolution-part-iv-computing-in-a-mobile-world/#325625c71cd5 [2019-05-14].
  15. Truong, A. (2016). China saw a massive shift from desktop to mobile computing in 2015. https://qz.com/622543/china-saw-a-massive-shift-from-desktop-to-mobile-computing-in-2015/ [2019-05-14].
  16. Varshney, U. (2003). Issues, requirements and support for location-intensive mobile commerce applications. International Journal of Mobile Communications, 1(3), 247–263.
  17. Wasserman, T. (2010). Software engineering issues for mobile application development. FoSER 2010.
  18. Parsons, D., Ryu, H., & Cranshaw, M. (2006, July). A study of design requirements for mobile learning environments. In Sixth IEEE International Conference on Advanced Learning Technologies (ICALT'06), 96–100. IEEE.
  19. Nilsson, E. G. (2009). Design patterns for user interface for mobile applications. Advances in engineering software, 40(12), 1318–1328.
  20. Hoebeke, J., Moerman, I., Dhoedt, B., & Demeester, P. (2004). An overview of mobile ad hoc networks: applications and challenges. Journal-Communications Network, 3(3), 60–66.
  21. Anagnostopoulos, C. B., Tsounis, A., & Hadjiefthymiades, S. (2007). Context awareness in mobile computing environments. Wireless Personal Communications, 42(3), 445–464.
  22. Raptis, D., Tselios, N., Kjeldskov, J., & Skov, M. B. (2013, August). Does size matter?: investigating the impact of mobile phone screen size on users' perceived usability, effectiveness and efficiency. In Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services, 127–136. ACM.
  23. Nkosi, M., & Mekuria, F. (2011). Improving the capacity, reliability & life of mobile devices with Cloud Computing. In 2011 IST-Africa Conference Proceedings, 1–9. IEEE.
  24. Wajahat, N. (2019). Mobile App Performance Testing: CheckList, Tools (Andriod & iOS). https://www.guru99.com/mobile-app-performance-testing-strategy-tools.html [2019-05-17].
  25. Weichbroth, P. (2019). Usability of mobile applications: a systematic literature study (in print).
  26. ISO 9241-11:2018(en). Ergonomics of human-system interaction. Part 11: Usability: Definitions and concepts.
  27. Harrison, R., Flood, D., & Duce, D. (2013). Usability of mobile applications: literature review and rationale for a new usability model. Journal of Interaction Science, 1(1).
  28. Ghazizadeh, F. Z., & Vafadar, S. (2017). A quantitative evaluation of usability in mobile applications: An empirical study. International Symposium on Computer Science and Software Engineering Conference (CSSE), 1–6. IEEE.
  29. Kabir, M. A., Salem, O. A., & Rehman, M. U. (2017). Discovering knowledge from mobile application users for usability improvement: A fuzzy association rule mining approach. In 2017 8th IEEE International Conference on Software Engineering and Service Science (ICSESS), 126–129. IEEE.
  30. Krzewińska, J., Indyka-Piasecka, A., Kopel, M., Kukla, E., Telec, Z., & Trawiński, B. (2018). Usability Testing of a Responsive Web System for a School for Disabled Children. In Asian Conference on Intelligent Information and Database Systems, 705–716. Springer.
  31. Widyanti, A., & Ainizzamani, S. A. Q. (2017). Usability evaluation of online transportation' user interface. International Conference on Information Technology Systems and Innovation, 82–86. IEEE.
  32. Saleh, A., Ismail, R., & Fabil, N. (2017). Evaluating usability for mobile application: a MAUEM approach. Proceedings of the 2017 International Conference on Software and e-Business, 71–77. ACM.
  33. McDonald, K., Abell, W., Smith, C., & Gibbs, S. (2016). Lessons learned from evaluating a mobile app out in the field. 4th International Conference on User Science and Engineering (i-USEr), 5–10. IEEE.
  34. do Nascimento Mendes, I., & Dias-Neto, A. C. (2016). A Process-Based Approach to Test Usability of Multi-platform Mobile Applications. International Conference of Design, User Experience, and Usability, 456–468. Springer.
  35. Aljaber, T., & Gordon, N. (2017). A guidance and evaluation approach for mHealth education applications. International Conf. on Learning and Collaboration Technologies, 330–340. Springer.
  36. Parsazadeh, N., Ali, R., & Rezaei, M. (2018). A framework for cooperative and interactive mobile learning to improve online information evaluation skills. Computers & Education, 120, 75–89.
  37. Lyons, K. M. (2014). U.S. Patent Application No. 13/995,514.
  38. Sweller, J., & Chandler, P. (1991). Evidence for cognitive load theory. Cognition and instruction, 8(4), 351–362.
  39. Chynał, P., Szymański, J. M., & Sobecki, J. (2012). Using eyetracking in a mobile applications usability testing. Asian Conf. on Intelligent Information and Database Systems, 178–186. Springer.
  40. Ryan, C., & Gonsalves, A. (2005). The effect of context and application type on mobile usability: an empirical study. Proceedings of the Twenty-eighth Australasian Conf. 38, 115–124.
  41. Wang, Y., & Brennan, R. (2017). Building High Usability Consumer-oriented Linked Data Mobile Apps. AICS, 295–307.
  42. Korhan, O., & Ersoy, M. (2016). Usability and functionality factors of the social network site application users from the perspective of uses and gratification theory. Quality & Quantity, 50(4), 1799–1816.
  43. Biel, B., Grill, T., & Gruhn, V. (2010). Exploring the benefits of the combination of a software architecture analysis and a usability evaluation of a mobile application. Journal of Systems and Software, 83(11), 2031–2044.
  44. de Paula, D. F., Menezes, B. H., & Araújo, C. C. (2014). Building a quality mobile application: A user-centered study focusing on design thinking, user experience and usability. In International Conference of Design, User Experience, and Usability, 313–322. Springer.
  45. Tomaschko, M., & Hohenwarter, M. (2018). Usability Evaluation of a Mobile Graphing Calculator Application Using Eye Tracking. In International Conference on Learning and Collaboration Technologies, 180–190. Springer.
  46. Merriam-Webster English Dictionary, https://www.merriam-webster.com/dictionary/methodology [2019-04-06]
  47. https://www.merriam-webster.com/dictionary/method
  48. https://www.merriam-webster.com/dictionary/rule
  49. https://www.merriam-webster.com/dictionary/postulate
  50. Denzin, N. K. (2017). The research act: A theoretical introduction to sociological methods. Routledge.
  51. Thurmond, V. A. (2001). The point of triangulation. Journal of nursing scholarship, 33(3), 253–258.
  52. Brown, P. A. (2008). A review of the literature on case study research. Canadian Journal for New Scholars in Education.
  53. Sale, J. E., Lohfeld, L. H., & Brazil, K. (2002). Revisiting the quantitative-qualitative debate: Implications for mixed-methods research. Quality and quantity, 36(1), 43–53.
  54. Altheide, D. L. & Johnson, J. M. (1994). Criteria for assessing interpretive validity in qualitative research. In: Denzin, N. K. & Lincoln, Y. S. (Eds), Handbook of Qualitative Research. Thousand Oaks, CA: Sage Publications, 485–499.
  55. Holtz, P., & Odağ, Ö. (2018). Popper was not a Positivist: Why Critical Rationalism Could be an Epistemology for Qualitative as well as Quantitative Social Scientific Research. Qualitative Research in Psychology, 1–24.
  56. Mack, N. (2005). Qualitative research methods: A data collector’s field guide.
  57. Miller, R. L., & Brewer, J. D. (Eds.). (2003). The A–Z of social research: a dictionary of key social science research concepts. Sage.
  58. Kasper, G. (1998). Analysing verbal protocols. Tesol Quarterly, 32(2), 358–362.
  59. Guterman, O., & Neuman, A. (2017). Different reasons for one significant choice: Factors influencing homeschooling choice in Israel. International Review of Education, 63(3), 303–318.
  60. Lockton, D., Harrison, D., Holley, T., & Stanton, N. A. (2009). Influencing interaction: development of the design with intent method. 4th International Conf. on Persuasive Technology, ACM.
  61. Van Den Haak, M., De Jong, M., & Jan Schellens, P. (2003). Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behaviour & information technology, 22(5), 339–351.
  62. Electron Documentation, https://electronjs.org/docs/tutorial/about.