Logo PTI
Polish Information Processing Society
Logo FedCSIS

Annals of Computer Science and Information Systems, Volume 11

Proceedings of the 2017 Federated Conference on Computer Science and Information Systems

Evaluation of Mutant Sampling Criteria in Object-Oriented Mutation Testing

,

DOI: http://dx.doi.org/10.15439/2017F375

Citation: Proceedings of the 2017 Federated Conference on Computer Science and Information Systems, M. Ganzha, L. Maciaszek, M. Paprzycki (eds). ACSIS, Vol. 11, pages 1315–1324 ()

Full text

Abstract. Mutation testing of object-oriented programs differs from that of standard (traditional) mutation operators in accordance to the number of generated mutants and ability of tests to kill mutants. Therefore, outcomes of cost reduction analysis cannot be directly transferred from a standard mutation to an object-oriented one. Mutant sampling is one of reduction methods of the number of generated and tested mutants. We proposed different mutant sampling criteria based on equivalence partitioning in respect to object-oriented program features. The criteria were experimentally evaluated for object-oriented and standard mutation operators applied in C# programs. We compared results using a quality metric, which combines mutation score accuracy with mutation cost factors. In result, class random sampling and operator random sampling are recommended for OO and standard mutation testing, accordingly. With a reasonable decline of result accuracy, the mutant sampling technique is easily applicable in comparison to other cost reduction techniques.

References

  1. Y. Jia and M. Harman, “An analysis and survey of the development of mutation testing,” IEEE Transactions on Software Engineering, vol. 37, no.5, pp. 649—678, Sept-Oct. 2011, https://dx.doi.org/10.1109/TSE.2010.62
  2. A. Derezińska and A. Szustek, “Object-Oriented testing capabilities and performance evaluation of the C# mutation system,” in Proc. CEE-SET 2009, Szmuc, T., Szpyrka, M., Zendulka, J. Eds., LNCS, vol. 7054, 2012, pp. 229–242, https://dx.doi.org/10.1007/978-3-642-28038-2_18
  3. M. P. Usaola and P. R. Mateo, “Mutation testing cost reduction techniques: a survey,” IEEE Software, vol. 27, no. 3, pp. 80–86, May-June 2010, https://dx.doi.org/10.1109/MS.2010.79
  4. A. Derezińska and M. Rudnik, “Quality evaluation of Object-Oriented and standard mutation operators applied to C# programs,” in Proc. TOOLS Europe 2012, C.A. Furia, S. Nanz Eds., LNCS, vol. 7304, Springer Berlin Heildelberg, 2012, pp. 42–57, https://dx.doi.org/10.1007/978-3-642-30561-0_5
  5. A. P. Mathur and W. E Wong, “Reducing the cost of mutation testing: an empirical study,” J. of Systems and Software, vol. 31, no. 3, pp. 185–196, Dec. 1995, http://dx.doi.org/10.1016/0164-1212(94)00098-0
  6. J. Offut, G. Rothermel, and C.Zapf, “An experimental evaluation of selective mutation,” in Proc. 15th International Conference on Software Engineering, IEEE Comp. Soc. Press, 1993, pp. 100–107, https://dx.doi.org/10.1109/ICSE.1993.346062
  7. A. Derezińska, “A quality estimation of mutation clustering in C# programs,” in New Results in Dependability and Computer Systems W. Zamojski et al. Eds., AISC vol. 224, Springer, Switzerland, 2013, pp.183-194, https://dx.doi.org/10.1007/978-3-319-00945-2_11
  8. A. T. Acree, “On Mutation,”, Ph.D. thesis, Georgia Institute of Technology, Atlanta, GA, 1980.
  9. T. A. Budd, “Mutation analysis of program test data,” Ph.D. thesis, Yale University, New Haven, CT, 1980.
  10. G. J. Myers, The Art of Software Testing, John Wiley & Sons, 1979, 3rd. ed 2011
  11. M. Kryszkiewicz, “Fast algorithm finding minima in monotonic Boolean functions,” Warsaw Univ. of Technology, ICS Res Rep. 42/93, 1993.
  12. A. Derezińska and A. Szustek, “Tool-supported mutation approach for verification of C# programs,” in Proc. International Conference on Dependability of Computer Systems, W. Zamojski, et al. Eds., pp. 261–268, 2008, https://dx.doi.org/10.1109/DepCoS-RELCOMEX.2008.51
  13. CREAM, http://galera.ii.pw.edu.pl/~adr/CREAM/
  14. E. F. Barbosa, J.C. Maldonado, and A.M.R. Vincenzi, “Toward the determination of sufficient mutant operators for C,” Softw. Test. Verif. and Reliab. vol. 11, pp. 113–136, June 2001, https://dx.doi.org/10.1002/stvr.226
  15. L Zhang, S-S., Hou, J-J. Hu, T., Xie, and H. Mei, “Is operator-based mutant selection superior to random mutant selection?” in Proc. 32nd International Conference on Software Engineering, ICSE 2010, 2010, pp. 435–444, https://dx.doi.org/10.1145/1806799.1806863
  16. J. Hu, N. Li, and J. Offutt, “An analysis of OO mutation operators,” in Proc. of 4th International Conference Software Testing Verification and Validation Workshops, 6th Workshop on Mutation Analysis, IEEE Comp. Soc., 2011, pp. 334–341, https://dx.doi.org/10.1109/ICSTW.2011.47
  17. K. Adamopoulos, M. Harman, and R. M. Hierons, “How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution,” GECCO’04, LNCS, vol. 3103, pp. 1338-1349. Springer, 2004, https://dx.doi.org/10.1007/978-3-540-24855-2_155
  18. M. Scholive, V. Beroulle, C. Robach, M. L. Flottes, and B. Rouzeyre, “Mutation sampling technique for the generation of structural test data,” in Proc. of the Conference on Design, Automation and Test in Europe, DATE’05, vol. 2, pp.1022 – 1023. IEEE Comp. Soc., 2005.
  19. C. Sun, F. Xue, H. Liu, and X. Zhang, “A path-aware approach to mutant reduction in mutation testing,” Information and Software Technology, vol. 81, pp. 65-81, Jun. 2017, https://dx.doi.org/10.1016/j.infsof.2016.02.006
  20. S. Segura, R. M. Hierons, D. Benavides, and A. Ruiz-Cortes, “Mutation testing on an object-oriented framework: An experience report,” Information and Software Technology, 53(10), pp. 1124—1136, Oct. 2011, https://dx.doi.org/10.1016/j.infsof.2011.03.006
  21. L. Zhang, M. Gligoric, D. Marinov, and S. Khurshid, “Operator-based and random mutant selection: better together,” in 28th IEEE/ACM Conference on Automated Software Engineering, Palo Alto, USA, 2013, pp. 92-102, https://dx.doi.org/10.1109/ASE.2013.6693070
  22. I. Bluemke and K. Kulesza, “Reduction of computational cost in mutation testing by sampling mutants,” in New Results in Dependability and Computer System, W. Zamojski at al. Eds., Springer, 2013, pp. 41-51, https://dx.doi.org/10.1007/978-3-319-07013-1_9
  23. Y.-S. Ma and S.-W. Kim, „Mutation testing cost reduction by clustering overlapped mutants,” J. of Systems and Software, vol. 115, pp. 18-30, May 2016, http://dx.doi.org/10.1016/j.jss.2016.01.007
  24. P. Delgado-Perez, S. Segura, and S. Media-Bulo, „Assessment of C++ object-oriented mutation operators: A selective mutation approach,” Softw TestVerif Reliab., 2017, https://dx.doi.org/10.1002/stvr.1630
  25. A. Estero-Botaro, F. Palomo-Lozano, I. Medina-Bulo, J. J. Dominguez-Jimenez, and A. Garcia-Dominguez, “Quality metrics for mutation testing with application to WS-BPL compositions,” Softw Test Verif Reliab, vol. 25, no. 5-7, pp. 536-571, Aug.-Nov. 2015, https://dx.doi.org/10.1002/stvr.1528
  26. M. Moghadam and S. Babamir, “Mutation score evaluation in terms of object-oriented metrics,” 4th International eConference on Computer and Knowledge Engineering (ICCKE), 2014, Mashhad, Iran 2014, pp. 775–780, https://dx.doi.org/10.1109/ICCKE.2014.6993419
  27. A. Derezińska and P. Trzpil, “Mutation testing process combined with Test-Driven Development in .NET Environment,” in Theory and Engineering of Complex Systems and Dependability, W. Zamojski et al. Eds., AISC vol. 365, Springer, pp. 131-140, 2015, https://dx.doi.org/10.1007/978-3-319-19216-1_13