Abstract
Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs. Since its first publication, we have witnessed a rapidly increasing body of work examining metamorphic testing from various perspectives, including metamorphic relation identification, test case generation, integration with other software engineering techniques, and the validation and evaluation of software systems. In this article, we review the current research of metamorphic testing and discuss the challenges yet to be addressed. We also present visions for further improvement of metamorphic testing and highlight opportunities for new research.
Topics

No keywords indexed for this article. Browse by subject →

References
97
[3]
Chittineni Aruna and R. Siva Ram Prasad. 2014. Testing approach for dynamic web applications based on automated test strategies . In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing , Vol. 249 . Springer, Berlin, 399--410. Chittineni Aruna and R. Siva Ram Prasad. 2014. Testing approach for dynamic web applications based on automated test strategies. In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing, Vol. 249. Springer, Berlin, 399--410.
[6]
Arlinta Christy Barus , Tsong Yueh Chen , Fei-Ching Kuo , Huai Liu , and Heinz W. Schmidt . 2016. The impact of source test case selection on the effectiveness of metamorphic testing . In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16) . ACM, New York, NY, 5--11. Arlinta Christy Barus, Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, and Heinz W. Schmidt. 2016. The impact of source test case selection on the effectiveness of metamorphic testing. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16). ACM, New York, NY, 5--11.
[14]
Tsong Yueh Chen Joshua W. K. Ho Huai Liu and Xiaoyuan Xie. 2009. An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10 article no. 24. Tsong Yueh Chen Joshua W. K. Ho Huai Liu and Xiaoyuan Xie. 2009. An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10 article no. 24. 10.1186/1471-2105-10-24
[15]
Tsong Yueh Chen , De Hao Huang , T. H. Tse , and Zhi Quan Zhou . 2004 . Case studies on the selection of useful relations in metamorphic testing . In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC’04) . Polytechnic University of Madrid, Madrid, Spain, 569--583. Tsong Yueh Chen, De Hao Huang, T. H. Tse, and Zhi Quan Zhou. 2004. Case studies on the selection of useful relations in metamorphic testing. In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC’04). Polytechnic University of Madrid, Madrid, Spain, 569--583.
[17]
Tsong Yueh Chen , Fei-Ching Kuo , Ying Liu , and Antony Tang . 2004 . Metamorphic testing and testing with special values . In Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing (SNPD’04) . International Association for Computer and Information Science, Mt. Pleasant, MI, 128--134. Tsong Yueh Chen, Fei-Ching Kuo, Ying Liu, and Antony Tang. 2004. Metamorphic testing and testing with special values. In Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing (SNPD’04). International Association for Computer and Information Science, Mt. Pleasant, MI, 128--134.
[20]
Tsong Yueh Chen Fei-Ching Kuo Dave Towey and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58 5 052104:1--052104:9. Tsong Yueh Chen Fei-Ching Kuo Dave Towey and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58 5 052104:1--052104:9. 10.1007/s11432-015-5314-x
[28]
David Cohen , Mikael Lindvall , and Patricia Costa . 2004. An introduction to agile methods . In Advances in Computers , Vol. 62 . Elsevier, Amsterdam , The Netherlands , 1--66. David Cohen, Mikael Lindvall, and Patricia Costa. 2004. An introduction to agile methods. In Advances in Computers, Vol. 62. Elsevier, Amsterdam, The Netherlands, 1--66.
[30]
Junhua Ding , Xin-Hua Hu , and Venkat Gudivada . 2017. A machine learning based framework for verification and validation of massive scale image data . IEEE Transactions on Big Data . DOI:10.1109/TBDATA.2017.2680460. 10.1109/TBDATA.2017.2680460 Junhua Ding, Xin-Hua Hu, and Venkat Gudivada. 2017. A machine learning based framework for verification and validation of massive scale image data. IEEE Transactions on Big Data. DOI:10.1109/TBDATA.2017.2680460.
[31]
Guowei Dong , Baowen Xu , Lin Chen , Changhai Nie , and Lulu Wang . 2008 . Case studies on testing with compositional metamorphic relations . Journal of Southeast University (English Edition) 24 , 4, 437 -- 443 . Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2008. Case studies on testing with compositional metamorphic relations. Journal of Southeast University (English Edition) 24, 4, 437--443.
[32]
Guowei Dong , Baowen Xu , Lin Chen , Changhai Nie , and Lulu Wang . 2009 . Survey of metamorphic testing . Journal of Frontiers of Computer Science and Technology 3 , 2, 130 -- 143 . Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2009. Survey of metamorphic testing. Journal of Frontiers of Computer Science and Technology 3, 2, 130--143.
[37]
Richard Hamlet . 2002. Random testing . In Encyclopedia of Software Engineering . John Wiley , New York, NY . Richard Hamlet. 2002. Random testing. In Encyclopedia of Software Engineering. John Wiley, New York, NY.
[39]
Peifeng Hu , Zhenyu Zhang , Wing Kwong Chan , and T. H. Tse . 2006. An empirical comparison between direct and indirect test result checking approaches . In Proceedings of the 3rd International Workshop on Software Quality Assurance (SOQUA’06) in Conjunction with the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14). ACM, New York, NY, 6--13. Peifeng Hu, Zhenyu Zhang, Wing Kwong Chan, and T. H. Tse. 2006. An empirical comparison between direct and indirect test result checking approaches. In Proceedings of the 3rd International Workshop on Software Quality Assurance (SOQUA’06) in Conjunction with the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14). ACM, New York, NY, 6--13.
[42]
Mingyue Jiang , Tsong Yueh Chen , Fei-Ching Kuo , and Zuohua Ding . 2013 . Testing central processing unit scheduling algorithms using metamorphic testing . In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS’13) . IEEE Computer Society, Los Alamitos, CA, 530--536. Mingyue Jiang, Tsong Yueh Chen, Fei-Ching Kuo, and Zuohua Ding. 2013. Testing central processing unit scheduling algorithms using metamorphic testing. In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS’13). IEEE Computer Society, Los Alamitos, CA, 530--536.
[46]
Andrew J. Ko , Robin Abraham , Laura Beckwith , Alan Blackwell , Margaret Burnett , Martin Erwig , Chris Scaffidi , Joseph Lawrance , Henry Lieberman , Brad Myers , Mary Beth Rosson , Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011 . The state of the art in end-user software engineering. ACM Computing Surveys 43, 3, 21:1--21:44. Andrew J. Ko, Robin Abraham, Laura Beckwith, Alan Blackwell, Margaret Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, Henry Lieberman, Brad Myers, Mary Beth Rosson, Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011. The state of the art in end-user software engineering. ACM Computing Surveys 43, 3, 21:1--21:44.
[49]
Christopher Lidbury , Andrei Lascu , Nathan Chong , and Alastair F. Donaldson . 2015. Many-core compiler fuzzing . In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’15) . ACM, New York, NY, 65--76. Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core compiler fuzzing. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’15). ACM, New York, NY, 65--76.
[50]
Mikael Lindvall , Dharmalingam Ganesan , Ragnar Árdal , and Robert E. Wiegand . 2015. Metamorphic model-based testing applied on NASA DAT: An experience report . In Proceedings of the 37th International Conference on Software Engineering (ICSE’15) , Vol. 2 . IEEE, Piscataway, NJ, 129--138. Mikael Lindvall, Dharmalingam Ganesan, Ragnar Árdal, and Robert E. Wiegand. 2015. Metamorphic model-based testing applied on NASA DAT: An experience report. In Proceedings of the 37th International Conference on Software Engineering (ICSE’15), Vol. 2. IEEE, Piscataway, NJ, 129--138.

Showing 50 of 97 references

Cited By
398
Proceedings of the ACM on Programmi...
Metrics
398
Citations
97
References
Details
Published
Jan 04, 2018
Vol/Issue
51(1)
Pages
1-27
License
View
Funding
Australian Research Council Award: LP160101691
University of Nottingham
General Research Fund of the Research Grants Council of Hong Kong Award: 716612
International Doctoral Innovation Centre
Artificial Intelligence and Optimisation Research Group of University of Nottingham Ningbo China
Cite This Article
Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, et al. (2018). Metamorphic Testing. ACM Computing Surveys, 51(1), 1-27. https://doi.org/10.1145/3143561
Related

You May Also Like

Data clustering

A. K. Jain, M. N. Murty · 1999

9,568 citations

Anomaly detection

Varun Chandola, Arindam Banerjee · 2009

8,799 citations

Machine learning in automated text categorization

Fabrizio Sebastiani · 2002

5,027 citations

Object tracking

Alper Yilmaz, Omar Javed · 2006

3,632 citations

A Survey on Bias and Fairness in Machine Learning

Ninareh Mehrabi, Fred Morstatter · 2021

3,466 citations