Metamorphic Testing

Tsong Yueh Chen; Fei-Ching Kuo; Huai Liu; Pak-Lok Poon; Dave Towey; T. H. Tse; Zhi Quan Zhou

doi:10.1145/3143561

journal article Jan 04, 2018

Metamorphic Testing

Tsong Yueh Chen Fei-Ching Kuo Huai Liu

Pak-Lok Poon Dave Towey

T. H. Tse Zhi Quan Zhou

ACM Computing Surveys Vol. 51 No. 1 pp. 1-27 · Association for Computing Machinery (ACM)

View at Publisher Save 10.1145/3143561

Abstract

Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs. Since its first publication, we have witnessed a rapidly increasing body of work examining metamorphic testing from various perspectives, including metamorphic relation identification, test case generation, integration with other software engineering techniques, and the validation and evaluation of software systems. In this article, we review the current research of metamorphic testing and discuss the challenges yet to be addressed. We also present visions for further improvement of metamorphic testing and highlight opportunities for new research.

Topics

No keywords indexed for this article. Browse by subject →

References

97

[1]

10.1145/2479440.2482677

[2]

10.1109/12.2185

[3]

Chittineni Aruna and R. Siva Ram Prasad. 2014. Testing approach for dynamic web applications based on automated test strategies . In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing , Vol. 249 . Springer, Berlin, 399--410. Chittineni Aruna and R. Siva Ram Prasad. 2014. Testing approach for dynamic web applications based on automated test strategies. In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing, Vol. 249. Springer, Berlin, 399--410.

[4]

10.1109/tse.2014.2372785

[5]

10.1109/tc.2016.2547380

[6]

Arlinta Christy Barus , Tsong Yueh Chen , Fei-Ching Kuo , Huai Liu , and Heinz W. Schmidt . 2016. The impact of source test case selection on the effectiveness of metamorphic testing . In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16) . ACM, New York, NY, 5--11. Arlinta Christy Barus, Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, and Heinz W. Schmidt. 2016. The impact of source test case selection on the effectiveness of metamorphic testing. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16). ACM, New York, NY, 5--11.

[7]

10.1109/compsac.2006.161

[8]

10.1109/ms.1984.233702

[9]

10.1016/j.future.2008.12.001

[10]

10.1145/2408776.2408795

[11]

10.1109/qsic.2013.43

[12]

10.1142/s0218194006002951

[13]

10.4018/jwsr.2007040103

[14]

Tsong Yueh Chen Joshua W. K. Ho Huai Liu and Xiaoyuan Xie. 2009. An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10 article no. 24. Tsong Yueh Chen Joshua W. K. Ho Huai Liu and Xiaoyuan Xie. 2009. An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10 article no. 24. 10.1186/1471-2105-10-24

[15]

Tsong Yueh Chen , De Hao Huang , T. H. Tse , and Zhi Quan Zhou . 2004 . Case studies on the selection of useful relations in metamorphic testing . In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC’04) . Polytechnic University of Madrid, Madrid, Spain, 569--583. Tsong Yueh Chen, De Hao Huang, T. H. Tse, and Zhi Quan Zhou. 2004. Case studies on the selection of useful relations in metamorphic testing. In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC’04). Polytechnic University of Madrid, Madrid, Spain, 569--583.

[16]

10.1007/978-3-642-02138-1_19

[17]

Tsong Yueh Chen , Fei-Ching Kuo , Ying Liu , and Antony Tang . 2004 . Metamorphic testing and testing with special values . In Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing (SNPD’04) . International Association for Computer and Information Science, Mt. Pleasant, MI, 128--134. Tsong Yueh Chen, Fei-Ching Kuo, Ying Liu, and Antony Tang. 2004. Metamorphic testing and testing with special values. In Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing (SNPD’04). International Association for Computer and Information Science, Mt. Pleasant, MI, 128--134.

[18]

10.1109/mc.2016.176

[19]

10.1016/j.jss.2009.02.022

[20]

Tsong Yueh Chen Fei-Ching Kuo Dave Towey and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58 5 052104:1--052104:9. Tsong Yueh Chen Fei-Ching Kuo Dave Towey and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58 5 052104:1--052104:9. 10.1007/s11432-015-5314-x

[21]

10.1145/1083231.1083236

[22]

10.1109/tse.2003.1214323

[23]

10.1016/j.jss.2015.07.037

[24]

10.1016/s0164-1212(01)00028-0

[25]

10.1145/566172.566202

[26]

10.1016/s0950-5849(02)00129-5

[27]

10.1109/tse.2010.23

[28]

David Cohen , Mikael Lindvall , and Patricia Costa . 2004. An introduction to agile methods . In Advances in Computers , Vol. 62 . Elsevier, Amsterdam , The Netherlands , 1--66. David Cohen, Mikael Lindvall, and Patricia Costa. 2004. An introduction to agile methods. In Advances in Computers, Vol. 62. Elsevier, Amsterdam, The Netherlands, 1--66.

[29]

10.1145/355604.361591

[30]

Junhua Ding , Xin-Hua Hu , and Venkat Gudivada . 2017. A machine learning based framework for verification and validation of massive scale image data . IEEE Transactions on Big Data . DOI:10.1109/TBDATA.2017.2680460. 10.1109/TBDATA.2017.2680460 Junhua Ding, Xin-Hua Hu, and Venkat Gudivada. 2017. A machine learning based framework for verification and validation of massive scale image data. IEEE Transactions on Big Data. DOI:10.1109/TBDATA.2017.2680460.

[31]

Guowei Dong , Baowen Xu , Lin Chen , Changhai Nie , and Lulu Wang . 2008 . Case studies on testing with compositional metamorphic relations . Journal of Southeast University (English Edition) 24 , 4, 437 -- 443 . Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2008. Case studies on testing with compositional metamorphic relations. Journal of Southeast University (English Edition) 24, 4, 437--443.

[32]

Guowei Dong , Baowen Xu , Lin Chen , Changhai Nie , and Lulu Wang . 2009 . Survey of metamorphic testing . Journal of Frontiers of Computer Science and Technology 3 , 2, 130 -- 143 . Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2009. Survey of metamorphic testing. Journal of Frontiers of Computer Science and Technology 3, 2, 130--143.

[33]

10.1145/1181775.1181806

[34]

10.1145/359488.359495

[35]

10.1109/cmpsac.2003.1245319

[36]

10.1109/qsic.2007.4385527

[37]

Richard Hamlet . 2002. Random testing . In Encyclopedia of Software Engineering . John Wiley , New York, NY . Richard Hamlet. 2002. Random testing. In Encyclopedia of Software Engineering. John Wiley, New York, NY.

[38]

10.1109/icst.2015.7102580

[39]

Peifeng Hu , Zhenyu Zhang , Wing Kwong Chan , and T. H. Tse . 2006. An empirical comparison between direct and indirect test result checking approaches . In Proceedings of the 3rd International Workshop on Software Quality Assurance (SOQUA’06) in Conjunction with the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14). ACM, New York, NY, 6--13. Peifeng Hu, Zhenyu Zhang, Wing Kwong Chan, and T. H. Tse. 2006. An empirical comparison between direct and indirect test result checking approaches. In Proceedings of the 3rd International Workshop on Software Quality Assurance (SOQUA’06) in Conjunction with the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14). ACM, New York, NY, 6--13.

[40]

10.1109/wcse.2013.16

[41]

10.5555/257734.257766

[42]

Mingyue Jiang , Tsong Yueh Chen , Fei-Ching Kuo , and Zuohua Ding . 2013 . Testing central processing unit scheduling algorithms using metamorphic testing . In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS’13) . IEEE Computer Society, Los Alamitos, CA, 530--536. Mingyue Jiang, Tsong Yueh Chen, Fei-Ching Kuo, and Zuohua Ding. 2013. Testing central processing unit scheduling algorithms using metamorphic testing. In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS’13). IEEE Computer Society, Los Alamitos, CA, 530--536.

[43]

10.1016/j.jss.2016.04.002

[44]

10.1109/compsac.2015.79

[45]

10.1002/stvr.1594

[46]

Andrew J. Ko , Robin Abraham , Laura Beckwith , Alan Blackwell , Margaret Burnett , Martin Erwig , Chris Scaffidi , Joseph Lawrance , Henry Lieberman , Brad Myers , Mary Beth Rosson , Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011 . The state of the art in end-user software engineering. ACM Computing Surveys 43, 3, 21:1--21:44. Andrew J. Ko, Robin Abraham, Laura Beckwith, Alan Blackwell, Margaret Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, Henry Lieberman, Brad Myers, Mary Beth Rosson, Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011. The state of the art in end-user software engineering. ACM Computing Surveys 43, 3, 21:1--21:44.

[47]

10.1109/lcn.2011.6115306

[48]

10.1145/2594291.2594334

[49]

Christopher Lidbury , Andrei Lascu , Nathan Chong , and Alastair F. Donaldson . 2015. Many-core compiler fuzzing . In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’15) . ACM, New York, NY, 65--76. Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core compiler fuzzing. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’15). ACM, New York, NY, 65--76.

[50]

Mikael Lindvall , Dharmalingam Ganesan , Ragnar Árdal , and Robert E. Wiegand . 2015. Metamorphic model-based testing applied on NASA DAT: An experience report . In Proceedings of the 37th International Conference on Software Engineering (ICSE’15) , Vol. 2 . IEEE, Piscataway, NJ, 129--138. Mikael Lindvall, Dharmalingam Ganesan, Ragnar Árdal, and Robert E. Wiegand. 2015. Metamorphic model-based testing applied on NASA DAT: An experience report. In Proceedings of the 37th International Conference on Software Engineering (ICSE’15), Vol. 2. IEEE, Piscataway, NJ, 129--138.

Showing 50 of 97 references

Cited By

398

When Specifications Meet Reality: Uncovering API Inconsistencies in Ethereum Infrastructure

Jie Ma, Ningyu He · 2026

Proceedings of the ACM on Programmi...

Metrics

398

Citations

97

References

Details