Instance Space Analysis of Testing of Autonomous Vehicles in Critical Scenarios

Victor Crespo-Rodriguez; Neelofar; Aldeida Aleti; Burak Turhan

doi:10.1145/3699596

journal article Open Access Feb 23, 2025

Instance Space Analysis of Testing of Autonomous Vehicles in Critical Scenarios

Victor Crespo-Rodriguez

Neelofar

Aldeida Aleti

Burak Turhan

ACM Transactions on Software Engineering and Methodology Vol. 34 No. 3 pp. 1-36 · Association for Computing Machinery (ACM)

View at Publisher Save 10.1145/3699596

Abstract

Before being deployed on roads, Autonomous Vehicles (AVs) must undergo comprehensive testing. Safety-critical situations, however, are infrequent in usual driving conditions, so simulated scenarios are used to create them. A test scenario comprises static and dynamic features related to the AV and the test environment; the representation of these features is complex and makes testing a heavy process. A test scenario is effective if it identifies incorrect behaviors of the AV. In this article, we present a technique for identifying key features of test scenarios associated with their effectiveness using Instance Space Analysis (ISA). ISA generates a (

\(2D\)

) representation of test scenarios and their features. This visualization helps to identify combinations of features that make a test scenario effective. We present a graphical representation of each feature that helps identify how well each testing technique explores the search space. While identifying key features is a primary goal, this study specifically seeks to determine the critical features that differentiate the performance of algorithms. Finally, we present metrics to assess the robustness of testing algorithms and the scenarios generated. Collecting essential features in combination with their values associated with effectiveness can be used for selection and prioritization of effective test cases.

Topics

No keywords indexed for this article. Browse by subject →

References

76

[1]

BeamNG.tech. Retrieved from https://beamng.tech/

[2]

GitHub. ApolloAuto/apollo: An open autonomous driving platform. Retrieved July 1 2024 from https://github.com/ApolloAuto/apollo

[3]

10.1145/3238147.3238192

[4]

National Highway Traffic Safety Administration. 2022. Summary Report: Standing General Order on Crash Reporting for Automated Driving Systems. 9 pages. Retrieved from https://www.nhtsa.gov/sites/nhtsa.gov/files/2022-06/ADS-SGO-Report-June-2022.pdf

[5]

10.1109/tse.2023.3243522

[6]

Aldeida Aleti and Matias Martinez. 2021. E-APR: Mapping the effectiveness of automated program repair techniques. Empirical Software Engineering 26 (2021) 1–30. 10.1007/s10664-021-09989-x

[7]

10.1016/j.ejor.2022.04.012

[8]

10.1109/iccima.2007.328

[9]

10.1109/icst49551.2021.00042

[10]

Search-Based test case prioritization for simulation-Based testing of cyber-Physical system product lines

Aitor Arrieta, Shuai Wang, Goiuria Sagardui et al.

Journal of Systems and Software 10.1016/j.jss.2018.09.055

[11]

10.1109/ivs.2018.8500632

[12]

10.1109/saner53432.2022.00030

[13]

Christian Birchler Sajad Khatiri Pouria Derakhshanfar Sebastiano Panichella and Annibale Panichella. 2021. Automated test cases prioritization for self-driving cars in virtual environments. arXiv:2107.09614. Retrieved from https://arxiv.org/abs/2107.09614

[14]

10.1145/3533818

[15]

10.1109/qrs54544.2021.00028

[16]

10.1109/sbst52555.2021.00016

[17]

10.1109/tits.2021.3091477

[18]

Junjie Chen Ming Yan Zan Wang Yuning Kang and Zhuo Wu. 2020. Deep neural network test coverage: How far are we? arXiv:2010.04946.

[19]

10.1145/3644032.3644457

[20]

10.1037/met0000079

[21]

10.1145/3540250.3549152

[22]

10.1109/tits.2023.3259322

[23]

10.1109/tit.1983.1056714

[24]

Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD ’96), Vol. 96, 226–231.

[25]

10.1371/journal.pone.0184952

[26]

10.1145/3526072.3527538

[27]

10.1109/icse-companion.2019.00030

[28]

10.1145/3377811.3380391

[29]

Aurélien Géron. 2022. Hands-on Machine Learning with Scikit-Learn Keras and TensorFlow. O’Reilly Media Inc.

[30]

10.1109/iv48863.2021.9576023

[31]

10.1109/iv47402.2020.9304636

[32]

10.1109/itsc.2019.8917326

[33]

Zhisheng Hu Shengjian Guo Zhenyu Zhong and Kang Li. 2021. Coverage-based scene fuzzing for virtual autonomous driving testing. arXiv:2106.00873. Retrieved from https://arxiv.org/abs/2106.00873

[34]

10.1109/sbst52555.2021.00019

[35]

Bernhard Kaiser. 2021. Application Story of ODD as part of Safety Assurance.

[36]

Nidhi Kalra and Susan M. Paddock. 2016. Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability? RAND Corporation 1–16. Retrieved from http://www.jstor.org/stable/10.7249/j.ctt1btc0xw.1 10.7249/rr1478

[37]

10.1016/j.ijforecast.2016.09.004

[38]

10.1109/les.2016.2644619

[39]

10.1109/icse.2019.00108

[40]

10.1109/cvpr42600.2020.01426

[41]

10.1145/3550270

[42]

10.1109/issre5003.2020.00012

[43]

10.1109/tse.2022.3150788

[44]

10.1007/978-3-030-88106-1_4

[45]

10.1145/3238147.3238202

[46]

Kang Meng Rui Zhou Zhiheng Li and Kai Zhang. 2023. A quantitative approach of generating challenging testing scenarios based on functional safety standard. Applied Sciences 13 6 (Mar. 2023) 3494. DOI: 10.3390/app13063494 10.3390/app13063494

[47]

10.1016/s0001-4575(00)00019-1

[48]

10.1109/sbst52555.2021.00018

[49]

Jean-Baptiste Mouret and Jeff Clune. 2015. Illuminating search spaces by mapping elites. arXiv:1504.04909. Retrieved from http://arxiv.org/abs/1504.04909

[50]

10.1145/3436893

Showing 50 of 76 references

Cited By

2

The role of road features and vehicle dynamics in cost-effective autonomous vehicles safety testing: Insights from instance space analysis

Victor Crespo-Rodriguez, Christian Birchler · 2026

Information and Software Technology

Metrics

2

Citations

76

References

Details

Published: Feb 23, 2025
Vol/Issue: 34(3)
Pages: 1-36
License: View

Authors

V

Victor Crespo-Rodriguez

Faculty of Information Technology, Monash University, Clayton, Australia

N

Neelofar

Faculty of Information Technology, Monash University, Clayton, Australia and RMIT University, Melbourne, Australia

A

Aldeida Aleti

Faculty of Information Technology, Monash University, Clayton, Australia

B

Burak Turhan

Faculty of ITEE, University of Oulu, Oulu, Finland and Monash University, Clayton, Australia

Funding

Australian Research Council Award: DP210100041

Cite This Article

Victor Crespo-Rodriguez, Neelofar, Aldeida Aleti, et al. (2025). Instance Space Analysis of Testing of Autonomous Vehicles in Critical Scenarios. ACM Transactions on Software Engineering and Methodology, 34(3), 1-36. https://doi.org/10.1145/3699596

Instance Space Analysis of Testing of Autonomous Vehicles in Critical Scenarios

You May Also Like