A survey of deep learning techniques for autonomous driving

Sorin Grigorescu; Bogdan Trasnea; Tiberiu Cocias; Gigel Macesanu

doi:10.1002/rob.21918

journal article Nov 14, 2019

A survey of deep learning techniques for autonomous driving

Sorin Grigorescu

Bogdan Trasnea

Tiberiu Cocias

Gigel Macesanu

Journal of Field Robotics Vol. 37 No. 3 pp. 362-386 · Wiley

View at Publisher Save 10.1002/rob.21918

Abstract

AbstractThe last decade witnessed increasingly rapid progress in self‐driving vehicle technology, mainly backed up by advances in the area of deep learning and artificial intelligence (AI). The objective of this paper is to survey the current state‐of‐the‐art on deep learning technologies used in autonomous driving. We start by presenting AI‐based self‐driving architectures, convolutional and recurrent neural networks, as well as the deep reinforcement learning paradigm. These methodologies form a base for the surveyed driving scene perception, path planning, behavior arbitration, and motion control algorithms. We investigate both the modular perception‐planning‐action pipeline, where each module is built using deep learning methods, as well as End2End systems, which directly map sensory information to steering commands. Additionally, we tackle current challenges encountered in designing AI architectures for autonomous driving, such as their safety, training data sources, and computational hardware. The comparison presented in this survey helps gain insight into the strengths and limitations of deep learning and AI approaches for autonomous driving and assist with design choices.

Topics

No keywords indexed for this article. Browse by subject →

References

175

[1]

Amodei D. "Concrete problems in AI safety" arXiv preprint (2016)

[2]

Andrychowicz M. "Learning dexterous in‐hand manipulation" arXiv preprint (2018)

[3]

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

Vijay Badrinarayanan, Alex Kendall, Roberto Cipolla

IEEE Transactions on Pattern Analysis and Machine... 10.1109/tpami.2016.2644615

[4]

Barnes D. Maddern W. Pascoe G. &Posner I.(2018). Driven to distraction: Self‐supervised distractor learning for robust monocular visual odometry in urban environments. In2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE. 10.1109/icra.2018.8460564

[5]

Barsan I. A. Wang S. Pokrovsky A. &Urtasun R.(2018). Learning to localize using a LiDAR intensity map. InProceedings of the 2nd Conference on Robot Learning (CoRL).

[6]

10.1109/rtcsa.2018.00011

[7]

Bellman R. (1957)

[8]

Representation Learning: A Review and New Perspectives

Y. Bengio, A. Courville, P. Vincent

IEEE Transactions on Pattern Analysis and Machine... 10.1109/tpami.2013.50

[9]

Bernd S. Detlev R. Susanne E. Ulf W. Wolfgang B. &Carsten P.(2012). Challenges in applying the ISO 26262 for driver assistance systems. InSchwerpunkt Vernetzung 5. Tagung Fahrerassistenz.

[10]

Bojarski M. Del Testa D. Dworakowski D. Firner B. Flepp B. Goyal P. …Zhao J.(2016). End to End learning for self‐driving cars. arXiv preprint 1604.07316.

[11]

Bojarski M. Yeres P. Choromanska A. Choromanski K. Firner B. Jackel L. &Muller U.(2017). Explaining how a deep neural network trained with end‐to‐end learning steers a car. arXiv preprint 1704.07911.

[12]

Brachmann E. &Rother C.(2018). Learning less is more—6D camera localization via 3D surface regression. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018. 10.1109/cvpr.2018.00489

[13]

10.1109/tiv.2017.2749181

[14]

Semantic object classes in video: A high-definition ground truth database

Gabriel J. Brostow, Julien Fauqueur, Roberto Cipolla

Pattern Recognition Letters 10.1016/j.patrec.2008.04.005

[15]

10.1109/cdc.2017.8264027

[16]

10.1007/978-3-319-66284-8_1

[17]

Caesar H. Bankiti V. Lang A. H. Vora S. Liong V. E. Xu Q. …Beijbom O.(2019). nuScenes: A multimodal dataset for autonomous driving. arXiv preprint 1903.11027. 10.1109/cvpr42600.2020.01164

[18]

10.1145/2783258.2788613

[19]

Chakarov A. Nori A. Rajamani S. Sen S. &Vijaykeerthy D.(2018). Debugging machine learning tasks. arXiv preprint 1603.07292.

[20]

10.1109/iccv.2015.312

[21]

Chen X. Ma H. Wan J. Li B. &Xia T.(2017). Multi‐view 3D object detection network for autonomous driving. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017. 10.1109/cvpr.2017.691

[22]

Cityscapes. (2018).Cityscapes data collection. Retrieved fromhttps://www.cityscapes‐dataset.com/

[23]

Cong J. Fang Z. Lo M. Wang H. Xu J. &Zhang S.(2018). Understanding performance differences of FPGAs and GPUs: (Abtract only). InProceedings of the 2018 ACM/SIGDA International Symposium on Field‐Programmable Gate Arrays (FPGA ’18) (p. 288). New York NY: ACM. 10.1145/3174243.3174970

[24]

Dai J. Li Y. He K. &Sun J.(2016). R‐FCN: Object detection via region‐based fully convolutional networks.Advances in Neural Information Processing Systems NIPS 2016 379–387.

[25]

Dalal N. &Triggs B.(2005). Histograms of oriented gradients for human detection. InIEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR 2005.

[26]

10.1613/jair.1872

[27]

Dynamic monocular machine vision

Ernst Dieter Dickmanns, Volker Graefe

Machine Vision and Applications 10.1007/bf01212361

[28]

10.1109/cvpr.2009.5206631

[29]

Drews P. "Aggressive deep driving: Combining convolutional neural networks and model predictive control" Conference on Robot Learning (2017)

[30]

Duchi J. "Adaptive subgradient methods for online learning and stochastic optimization" Journal of Machine Learning Research (2011)

[31]

Eraqi H. M. "End‐to‐end deep learning for steering autonomous vehicles considering temporal dependencies" Machine Learning for Intelligent Transportation Systems Workshop in the 31st Conference on Neural Information Processing Systems NIPS 2017. (2017)

[32]

Faria J. M. "Machine Learning Safety: An Overview" Safety‐Critical Systems Club (2018)

[33]

Ferrel T.(2010).Engineering safety‐critical systems in the 21st century.

[34]

Flohr F. &Gavrila D. M.(2013). Daimler pedestrian segmentation benchmark dataset. InProceedings of the British Machine Vision Conference.

[35]

Fridman L. Brown D. E. Glazer M. Angell W. Dodd S. Jenik B. …Reimer B.(2017). MIT autonomous vehicle technology study: Large‐scale deep learning based analysis of driver behavior and interaction with automation. InIEEE Access 2017.

[36]

Garcia‐Favrot O. &Parent M.(2009). Laser scanner based SLAM in real road and traffic environment. InIEEE International Conference on Robotics and Automation (ICRA09). Workshop on Safe Navigation in Open and Dynamic Environments Application to Autonomous Vehicles.

[37]

Vision meets robotics: The KITTI dataset

A Geiger, P Lenz, C Stiller et al.

The International Journal of Robotics Research 10.1177/0278364913491297

[38]

Fast R-CNN

Ross Girshick

2015 IEEE International Conference on Computer Vis... 10.1109/iccv.2015.169

[39]

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

Ross Girshick, Jeff Donahue, Trevor Darrell et al.

2014 IEEE Conference on Computer Vision and Patter... 10.1109/cvpr.2014.81

[40]

10.2200/s00762ed1v01y201703hlt037

[41]

10.1016/0166-2236(92)90344-8

[42]

10.1109/lra.2019.2926224

[43]

Gu S. (2016)

[44]

10.1109/ivs.2016.7535466

[45]

Harris M. "Google reports self‐driving car mistakes: 272 Failures and 13 near misses" The Guardian (2016)

[46]

10.1109/itsc.2016.7795918

[47]

Mask R-CNN

Kaiming He, Georgia Gkioxari, Piotr Dollar et al.

2017 IEEE International Conference on Computer Vis... 10.1109/iccv.2017.322

[48]

Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren et al.

2016 IEEE Conference on Computer Vision and Patter... 10.1109/cvpr.2016.90

[49]

Hecker S. Dai D. &VanGool L.(2018). End‐to‐end learning of driving models with surround‐view cameras and route planners. InEuropean Conference on Computer Vision (ECCV). 10.1007/978-3-030-01234-2_27

[50]

Hessel M. "Rainbow: Combining improvements in deep reinforcement learning" Artificial Intelligence 2018? (2018)

Showing 50 of 175 references

Cited By

1,443

When Lie Groups Meet Hyperspectral Images: Equivariant Manifold Network for Few-Shot HSI Classification

Haolong Ban, Junchao Feng · 2026

Sensors

X-ray image translation based on weakly supervised CycleGAN

Ziteng Hu, Ji Hu · 2026

Journal of Visual Communication and...

LLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer

Kunpeng Zhang, Dongwei Xiao · 2026

Proceedings of the ACM on Programmi...

Exploring AI in metasurface structures with forward and inverse design

Guantai Yang, Qingxiong Xiao · 2025

iScience

Uncertainty Quantification for Safe and Reliable Autonomous Vehicles: A Review of Methods and Applications

KE WANG, Chongqiang Shen · 2025

IEEE Transactions on Intelligent Tr...

Do restaurant customers and owners differ in their preferences for serving robots? A choice experiment and latent class approach

Changeun Park, HyungBin Moon · 2025

Technology in Society

A Comprehensive Survey of Deep Learning Approaches in Image Processing

Maria Trigka, Elias Dritsas · 2025

Sensors

Dyn-Bitpool: A 28 nm 27 TOPS/W Two-Sided Sparse CIM Accelerator Featuring a Balanced Workload Scheme and High CIM Macro Utilization

Xujiang Xiang, Zhiheng Yue · 2025

IEEE Transactions on Circuits and S...

Semi-Supervised Risk Control via Prediction-Powered Inference

Bat-Sheva Einbinder, Liran Ringel · 2025

IEEE Transactions on Pattern Analys...

Adversarial machine learning: a review of methods, tools, and critical industry sectors

Sotiris Pelekis, Thanos Koutroubas · 2025

Artificial Intelligence Review

Comparison of Different Controller Architectures for Autonomous Driving and Recommendations for Robust and Safe Implementations

M. A. Shadab Siddiqui, M. S. Rabbi · 2025

Journal of Advanced Transportation

SWRD: A Dataset of Radiographic Image of Seam Weld for Defect Detection

Xuefeng Zhao, Juntao Wu · 2025

Journal of Nondestructive Evaluatio...

Advanced approaches to decoupled sensory signal monitoring in human interface systems

Se Gi Lee, Ki Jun Yu · 2025

International Journal of Extreme Ma...

Deep reinforcement learning based performance optimization of hybrid system for base-isolated structure and shape memory alloy-inerter

Chunxiang Li, Jie Zhao · 2025

Engineering Structures

Synchronous control of multiple hydraulic cylinders in aerial building machine using improved deep reinforcement learning

Limao Zhang, Jiaqi Wang · 2025

Automation in Construction

A Review of DEtection TRansformer: From Basic Architecture to Advanced Developments and Visual Perception Applications

Liang Yu, Lin Tang · 2025

Sensors

HRYNet: A Highly Robust YOLO Network for Complex Road Traffic Object Detection

Lindong Tang, Lijun Yun · 2024

Sensors

Prospective Role of Foundation Models in Advancing Autonomous Vehicles

JIANHUA WU, Bingzhao Gao · 2024

Research

Majority voting of doctors improves appropriateness of AI reliance in pathology

Hongyan Gu, Chunxu Yang · 2024

International Journal of Human-Comp...

Reinforcement learning path planning method incorporating multi-step Hindsight Experience Replay for lightweight robots

Jiaqi Wang, Huiyan Han · 2024

Displays

Metrics

1,443

Citations

175

References

Details

Published: Nov 14, 2019
Vol/Issue: 37(3)
Pages: 362-386
License: View

Authors

S

Sorin Grigorescu

Artificial Intelligence, Elektrobit Automotive Robotics, Vision and Control Laboratory, Transilvania University of Brasov Brasov Romania

B

Bogdan Trasnea

Artificial Intelligence, Elektrobit Automotive Robotics, Vision and Control Laboratory, Transilvania University of Brasov Brasov Romania

T

Tiberiu Cocias

Artificial Intelligence, Elektrobit Automotive Robotics, Vision and Control Laboratory, Transilvania University of Brasov Brasov Romania

G

Gigel Macesanu

Artificial Intelligence, Elektrobit Automotive Robotics, Vision and Control Laboratory, Transilvania University of Brasov Brasov Romania

Cite This Article

Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, et al. (2019). A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3), 362-386. https://doi.org/10.1002/rob.21918

A survey of deep learning techniques for autonomous driving

You May Also Like