Self-Optimizing Path Tracking Controller for Intelligent Vehicles Based on Reinforcement Learning

Jichang Ma; Hui Xie; Kang Song; Hao Liu

doi:10.3390/sym14010031

journal article Open Access Dec 27, 2021

Self-Optimizing Path Tracking Controller for Intelligent Vehicles Based on Reinforcement Learning

Jichang Ma

Hui Xie

Kang Song

Hao Liu

Symmetry Vol. 14 No. 1 pp. 31 · MDPI AG

View at Publisher Save 10.3390/sym14010031

Abstract

The path tracking control system is a crucial component for autonomous vehicles; it is challenging to realize accurate tracking control when approaching a wide range of uncertain situations and dynamic environments, particularly when such control must perform as well as, or better than, human drivers. While many methods provide state-of-the-art tracking performance, they tend to emphasize constant PID control parameters, calibrated by human experience, to improve tracking accuracy. A detailed analysis shows that PID controllers inefficiently reduce the lateral error under various conditions, such as complex trajectories and variable speed. In addition, intelligent driving vehicles are highly non-linear objects, and high-fidelity models are unavailable in most autonomous systems. As for the model-based controller (MPC or LQR), the complex modeling process may increase the computational burden. With that in mind, a self-optimizing, path tracking controller structure, based on reinforcement learning, is proposed. For the lateral control of the vehicle, a steering method based on the fusion of the reinforcement learning and traditional PID controllers is designed to adapt to various tracking scenarios. According to the pre-defined path geometry and the real-time status of the vehicle, the interactive learning mechanism, based on an RL framework (actor–critic—a symmetric network structure), can realize the online optimization of PID control parameters in order to better deal with the tracking error under complex trajectories and dynamic changes of vehicle model parameters. The adaptive performance of velocity changes was also considered in the tracking process. The proposed controlling approach was tested in different path tracking scenarios, both the driving simulator platforms and on-site vehicle experiments have verified the effects of our proposed self-optimizing controller. The results show that the approach can adaptively change the weights of PID to maintain a tracking error (simulation: within ±0.071 m; realistic vehicle: within ±0.272 m) and steering wheel vibration standard deviations (simulation: within ±0.04°; realistic vehicle: within ±80.69°); additionally, it can adapt to high-speed simulation scenarios (the maximum speed is above 100 km/h and the average speed through curves is 63–76 km/h).

Topics

No keywords indexed for this article. Browse by subject →

References

55

[1]

Visioli, A. (2006). Practical PID Control, Springer.

[2]

Jeffrey, S., Wit, J., Crane, C.D., and Armstrong, D. (2000). Autonomous Ground Vehicle Path Tracking, University of Florida.

[3]

Johary, N.M. (2014). Path Tracking Algorithm for An Autonomous Ground Robot. [Ph.D. Thesis, Universiti Tun Hussein Onn Malaysia].

[4]

Goh, J.Y., Goel, T., and Gerdes, J.C. (2018, January 16–20). A controller for automated drifting along complex trajectories. Proceedings of the 14th International Symposium on Advanced Vehicle Control (AVEC 2018), Beijing, China.

[5]

Goh, J.Y., and Gerdes, J.C. (2017, January 11–14). Simultaneous stabilization and tracking of basic automobile drifting trajectories. Proceedings of the 2016 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA. 10.1109/ivs.2016.7535448

[6]

Hindiyeh "A controller framework for autonomous drifting: Design, stability, and experimental validation" J. Dyn. Syst. Meas. Control. (2014) 10.1115/1.4027471

[7]

Kim "Design of a Path for Collision Avoidance and Path Tracking Scheme for Autonomous Vehicles" IFAC Proc. Vol. (2009) 10.3182/20090902-3-us-2007.0077

[8]

Chen "MPC-based path tracking with PID speed control for high-speed autonomous vehicles considering time-optimal travel" J. Central South Univ. (2020) 10.1007/s11771-020-4561-1

[9]

Wang "Path Tracking Control for Autonomous Vehicles Based on an Improved MPC" IEEE Access (2019) 10.1109/access.2019.2944894

[10]

Kim, D., Kang, J., and Yi, K. (2011, January 5–7). Control strategy for high-speed autonomous driving in structured road. Proceedings of the 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), Washington, DC, USA. 10.1109/itsc.2011.6082856

[11]

Vivek, K., Sheta, M.A., and Gumtapure, V. (2019, January 29–30). A Comparative Study of Stanley, LQR and MPC Controllers for Path Tracking Application (ADAS/AD). Proceedings of the 2019 IEEE International Conference on Intelligent Systems and Green Technology (ICISGT), Visakhapatnam, India.

[12]

Tiep "Design of Fuzzy-PID Controller for Path Tracking of Mobile Robot with Differential Drive" Int. J. Fuzzy Log. Intell. Syst. (2018) 10.5391/ijfis.2018.18.3.220

[13]

Mjahed "Neural Network and Fuzzy-logic-based Self-tuning PID Control for Quadcopter Path Tracking" Stud. Inform. Control (2019) 10.24846/v28i4y201904

[14]

Liang, X., Zhang, W., and Wu, Y. (2017, January 9–10). Automatic Collimation of Optical Path Based on BP-PID Control. Proceedings of the 2017 10th International Conference on Intelligent Computation Technology and Automation (ICICTA), Changsha, China. 10.1109/icicta.2017.81

[15]

Ma, L., Yao, Y., and Wang, M. (2016, January 3–4). The Optimizing Design of Wheeled Robot Tracking System by PID Control Algorithm Based on BP Neural Network. Proceedings of the 2016 International Conference on Industrial Informatics-Computing Technology, Wuhan, China. 10.1109/iciicii.2016.0020

[16]

Abdou "Deep Reinforcement Learning framework for Autonomous Driving" Electron. Imaging (2017)

[17]

Wang, S., Jia, D., and Weng, X. (2018). Deep Reinforcement Learning for Autonomous Driving. arXiv.

[18]

Dong, L., Zhao, D., Zhang, Q., and Chen, Y. (2018). Reinforcement Learning and Deep Learning based Lateral Control for Autonomous Driving. arXiv.

[19]

Wymann, B., Espié, E., Guionneau, C., Dimitrakakis, C., Coulom, R., and Sumner, A. (2019, December 10). TORCS, The Open Racing Car Simulator, v1.3.5. Available online: http://torcs.sourceforge.net/.

[20]

Ingram, A. (2020, June 01). Gran Turismo Sport—Exploring Its Impact on Real-World Racing with Kazunori. Available online: Yamauchi.evo.co.uk.

[21]

Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning

Florian Fuchs, Yunlong Song, Elia Kaufmann et al.

IEEE Robotics and Automation Letters 10.1109/lra.2021.3064284

[22]

Cai "High-Speed Autonomous Drifting With Deep Reinforcement Learning" IEEE Robot. Autom. Lett. (2020) 10.1109/lra.2020.2967299

[23]

Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017). Carla: An open urban driving simulator. arXiv.

[24]

Gao "A Hybrid Tracking Control Strategy for Nonholonomic Wheeled Mobile Robot Incorporating Deep Reinforcement Learning Approach" IEEE Access (2021) 10.1109/access.2021.3053396

[25]

Zhang "Path Following Control for UAV Using Deep Reinforcement Learning Approach" Guid. Navig. Control (2021) 10.1142/s2737480721500059

[26]

Duan "Reinforcement Learning Based Model-free Optimized Trajectory Tracking Strategy Design for an AUV" Neurocomputing (2022) 10.1016/j.neucom.2021.10.056

[27]

Path Planning for UAV Ground Target Tracking via Deep Reinforcement Learning

Bohao Li, Yunjie Wu

IEEE Access 2020 10.1109/access.2020.2971780

[28]

Wang "Trajectory Tracking Control for Mobile Robots Using Reinforcement Learning and PID" Iran. J. Sci. Technol. Trans. Electr. Eng. (2020) 10.1007/s40998-019-00286-4

[29]

Xiao, J., Li, L., Zou, Y., and Zhang, T. (2019). Reinforcement Learning for Robotic Time-optimal Path Tracking Using Prior Knowledge. arXiv.

[30]

Zhang, S., and Wang, W. (March, January 28). Tracking Control for Mobile Robot Based on Deep Reinforcement Learning. Proceedings of the 2019 2nd International Conference on Intelligent Autonomous Systems (ICoIAS), Singapore. 10.1109/icoias.2019.00034

[31]

Arroyo, M.A., and Giraldo, L.F. (2020). Data-driven Outer-Loop Control Using Deep Reinforcement Learning for Trajectory Tracking. arXiv.

[32]

Shan "A Reinforcement Learning-Based Adaptive Path Tracking Approach for Autonomous Driving" IEEE Trans. Veh. Technol. (2020) 10.1109/tvt.2020.3014628

[33]

Puccetti, L., Köpf, F., Rathgeber, C., and Hohmann, S. (2020, January 20–23). Speed Tracking Control using Online Reinforcement Learning in a Real Car. Proceedings of the 6th IEEE International Conference on Control, Automation and Robotics (ICCAR), Singapore. 10.1109/iccar49639.2020.9108051

[34]

Wang, N., Gao, Y., Yang, C., and Zhang, X. (2021). Reinforcement Learning-based Finite-time Tracking Control of an Unknown Unmanned Surface Vehicle with Input Constraints. Neurocomputing, Available online: https://www.sciencedirect.com/science/article/abs/pii/S0925231221015733. 10.1016/j.neucom.2021.04.133

[35]

Jiang, L., Wang, Y., Wang, L., and Wu, J. (2019, January 21–22). Path tracking control based on Deep reinforcement learning in Autonomous driving. Proceedings of the 2019 3rd Conference on Vehicle Control and Intelligence (CVCI), Hefei, China. 10.1109/cvci47823.2019.8951665

[36]

Kamran, D., Zhu, J., and Lauer, M. (2019, January 4–6). Learning Path Tracking for Real Car-like Mobile Robots From Simulation. Proceedings of the 2019 European Conference on Mobile Robots (ECMR), Prague, Czech Republic. 10.1109/ecmr.2019.8870947

[37]

Riedmiller, M., Montemerlo, M., and Dahlkamp, H. (2007, January 11–13). Learning to Drive a Real Car in 20 Minutes. Proceedings of the Frontiers in the Convergence of Bioscience & Information Technologies IEEE Computer Society, Jeju City, Korea. 10.1109/fbit.2007.37

[38]

Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J.-M., Lam, V.-D., Bewley, A., and Shah, A. (2018). Learning to Drive in a Day. arXiv. 10.1109/icra.2019.8793742

[39]

Rajamani, R. (2011). Vehicle Dynamics and Control, Springer Science & Business Media. 10.1007/978-1-4614-1433-9_2

[40]

Kong, J., Pfeiffer, M., Schildbach, G., and Borrelli, F. (July, January 29). Kinematic and dynamic vehicle models for autonomous driving control design. Proceedings of the 2015 IEEE Intelligent Vehicles Symposium (IV), Seoul, Korea. 10.1109/ivs.2015.7225830

[41]

Human-like autonomous car-following model with deep reinforcement learning

Meixin Zhu, Xuesong Wang, Yinhai Wang

Transportation Research Part C: Emerging Technolog... 2018 10.1016/j.trc.2018.10.024

[42]

Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.

[43]

Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv.

[44]

Yu, A., Palefsky-Smith, R., and Bedi, R. (2016). Course Project Reports: Deep Reinforcement Learning for Simulated Autonomous Vehicle Control. Course Proj. Rep. Winter, Available online: http://cs231n.stanford.edu/reports/2016/pdfs/112_Report.pdf.

[45]

Yu, R., Shi, Z., Huang, C., Li, T., and Ma, Q. (2017, January 26–28). Deep reinforcement learning based optimal trajectory tracking control of autonomous underwater vehicle. Proceedings of the 2017 36th Chinese Control Conference (CCC), Dalian, China. 10.23919/chicc.2017.8028138

[46]

Monahan "A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms" Manag. Sci. (1982) 10.1287/mnsc.28.1.1

[47]

Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press.

[48]

OnActor-Critic Algorithms

Vijay R. Konda, John N. Tsitsiklis

SIAM Journal on Control and Optimization 2002 10.1137/s0363012901385691

[49]

Yan, Z., and Zhuang, J. (2020). Active Disturbance Rejection Algorithm Applied to Path Tracking in Autonomous Vehicles. J. Chongqing Univ. Technol. Nat. Sci., 1–10. Available online: http://kns.cnki.net/kcms/detail/50.1205.T.20200522.1459.004.html.

[50]

Chao "Trajectory tracking control of wmrs with lateral and longitudinal slippage based on active disturbance rejection control" Robot. Auton. Syst. (2018) 10.1016/j.robot.2018.06.011

Showing 50 of 55 references

Metrics

23

Citations

55

References

Details

Published: Dec 27, 2021
Vol/Issue: 14(1)
Pages: 31
License: View

Authors

J

Jichang Ma

State Key Laboratory of Engines, Tianjin University, Tianjin 300072, China

H

Hui Xie

State Key Laboratory of Engines, Tianjin University, Tianjin 300072, China

K

Kang Song

State Key Laboratory of Engines, Tianjin University, Tianjin 300072, China

H

Hao Liu

State Key Laboratory of Engines, Tianjin University, Tianjin 300072, China

Funding

This work was supported by the Tianjin Science and Technology Planning Project(2019): Research and Application of Deep Reinforcement Learning Control Algorithm for Intelligent Unmanned System Award: 19ZXZNGX00050

Cite This Article

Jichang Ma, Hui Xie, Kang Song, et al. (2021). Self-Optimizing Path Tracking Controller for Intelligent Vehicles Based on Reinforcement Learning. Symmetry, 14(1), 31. https://doi.org/10.3390/sym14010031

Self-Optimizing Path Tracking Controller for Intelligent Vehicles Based on Reinforcement Learning

You May Also Like