Automatic License Plate Recognition in Real-World Traffic Videos Captured in Unconstrained Environment by a Mobile Camera

Ishtiaq Rasool Khan; Syed Talha Abid Ali; Asif Siddiq; Muhammad Murtaza Khan; Muhammad Usman Ilyas; Saleh Alshomrani; Susanto Rahardja

doi:10.3390/electronics11091408

journal article Open Access Apr 28, 2022

Automatic License Plate Recognition in Real-World Traffic Videos Captured in Unconstrained Environment by a Mobile Camera

Ishtiaq Rasool Khan

Syed Talha Abid Ali Asif Siddiq Muhammad Murtaza Khan Muhammad Usman Ilyas Saleh Alshomrani Susanto Rahardja

Electronics Vol. 11 No. 9 pp. 1408 · MDPI AG

View at Publisher Save 10.3390/electronics11091408

Abstract

Automatic License Plate Recognition (ALPR) has remained an active research topic for years due to various applications, especially in Intelligent Transportation Systems (ITS). This paper presents an efficient ALPR technique based on deep learning, which accurately performs license plate (LP) recognition tasks in an unconstrained environment, even when trained on a limited dataset. We capture real traffic videos in the city and label the LPs and the alphanumeric characters in the LPs within different frames to generate training and testing datasets. Data augmentation techniques are applied to increase the number of training and testing samples. We apply the transfer learning approach to train the recently released YOLOv5 object detecting framework to detect the LPs and the alphanumerics. Next, we train a convolutional neural network (CNN) to recognize the detected alphanumerics. The proposed technique achieved a recognition rate of 92.8% on a challenging proprietary dataset collected in several jurisdictions of Saudi Arabia. This accuracy is higher than what was achieved on the same dataset by commercially available Sighthound (86%), PlateRecognizer (67%), OpenALPR (77%), and a state-of-the-art recent CNN model (82%). The proposed system also outperformed the existing ALPR solutions on several benchmark datasets.

Topics

No keywords indexed for this article. Browse by subject →

References

47

[1]

Wermter, S., Mascioli, F.M.F., Rizzi, A., Majumdar, A., Matei, B., Oliveira-Santos, T., Blumenstein, M., Xu, B., Chen, B., and Badue, C. (2018, January 8–13). A robust real-time automatic license plate recognition based on the YOLO detector. Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil.

[2]

"Automatic vehicle identification by plate recognition" World Acad. Sci. Eng. Technol. (2005)

[3]

Kilic, I., and Aydin, G. (2018, January 28–30). Turkish vehicle license plate recognition using deep learning. Proceedings of the 2018 International Conference on Artificial Intelligence and Data Processing (IDAP), Malatya, Turkey. 10.1109/idap.2018.8620744

[4]

Roy, A., and Ghoshal, D.P. (2011, January 4–5). Number Plate Recognition for use in different countries using an improved segmentation. Proceedings of the 2011 2nd National Conference on Emerging Trends and Applications in Computer Science, Shillong, India. 10.1109/ncetacs.2011.5751407

[5]

Xiang "Lightweight fully convolutional network for license plate detection" Optik (2019) 10.1016/j.ijleo.2018.10.098

[6]

Chen "Automatic license-plate location and recognition based on feature salience" IEEE Trans. Veh. Technol. (2009) 10.1109/tvt.2009.2013139

[7]

Bulan "Segmentation-and annotation-free license plate recognition with deep localization and failure identification" IEEE Trans. Intell. Transp. Syst. (2017) 10.1109/tits.2016.2639020

[8]

Safaei "Real-time search-free multiple license plate recognition via likelihood estimation of saliency" Comput. Electr. Eng. (2016) 10.1016/j.compeleceng.2016.09.010

[9]

Zhai, X., Benssali, F., and Ramalingam, S. (2010, January 7–10). License plate localisation based on morphological operations. Proceedings of the 2010 11th International Conference on Control Automation Robotics & Vision, Singapore. 10.1109/icarcv.2010.5707933

[10]

Wang, S.-Z., and Lee, H.-J. (2003, January 12–15). Detection and recognition of license plate characters with different appearances. Proceedings of the 2003 IEEE International Conference on Intelligent Transportation Systems, Shanghai, China.

[11]

Azam "Automatic license plate detection in hazardous condition" J. Vis. Commun. Image Represent. (2016) 10.1016/j.jvcir.2016.01.015

[12]

Ansari "License Number Plate Recognition using Template Matching" Int. J. Comput. Trends Technol. (2016) 10.14445/22312803/ijctt-v35p133

[13]

Montazzolli, S., and Jung, C. (2017, January 17–20). Real-time brazilian license plate detection and recognition using deep convolutional neural networks. Proceedings of the 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Niteroi, Brazil. 10.1109/sibgrapi.2017.14

[14]

Marzuki "A design of license plate recognition system using convolutional neural network" Int. J. Electr. Comput. Eng. (2019)

[15]

Gou "Vehicle license plate recognition based on extremal regions and restricted Boltzmann machines" IEEE Trans. Intell. Transp. Syst. (2015) 10.1109/tits.2015.2496545

[16]

ImageNet classification with deep convolutional neural networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton

Communications of the ACM 2017 10.1145/3065386

[17]

A survey of the recent architectures of deep convolutional neural networks

Asifullah Khan, Anabia Sohail, Umme Zahoora et al.

Artificial Intelligence Review 2020 10.1007/s10462-020-09825-6

[18]

Laroca, R., Zanlorensi, L.A., Gonçalves, G.R., Todt, E., Schwartz, W.R., and Menotti, D. (2019). An efficient and layout-independent automatic license plate recognition system based on the YOLO detector. arXiv. 10.1109/ijcnn.2018.8489629

[19]

Yonetsu, S., Iwamoto, Y., and Chen, Y.W. (2019, January 11–13). Two-stage YOLOv2 for accurate license-plate detection in complex scenes. Proceedings of the 2019 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA. 10.1109/icce.2019.8661944

[20]

Liu, Z., Wang, Z., and Xing, Y. (2019, January 16–18). Wagon number recognition based on the YOLOv3 detector. Proceedings of the 2019 IEEE 2nd International Conference on Computer and Communication Engineering Technology (CCET), Beijing, China. 10.1109/ccet48361.2019.8989186

[21]

Jain, V., Sasindran, Z., Rajagopal, A., Biswas, S., Bharadwaj, H.S., and Ramakrishnan, K. (2016, January 18–22). Deep automatic license plate recognition system. Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing, Guwahati, India. 10.1145/3009977.3010052

[22]

Wang, Q. (2017, January 20–22). License plate recognition via convolutional neural networks. Proceedings of the 2017 8th IEEE International Conference on Software Engineering and Service Science (ICSESS), Beijing, China. 10.1109/icsess.2017.8343061

[23]

Zang "Vehicle license plate recognition using visual attention model and deep learning" J. Electron. Imaging (2015) 10.1117/1.jei.24.3.033001

[24]

Izidio "An embedded automatic license plate recognition system using deep learning" Des. Autom. Embed. Syst. (2020) 10.1007/s10617-019-09230-5

[25]

Li, H., and Shen, C. (2016). Reading car license plates using deep convolutional neural networks and LSTMs. arXiv.

[26]

Špaňhel, J., Sochor, J., Juránek, R., and Herout, A. (2018, January 4–7). Geometric alignment by deep learning for recognition of challenging license plates. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA. 10.1109/itsc.2018.8569259

[27]

Špaňhel, J., Sochor, J., Juránek, R., Herout, A., Maršík, L., and Zemčík, P. (September, January 29). Holistic recognition of low quality license plates by CNN using track annotated data. Proceedings of the 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Lecce, Italy. 10.1109/avss.2017.8078501

[28]

Gonçalves, G.R., Diniz, M.A., Laroca, R., Menotti, D., and Schwartz, W.R. (November, January 29). Real-time automatic license plate recognition through deep multi-task networks. Proceedings of the 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Parana, Brazil.

[29]

Silva, S.M., and Jung, C.R. (2018, January 8–14). License plate detection and recognition in unconstrained scenarios. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany. 10.1007/978-3-030-01258-8_36

[30]

Li "Toward end-to-end car license plate detection and recognition with deep neural networks" IEEE Trans. Intell. Transp. Syst. (2018) 10.1109/tits.2018.2847291

[31]

Masood, S.Z., Shu, G., Dehghan, A., and Ortiz, E.G. (2017). License plate detection and recognition using deeply learned convolutional neural networks. arXiv.

[32]

Zhuang, J., Hou, S., Wang, Z., and Zha, Z.-J. (2018, January 23–28). Towards human-level license plate recognition. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK. 10.1007/978-3-030-01219-9_19

[33]

Henry "Multinational license plate recognition using generalized character sequence detection" IEEE Access (2020) 10.1109/access.2020.2974973

[34]

Steve (2022, February 17). Types of License Plates in Saudi Arabia. Available online: https://lifeinsaudiarabia.net/types-of-number-plates-in-saudi-arabia.

[35]

Lin, T. (2022, February 17). LabelImg, Annotation Tool. Available online: https://github.com/tzutalin/labelImg.

[36]

Takimoglu, A. (2022, February 17). Data Augmentation. Available online: https://research.aimultiple.com/data-augmentation.

[37]

You Only Look Once: Unified, Real-Time Object Detection

Joseph Redmon, Santosh Divvala, Ross Girshick et al.

2016 IEEE Conference on Computer Vision and Patter... 10.1109/cvpr.2016.91

[38]

Jocher, G. (2022, February 17). YOLOV5, An Open-Source Detection Model Repository. Available online: https://github.com/ultralytics/yolov5.

[39]

(2022, February 17). ProgrammerSought, YOLOV5 Learning Analysis. Available online: https://www.programmersought.com/article/16818671989/.

[40]

Tan, M., Pang, R., and Le, Q.V. (2020, January 14–19). Efficientdet: Scalable and efficient object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA. 10.1109/cvpr42600.2020.01079

[41]

Wang, C.-Y., Liao, H.-Y.M., Wu, Y.-H., Chen, P.-Y., Hsieh, J.-W., and Yeh, I.-H. (2021, January 19–25). CSPNet: A new backbone that can enhance learning capability of CNN. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Nashville, TN, USA. 10.1109/cvprw50498.2020.00203

[42]

Jiang "Research on early fire detection of Yolo V5 based on multiple transfer learning" Fire Sci. Technol. (2021)

[43]

(2022, February 17). Caltech Vision Lab, Datasets. Available online: https://www.vision.caltech.edu/datasets/.

[44]

(2022, February 17). Media Lab, LPRdatabase. Available online: http://www.medialab.ntua.gr/research/LPRdatabase.

[45]

OpenALPR (2022, February 17). Commercially Available Tool. Available online: https://github.com/openalpr/openalpr/.

[46]

SightHound (2022, February 17). Commercially Available Tool. Available online: https://www.sighthound.com/.

[47]

PlateRecognizer (2022, February 17). Commercially Available Tool. Available online: https://platerecognizer.com/.

Cited By

27

Enabling smart parking for smart cities using Internet of Things (IoT) and machine learning

Mofadal Alymani, Lenah Abdulaziz Almoqhem · 2025

PeerJ Computer Science

Metrics

27

Citations

47

References

Details

Published: Apr 28, 2022
Vol/Issue: 11(9)
Pages: 1408
License: View

Authors

I

Ishtiaq Rasool Khan

Department of Computer Science and Artificial Intelligence, College of Computer Science and Engineering, University of Jeddah, Jeddah 21589, Saudi Arabia

S

Syed Talha Abid Ali

Pakistan Institute of Engineering & Technology, Department of Electrical Engineering, Multan 61000, Pakistan

A

Asif Siddiq

Pakistan Institute of Engineering & Technology, Department of Electrical Engineering, Multan 61000, Pakistan

M

Muhammad Murtaza Khan

Department of Computer Science and Artificial Intelligence, College of Computer Science and Engineering, University of Jeddah, Jeddah 21589, Saudi Arabia

M

Muhammad Usman Ilyas

Department of Computer Science and Artificial Intelligence, College of Computer Science and Engineering, University of Jeddah, Jeddah 21589, Saudi Arabia

S

Saleh Alshomrani

Department of Computer Science and Artificial Intelligence, College of Computer Science and Engineering, University of Jeddah, Jeddah 21589, Saudi Arabia

S

Susanto Rahardja

School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an 710072, China

Funding

Deanship of Scientific Research (DSR), University of Jeddah Award: UJ-03-18-ICP

Cite This Article

Ishtiaq Rasool Khan, Syed Talha Abid Ali, Asif Siddiq, et al. (2022). Automatic License Plate Recognition in Real-World Traffic Videos Captured in Unconstrained Environment by a Mobile Camera. Electronics, 11(9), 1408. https://doi.org/10.3390/electronics11091408

Automatic License Plate Recognition in Real-World Traffic Videos Captured in Unconstrained Environment by a Mobile Camera

You May Also Like