journal article Open Access Apr 28, 2022

Automatic License Plate Recognition in Real-World Traffic Videos Captured in Unconstrained Environment by a Mobile Camera

Electronics Vol. 11 No. 9 pp. 1408 · MDPI AG
View at Publisher Save 10.3390/electronics11091408
Abstract
Automatic License Plate Recognition (ALPR) has remained an active research topic for years due to various applications, especially in Intelligent Transportation Systems (ITS). This paper presents an efficient ALPR technique based on deep learning, which accurately performs license plate (LP) recognition tasks in an unconstrained environment, even when trained on a limited dataset. We capture real traffic videos in the city and label the LPs and the alphanumeric characters in the LPs within different frames to generate training and testing datasets. Data augmentation techniques are applied to increase the number of training and testing samples. We apply the transfer learning approach to train the recently released YOLOv5 object detecting framework to detect the LPs and the alphanumerics. Next, we train a convolutional neural network (CNN) to recognize the detected alphanumerics. The proposed technique achieved a recognition rate of 92.8% on a challenging proprietary dataset collected in several jurisdictions of Saudi Arabia. This accuracy is higher than what was achieved on the same dataset by commercially available Sighthound (86%), PlateRecognizer (67%), OpenALPR (77%), and a state-of-the-art recent CNN model (82%). The proposed system also outperformed the existing ALPR solutions on several benchmark datasets.
Topics

No keywords indexed for this article. Browse by subject →

References
47
[1]
Wermter, S., Mascioli, F.M.F., Rizzi, A., Majumdar, A., Matei, B., Oliveira-Santos, T., Blumenstein, M., Xu, B., Chen, B., and Badue, C. (2018, January 8–13). A robust real-time automatic license plate recognition based on the YOLO detector. Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil.
[2]
"Automatic vehicle identification by plate recognition" World Acad. Sci. Eng. Technol. (2005)
[3]
Kilic, I., and Aydin, G. (2018, January 28–30). Turkish vehicle license plate recognition using deep learning. Proceedings of the 2018 International Conference on Artificial Intelligence and Data Processing (IDAP), Malatya, Turkey. 10.1109/idap.2018.8620744
[4]
Roy, A., and Ghoshal, D.P. (2011, January 4–5). Number Plate Recognition for use in different countries using an improved segmentation. Proceedings of the 2011 2nd National Conference on Emerging Trends and Applications in Computer Science, Shillong, India. 10.1109/ncetacs.2011.5751407
[5]
Xiang "Lightweight fully convolutional network for license plate detection" Optik (2019) 10.1016/j.ijleo.2018.10.098
[6]
Chen "Automatic license-plate location and recognition based on feature salience" IEEE Trans. Veh. Technol. (2009) 10.1109/tvt.2009.2013139
[7]
Bulan "Segmentation-and annotation-free license plate recognition with deep localization and failure identification" IEEE Trans. Intell. Transp. Syst. (2017) 10.1109/tits.2016.2639020
[8]
Safaei "Real-time search-free multiple license plate recognition via likelihood estimation of saliency" Comput. Electr. Eng. (2016) 10.1016/j.compeleceng.2016.09.010
[9]
Zhai, X., Benssali, F., and Ramalingam, S. (2010, January 7–10). License plate localisation based on morphological operations. Proceedings of the 2010 11th International Conference on Control Automation Robotics & Vision, Singapore. 10.1109/icarcv.2010.5707933
[10]
Wang, S.-Z., and Lee, H.-J. (2003, January 12–15). Detection and recognition of license plate characters with different appearances. Proceedings of the 2003 IEEE International Conference on Intelligent Transportation Systems, Shanghai, China.
[11]
Azam "Automatic license plate detection in hazardous condition" J. Vis. Commun. Image Represent. (2016) 10.1016/j.jvcir.2016.01.015
[12]
Ansari "License Number Plate Recognition using Template Matching" Int. J. Comput. Trends Technol. (2016) 10.14445/22312803/ijctt-v35p133
[13]
Montazzolli, S., and Jung, C. (2017, January 17–20). Real-time brazilian license plate detection and recognition using deep convolutional neural networks. Proceedings of the 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Niteroi, Brazil. 10.1109/sibgrapi.2017.14
[14]
Marzuki "A design of license plate recognition system using convolutional neural network" Int. J. Electr. Comput. Eng. (2019)
[15]
Gou "Vehicle license plate recognition based on extremal regions and restricted Boltzmann machines" IEEE Trans. Intell. Transp. Syst. (2015) 10.1109/tits.2015.2496545
[16]
ImageNet classification with deep convolutional neural networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton

Communications of the ACM 2017 10.1145/3065386
[17]
A survey of the recent architectures of deep convolutional neural networks

Asifullah Khan, Anabia Sohail, Umme Zahoora et al.

Artificial Intelligence Review 2020 10.1007/s10462-020-09825-6
[18]
Laroca, R., Zanlorensi, L.A., Gonçalves, G.R., Todt, E., Schwartz, W.R., and Menotti, D. (2019). An efficient and layout-independent automatic license plate recognition system based on the YOLO detector. arXiv. 10.1109/ijcnn.2018.8489629
[19]
Yonetsu, S., Iwamoto, Y., and Chen, Y.W. (2019, January 11–13). Two-stage YOLOv2 for accurate license-plate detection in complex scenes. Proceedings of the 2019 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA. 10.1109/icce.2019.8661944
[20]
Liu, Z., Wang, Z., and Xing, Y. (2019, January 16–18). Wagon number recognition based on the YOLOv3 detector. Proceedings of the 2019 IEEE 2nd International Conference on Computer and Communication Engineering Technology (CCET), Beijing, China. 10.1109/ccet48361.2019.8989186
[21]
Jain, V., Sasindran, Z., Rajagopal, A., Biswas, S., Bharadwaj, H.S., and Ramakrishnan, K. (2016, January 18–22). Deep automatic license plate recognition system. Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing, Guwahati, India. 10.1145/3009977.3010052
[22]
Wang, Q. (2017, January 20–22). License plate recognition via convolutional neural networks. Proceedings of the 2017 8th IEEE International Conference on Software Engineering and Service Science (ICSESS), Beijing, China. 10.1109/icsess.2017.8343061
[23]
Zang "Vehicle license plate recognition using visual attention model and deep learning" J. Electron. Imaging (2015) 10.1117/1.jei.24.3.033001
[24]
Izidio "An embedded automatic license plate recognition system using deep learning" Des. Autom. Embed. Syst. (2020) 10.1007/s10617-019-09230-5
[25]
Li, H., and Shen, C. (2016). Reading car license plates using deep convolutional neural networks and LSTMs. arXiv.
[26]
Špaňhel, J., Sochor, J., Juránek, R., and Herout, A. (2018, January 4–7). Geometric alignment by deep learning for recognition of challenging license plates. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA. 10.1109/itsc.2018.8569259
[27]
Špaňhel, J., Sochor, J., Juránek, R., Herout, A., Maršík, L., and Zemčík, P. (September, January 29). Holistic recognition of low quality license plates by CNN using track annotated data. Proceedings of the 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Lecce, Italy. 10.1109/avss.2017.8078501
[28]
Gonçalves, G.R., Diniz, M.A., Laroca, R., Menotti, D., and Schwartz, W.R. (November, January 29). Real-time automatic license plate recognition through deep multi-task networks. Proceedings of the 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Parana, Brazil.
[29]
Silva, S.M., and Jung, C.R. (2018, January 8–14). License plate detection and recognition in unconstrained scenarios. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany. 10.1007/978-3-030-01258-8_36
[30]
Li "Toward end-to-end car license plate detection and recognition with deep neural networks" IEEE Trans. Intell. Transp. Syst. (2018) 10.1109/tits.2018.2847291
[31]
Masood, S.Z., Shu, G., Dehghan, A., and Ortiz, E.G. (2017). License plate detection and recognition using deeply learned convolutional neural networks. arXiv.
[32]
Zhuang, J., Hou, S., Wang, Z., and Zha, Z.-J. (2018, January 23–28). Towards human-level license plate recognition. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK. 10.1007/978-3-030-01219-9_19
[33]
Henry "Multinational license plate recognition using generalized character sequence detection" IEEE Access (2020) 10.1109/access.2020.2974973
[34]
Steve (2022, February 17). Types of License Plates in Saudi Arabia. Available online: https://lifeinsaudiarabia.net/types-of-number-plates-in-saudi-arabia.
[35]
Lin, T. (2022, February 17). LabelImg, Annotation Tool. Available online: https://github.com/tzutalin/labelImg.
[36]
Takimoglu, A. (2022, February 17). Data Augmentation. Available online: https://research.aimultiple.com/data-augmentation.
[37]
You Only Look Once: Unified, Real-Time Object Detection

Joseph Redmon, Santosh Divvala, Ross Girshick et al.

2016 IEEE Conference on Computer Vision and Patter... 10.1109/cvpr.2016.91
[38]
Jocher, G. (2022, February 17). YOLOV5, An Open-Source Detection Model Repository. Available online: https://github.com/ultralytics/yolov5.
[39]
(2022, February 17). ProgrammerSought, YOLOV5 Learning Analysis. Available online: https://www.programmersought.com/article/16818671989/.
[40]
Tan, M., Pang, R., and Le, Q.V. (2020, January 14–19). Efficientdet: Scalable and efficient object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA. 10.1109/cvpr42600.2020.01079
[41]
Wang, C.-Y., Liao, H.-Y.M., Wu, Y.-H., Chen, P.-Y., Hsieh, J.-W., and Yeh, I.-H. (2021, January 19–25). CSPNet: A new backbone that can enhance learning capability of CNN. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Nashville, TN, USA. 10.1109/cvprw50498.2020.00203
[42]
Jiang "Research on early fire detection of Yolo V5 based on multiple transfer learning" Fire Sci. Technol. (2021)
[43]
(2022, February 17). Caltech Vision Lab, Datasets. Available online: https://www.vision.caltech.edu/datasets/.
[44]
(2022, February 17). Media Lab, LPRdatabase. Available online: http://www.medialab.ntua.gr/research/LPRdatabase.
[45]
OpenALPR (2022, February 17). Commercially Available Tool. Available online: https://github.com/openalpr/openalpr/.
[46]
SightHound (2022, February 17). Commercially Available Tool. Available online: https://www.sighthound.com/.
[47]
PlateRecognizer (2022, February 17). Commercially Available Tool. Available online: https://platerecognizer.com/.
Cited By
27
PeerJ Computer Science
Metrics
27
Citations
47
References
Details
Published
Apr 28, 2022
Vol/Issue
11(9)
Pages
1408
License
View
Funding
Deanship of Scientific Research (DSR), University of Jeddah Award: UJ-03-18-ICP
Cite This Article
Ishtiaq Rasool Khan, Syed Talha Abid Ali, Asif Siddiq, et al. (2022). Automatic License Plate Recognition in Real-World Traffic Videos Captured in Unconstrained Environment by a Mobile Camera. Electronics, 11(9), 1408. https://doi.org/10.3390/electronics11091408
Related

You May Also Like

Machine Learning Interpretability: A Survey on Methods and Metrics

Diogo V. Carvalho, Eduardo M. Pereira · 2019

1,384 citations

The k-means Algorithm: A Comprehensive Survey and Performance Evaluation

Mohiuddin Ahmed, Raihan Seraj · 2020

1,342 citations

Sentiment Analysis Based on Deep Learning: A Comparative Study

Nhan Cach Dang, María N. Moreno-García · 2020

550 citations