Real-Time Surveillance System for Analyzing Abnormal Behavior of Pedestrians

Dohun Kim; Heegwang Kim; Yeongheon Mok; Joonki Paik

doi:10.3390/app11136153

journal article Open Access Jul 02, 2021

Real-Time Surveillance System for Analyzing Abnormal Behavior of Pedestrians

Dohun Kim

Heegwang Kim Yeongheon Mok Joonki Paik

Applied Sciences Vol. 11 No. 13 pp. 6153 · MDPI AG

View at Publisher Save 10.3390/app11136153

Abstract

In spite of excellent performance of deep learning-based computer vision algorithms, they are not suitable for real-time surveillance to detect abnormal behavior because of very high computational complexity. In this paper, we propose a real-time surveillance system for abnormal behavior analysis in a closed-circuit television (CCTV) environment by constructing an algorithm and system optimized for a CCTV environment. The proposed method combines pedestrian detection and tracking to extract pedestrian information in real-time, and detects abnormal behaviors such as intrusion, loitering, fall-down, and violence. To analyze an abnormal behavior, it first determines intrusion/loitering through the coordinates of an object and then determines fall-down/violence based on the behavior pattern of the object. The performance of the proposed method is evaluated using an intelligent CCTV data set distributed by Korea Internet and Security Agency (KISA).

Topics

No keywords indexed for this article. Browse by subject →

References

37

[1]

Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv.

[2]

Henriques "High-speed tracking with kernelized correlation filters" IEEE Trans. Pattern Anal. Mach. Intell. (2014) 10.1109/tpami.2014.2345390

[3]

Hara, K., Kataoka, H., and Satoh, Y. (2018, January 18–23). Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA. 10.1109/cvpr.2018.00685

[4]

(2021, May 10). KISA Dataset. Available online: https://www.kisa.or.kr.

[5]

Amraee "Abnormal event detection in crowded scenes using one-class SVM" Signal Image Video Process. (2018) 10.1007/s11760-018-1267-z

[6]

Kalaivani, P., Roomi, S.M.M., and Jaishree, B. (2017, January 25–27). Video event representation for abnormal event detection. Proceedings of the 2017 IEEE International Conference on Circuits and Systems (ICCS), San Francisco, CA, USA. 10.1109/iccs1.2017.8326043

[7]

Wang, M., Zhu, Y., Sun, Z., Cao, Z., Xiong, P., Zheng, Y., and Song, S. (2019, January 12–15). Abnormal Behavior Detection of ATM Surveillance Videos Based on Pseudo-3D Residual Network. Proceedings of the 2019 IEEE 4th International Conference on Cloud Computing and Big Data Analysis (ICCCBDA), Chengdu, China. 10.1109/icccbda.2019.8725621

[8]

Simonyan, K., and Zisserman, A. (2014). Two-stream convolutional networks for action recognition in videos. arXiv.

[9]

Learning Spatiotemporal Features with 3D Convolutional Networks

Du Tran, Lubomir Bourdev, Rob Fergus et al.

2015 IEEE International Conference on Computer Vis... 10.1109/iccv.2015.510

[10]

Convolutional Two-Stream Network Fusion for Video Action Recognition

Christoph Feichtenhofer, Axel Pinz, Andrew Zisserman

2016 IEEE Conference on Computer Vision and Patter... 10.1109/cvpr.2016.213

[11]

Crasto, N., Weinzaepfel, P., Alahari, K., and Schmid, C. (2019, January 16–20). Mars: Motion-augmented rgb stream for action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. 10.1109/cvpr.2019.00807

[12]

Large-Scale Video Classification with Convolutional Neural Networks

Andrej Karpathy, George Toderici, Sanketh Shetty et al.

2014 IEEE Conference on Computer Vision and Patter... 10.1109/cvpr.2014.223

[13]

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

Joao Carreira, Andrew Zisserman

2017 IEEE Conference on Computer Vision and Patter... 10.1109/cvpr.2017.502

[14]

Wang, L., Koniusz, P., and Huynh, D.Q. (2019, January 27–28). Hallucinating idt descriptors and i3d optical flow features for action recognition with cnns. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea. 10.1109/iccv.2019.00879

[15]

Choutas, V., Weinzaepfel, P., Revaud, J., and Schmid, C. (2018, January 18–23). Potion: Pose motion representation for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA. 10.1109/cvpr.2018.00734

[16]

Zach, C., Pock, T., and Bischof, H. (2007). A duality based approach for realtime tv-l 1 optical flow. Joint Pattern Recognition Symposium, Springer. 10.1007/978-3-540-74936-3_22

[17]

Yan, A., Wang, Y., Li, Z., and Qiao, Y. (2019, January 16–20). PA3D: Pose-action 3D machine for video recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. 10.1109/cvpr.2019.00811

[18]

Kaddah "Securing smart cities using artificial intelligence: Intrusion and abnormal behavior detection system" Pattern Recognition and Tracking XXXII (2021)

[19]

Ji, H., Zeng, X., Li, H., Ding, W., Nie, X., Zhang, Y., and Xiao, Z. (2020, January 10–12). Human abnormal behavior detection method based on T-TINY-YOLO. Proceedings of the 5th International Conference on Multimedia and Image Processing, Nanjing, China. 10.1145/3381271.3381273

[20]

Hu "A weakly supervised framework for abnormal behavior detection and localization in crowded scenes" Neurocomputing (2020) 10.1016/j.neucom.2019.11.087

[21]

Mojarad, R., Attal, F., Chibani, A., and Amirat, Y. (2020, January 19–24). A hybrid context-aware framework to detect abnormal human daily living behavior. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK. 10.1109/ijcnn48605.2020.9206930

[22]

Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv.

[23]

Microsoft COCO: Common Objects in Context

Tsung-Yi Lin, Michael Maire, Serge Belongie et al.

Lecture Notes in Computer Science 10.1007/978-3-319-10602-1_48

[24]

Kim, D., Kim, H., Shin, J., Mok, Y., and Paik, J. (2019, January 8–11). Real-Time Multiple Pedestrian Tracking Based on Object Identification. Proceedings of the 2019 IEEE 9th International Conference on Consumer Electronics (ICCE-Berlin), Berlin/Heidelberg, Germany. 10.1109/icce-berlin47944.2019.8966205

[25]

Lu, C., Shi, J., and Jia, J. (2013, January 1–8). Abnormal event detection at 150 fps in matlab. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia. 10.1109/iccv.2013.338

[26]

Wang, J., and Cherian, A. (2019, January 27–28). Gods: Generalized one-class discriminative subspaces for anomaly detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea. 10.1109/iccv.2019.00829

[27]

Sultani, W., Chen, C., and Shah, M. (2018, January 18–23). Real-world anomaly detection in surveillance videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA. 10.1109/cvpr.2018.00678

[28]

Dubey, S., Boragule, A., and Jeon, M. (2019, January 24–27). 3D ResNet with Ranking Loss Function for Abnormal Activity Detection in Videos. Proceedings of the 2019 International Conference on Control, Automation and Information Sciences (ICCAIS), Chengdu, China. 10.1109/iccais46528.2019.9074586

[29]

Zhang, J., Qing, L., and Miao, J. (2019, January 22–25). Temporal convolutional network with complementary inner bag loss for weakly supervised anomaly detection. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan. 10.1109/icip.2019.8803657

[30]

Kamoona, A.M., Gosta, A.K., Bab-Hadiashar, A., and Hoseinnezhad, R. (2020). Multiple Instance-Based Video Anomaly Detection using Deep Temporal Encoding-Decoding. arXiv.

[31]

Zhong, J.X., Li, N., Kong, W., Liu, S., Li, T.H., and Li, G. (2019, January 16–20). Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. 10.1109/cvpr.2019.00133

[32]

Feng, J.C., Hong, F.T., and Zheng, W.S. (2021, January 21–24). Mist: Multiple instance self-training framework for video anomaly detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA. 10.1109/cvpr46437.2021.01379

[33]

Wu, P., Liu, J., Shi, Y., Sun, Y., Shao, F., Wu, Z., and Yang, Z. (2020). Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision. European Conference on Computer Vision, Springer. 10.1007/978-3-030-58577-8_20

[34]

Tian, Y., Pang, G., Chen, Y., Singh, R., Verjans, J.W., and Carneiro, G. (2021). Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning. arXiv. 10.1109/iccv48922.2021.00493

[35]

Schuldt, C., Laptev, I., and Caputo, B. (2004, January 23–26). Recognizing human actions: A local SVM approach. Proceedings of the 17th International Conference on Pattern Recognition, ICPR 2004, Cambridge, UK. 10.1109/icpr.2004.1334462

[36]

Liu, J., Luo, J., and Shah, M. (2009, January 19–25). Recognizing realistic actions from videos “in the wild”. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA. 10.1109/cvpr.2009.5206744

[37]

(2021, May 10). VLC Player. Available online: https://www.videolan.org.

Metrics

27

Citations

37

References

Details

Published: Jul 02, 2021
Vol/Issue: 11(13)
Pages: 6153
License: View

Authors

D

Dohun Kim

Department of Image, Graduate School of Advanced Imaging Science, Multimedia and Film Chung-Ang University, Seoul 06974, Korea

H

Heegwang Kim

Department of Image, Graduate School of Advanced Imaging Science, Multimedia and Film Chung-Ang University, Seoul 06974, Korea

Y

Yeongheon Mok

Department of Image, Graduate School of Advanced Imaging Science, Multimedia and Film Chung-Ang University, Seoul 06974, Korea

J

Joonki Paik

Department of Image, Graduate School of Advanced Imaging Science, Multimedia and Film Chung-Ang University, Seoul 06974, Korea

Cite This Article

Dohun Kim, Heegwang Kim, Yeongheon Mok, et al. (2021). Real-Time Surveillance System for Analyzing Abnormal Behavior of Pedestrians. Applied Sciences, 11(13), 6153. https://doi.org/10.3390/app11136153

Real-Time Surveillance System for Analyzing Abnormal Behavior of Pedestrians

You May Also Like