Efficient activity recognition using lightweight CNN and DS-GRU network for surveillance applications

Amin Ullah, Khan Muhammad, Weiping Ding, Vasile Palade, Ijaz Ul Haq, Sung Wook Baik

Research output: Contribution to journalArticlepeer-review


Recognizing human activities has become a trend in smart surveillance that contains several challenges, such as performing effective analyses of huge video data streams, while maintaining low computational complexity, and performing this task in real-time. Current activity recognition techniques are using convolutional neural network (CNN) models with computationally complex classifiers, creating hurdles in obtaining quick responses for abnormal activities. To address these challenges in real-time surveillance, this paper proposes a lightweight deep learning-assisted framework for activity recognition. First, we detect a human in the surveillance stream using an effective CNN model, which is trained on two surveillance datasets. The detected individual is tracked throughout the video stream via an ultra-fast object tracker called the ‘minimum output sum of squared error’ (MOSSE). Next, for each tracked individual, pyramidal convolutional features are extracted from two consecutive frames using the efficient LiteFlowNet CNN. Finally, a novel deep skip connection gated recurrent unit (DS-GRU) is trained to learn the temporal changes in the sequence of frames for activity recognition. Experiments are conducted over five benchmark activity recognition datasets, and the results indicate the efficiency of the proposed technique for real-time surveillance applications compared to the state-of-the-art.

Original languageEnglish
Article number107102
JournalApplied Soft Computing
Early online date15 Jan 2021
Publication statusE-pub ahead of print - 15 Jan 2021

Bibliographical note

Funding Information:
This work was supported by the National Research Foundation of Korea and funded by the Korean government (MSIP) under Grant No. 2019R1A2B5B01070067 .

Publisher Copyright:
© 2021 The Author(s)

Copyright 2021 Elsevier B.V., All rights reserved.


  • Activity recognition
  • Artificial intelligence
  • Deep learning
  • IoT
  • Machine learning
  • Pattern recognition
  • Video big data analytics

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'Efficient activity recognition using lightweight CNN and DS-GRU network for surveillance applications'. Together they form a unique fingerprint.

Cite this