Hierarchical transfer learning for online recognition of compound actions

Victoria Bloom, V. Argyriou, D. Makris

Research output: Contribution to journalArticle

17 Citations (Scopus)
46 Downloads (Pure)

Abstract

Recognising human actions in real-time can provide users with a natural user interface (NUI) enabling a range of innovative and immersive applications. A NUI application should not restrict users’ movements; it should allow users to transition between actions in quick succession, which we term as compound actions. However, the majority of action recognition researchers have focused on individual actions, so their approaches are limited to recognising single actions or multiple actions that are temporally separated. This paper proposes a novel online action recognition method for fast detection of compound actions. A key contribution is our hierarchical body model that can be automatically configured to detect actions based on the low level body parts that are the most discriminative for a particular action. Another key contribution is a transfer learning strategy to allow the tasks of action segmentation and whole body modelling to be performed on a related but simpler dataset, combined with automatic hierarchical body model adaption on a more complex target dataset. Experimental results on a challenging and realistic dataset show an improvement in action recognition performance of 16% due to the introduction of our hierarchical transfer learning. The proposed algorithm is fast with an average latency of just 2 frames (66 ms) and outperforms state of the art action recognition algorithms that are capable of fast online action recognition.
Original languageEnglish
Pages (from-to)62-72
JournalComputer Vision and Image Understanding
Volume144
Early online date11 Dec 2015
DOIs
Publication statusPublished - Mar 2016

Fingerprint

User interfaces

Bibliographical note

NOTICE: this is the author’s version of a work that was accepted for publication in Computer Vision and Image Understanding. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Computer Vision and Image Understanding, [VOL 144, (2015)] DOI: 10.1016/j.cviu.2015.12.001

© 2015, Elsevier. Licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/

Keywords

  • Online action recognition
  • Online interaction recognition
  • Hierarchical
  • Transfer learning

Cite this

Hierarchical transfer learning for online recognition of compound actions. / Bloom, Victoria; Argyriou, V.; Makris, D.

In: Computer Vision and Image Understanding, Vol. 144, 03.2016, p. 62-72.

Research output: Contribution to journalArticle

Bloom, Victoria ; Argyriou, V. ; Makris, D. / Hierarchical transfer learning for online recognition of compound actions. In: Computer Vision and Image Understanding. 2016 ; Vol. 144. pp. 62-72.
@article{d6606195f2b9436d98ffcccdb7ddcb3e,
title = "Hierarchical transfer learning for online recognition of compound actions",
abstract = "Recognising human actions in real-time can provide users with a natural user interface (NUI) enabling a range of innovative and immersive applications. A NUI application should not restrict users’ movements; it should allow users to transition between actions in quick succession, which we term as compound actions. However, the majority of action recognition researchers have focused on individual actions, so their approaches are limited to recognising single actions or multiple actions that are temporally separated. This paper proposes a novel online action recognition method for fast detection of compound actions. A key contribution is our hierarchical body model that can be automatically configured to detect actions based on the low level body parts that are the most discriminative for a particular action. Another key contribution is a transfer learning strategy to allow the tasks of action segmentation and whole body modelling to be performed on a related but simpler dataset, combined with automatic hierarchical body model adaption on a more complex target dataset. Experimental results on a challenging and realistic dataset show an improvement in action recognition performance of 16{\%} due to the introduction of our hierarchical transfer learning. The proposed algorithm is fast with an average latency of just 2 frames (66 ms) and outperforms state of the art action recognition algorithms that are capable of fast online action recognition.",
keywords = "Online action recognition, Online interaction recognition, Hierarchical, Transfer learning",
author = "Victoria Bloom and V. Argyriou and D. Makris",
note = "NOTICE: this is the author’s version of a work that was accepted for publication in Computer Vision and Image Understanding. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Computer Vision and Image Understanding, [VOL 144, (2015)] DOI: 10.1016/j.cviu.2015.12.001 {\circledC} 2015, Elsevier. Licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/",
year = "2016",
month = "3",
doi = "10.1016/j.cviu.2015.12.001",
language = "English",
volume = "144",
pages = "62--72",
journal = "Computer Vision and Image Understanding",
issn = "1077-3142",
publisher = "Elsevier",

}

TY - JOUR

T1 - Hierarchical transfer learning for online recognition of compound actions

AU - Bloom, Victoria

AU - Argyriou, V.

AU - Makris, D.

N1 - NOTICE: this is the author’s version of a work that was accepted for publication in Computer Vision and Image Understanding. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Computer Vision and Image Understanding, [VOL 144, (2015)] DOI: 10.1016/j.cviu.2015.12.001 © 2015, Elsevier. Licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/

PY - 2016/3

Y1 - 2016/3

N2 - Recognising human actions in real-time can provide users with a natural user interface (NUI) enabling a range of innovative and immersive applications. A NUI application should not restrict users’ movements; it should allow users to transition between actions in quick succession, which we term as compound actions. However, the majority of action recognition researchers have focused on individual actions, so their approaches are limited to recognising single actions or multiple actions that are temporally separated. This paper proposes a novel online action recognition method for fast detection of compound actions. A key contribution is our hierarchical body model that can be automatically configured to detect actions based on the low level body parts that are the most discriminative for a particular action. Another key contribution is a transfer learning strategy to allow the tasks of action segmentation and whole body modelling to be performed on a related but simpler dataset, combined with automatic hierarchical body model adaption on a more complex target dataset. Experimental results on a challenging and realistic dataset show an improvement in action recognition performance of 16% due to the introduction of our hierarchical transfer learning. The proposed algorithm is fast with an average latency of just 2 frames (66 ms) and outperforms state of the art action recognition algorithms that are capable of fast online action recognition.

AB - Recognising human actions in real-time can provide users with a natural user interface (NUI) enabling a range of innovative and immersive applications. A NUI application should not restrict users’ movements; it should allow users to transition between actions in quick succession, which we term as compound actions. However, the majority of action recognition researchers have focused on individual actions, so their approaches are limited to recognising single actions or multiple actions that are temporally separated. This paper proposes a novel online action recognition method for fast detection of compound actions. A key contribution is our hierarchical body model that can be automatically configured to detect actions based on the low level body parts that are the most discriminative for a particular action. Another key contribution is a transfer learning strategy to allow the tasks of action segmentation and whole body modelling to be performed on a related but simpler dataset, combined with automatic hierarchical body model adaption on a more complex target dataset. Experimental results on a challenging and realistic dataset show an improvement in action recognition performance of 16% due to the introduction of our hierarchical transfer learning. The proposed algorithm is fast with an average latency of just 2 frames (66 ms) and outperforms state of the art action recognition algorithms that are capable of fast online action recognition.

KW - Online action recognition

KW - Online interaction recognition

KW - Hierarchical

KW - Transfer learning

U2 - 10.1016/j.cviu.2015.12.001

DO - 10.1016/j.cviu.2015.12.001

M3 - Article

VL - 144

SP - 62

EP - 72

JO - Computer Vision and Image Understanding

JF - Computer Vision and Image Understanding

SN - 1077-3142

ER -