Multi-view region-adaptive multi-temporal DMM and RGB action recognition

Mahmoud Al-Faris, John P. Chiverton*, Yanyan Yang, David Ndzi

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)
26 Downloads (Pure)

Abstract

Human action recognition remains an important yet challenging task. This work proposes a novel action recognition system. It uses a novel multi-view region-adaptive multi-resolution-in-time depth motion map (MV-RAMDMM) formulation combined with appearance information. Multi-stream 3D convolutional neural networks (CNNs) are trained on the different views and time resolutions of the region-adaptive depth motion maps. Multiple views are synthesised to enhance the view invariance. The region-adaptive weights, based on localised motion, accentuate and differentiate parts of actions possessing faster motion. Dedicated 3D CNN streams for multi-time resolution appearance information are also included. These help to identify and differentiate between small object interactions. A pre-trained 3D-CNN is used here with fine-tuning for each stream along with multi-class support vector machines. Average score fusion is used on the output. The developed approach is capable of recognising both human action and human–object interaction. Three public-domain data-sets, namely MSR 3D Action, Northwestern UCLA multi-view actions and MSR 3D daily activity, are used to evaluate the proposed solution. The experimental results demonstrate the robustness of this approach compared with state-of-the-art algorithms.

Original languageEnglish
Pages (from-to)1587-1602
Number of pages16
JournalPattern Analysis & Applications
Volume23
DOIs
Publication statusPublished - 21 Apr 2020

Keywords

  • Action recognition
  • DMM
  • 3D CNN
  • Region adaptive

Fingerprint

Dive into the research topics of 'Multi-view region-adaptive multi-temporal DMM and RGB action recognition'. Together they form a unique fingerprint.

Cite this