Deep learning for real time facial expression recognition in social robots

Ariel Ruiz-Garcia, Nicola Webb, Vasile Palade, Mark Eastwood, Mark Elshaw

Research output: Chapter in Book/Report/Conference proceedingConference proceedingpeer-review

8 Citations (Scopus)


Human robot interaction is a rapidly growing topic of interest in today’s society. The development of real time emotion recognition will further improve the relationship between humans and social robots. However, contemporary real time emotion recognition in unconstrained environments has yet to reach the accuracy levels achieved on controlled static datasets. In this work, we propose a Deep Convolutional Neural Network (CNN), pre-trained as a Stacked Convolutional Autoencoder (SCAE) in a greedy layer-wise unsupervised manner, for emotion recognition from facial expression images taken by a NAO robot. The SCAE model is trained to learn an illumination invariant down-sampled feature vector. The weights of the encoder element are then used to initialize the CNN model, which is fine-tuned for classification. We train the model on a corpus composed of gamma corrected versions of the CK+, JAFFE, FEEDTUM and KDEF datasets. The emotion recognition model produces a state-of-the-art accuracy rate of 99.14% on this corpus. We also show that the proposed training approach significantly improves the CNN’s generalisation ability by over 30% on nonuniform data collected with the NAO robot in unconstrained environments.

Original languageEnglish
Title of host publicationNeural Information Processing - 25th International Conference, ICONIP 2018, Proceedings
PublisherSpringer-Verlag London Ltd
Number of pages11
ISBN (Electronic)978-3-030-04221-9
ISBN (Print)9783030042202
Publication statusE-pub ahead of print - 17 Nov 2018
Event25th International Conference on Neural Information Processing, ICONIP 2018 - Siem Reap, Cambodia
Duration: 13 Dec 201816 Dec 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11305 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference25th International Conference on Neural Information Processing, ICONIP 2018
CitySiem Reap


  • Deep convolutional neural networks
  • Emotion recognition
  • Greedy layer-wise training
  • Social robots
  • Stacked convolutional autoencoders

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'Deep learning for real time facial expression recognition in social robots'. Together they form a unique fingerprint.

Cite this