Deep learning for real time facial expression recognition in social robots

Ariel Ruiz-Garcia, Nicola Webb, Vasile Palade, Mark Eastwood, Mark Elshaw

    Research output: Chapter in Book/Report/Conference proceedingConference proceedingpeer-review

    14 Citations (Scopus)


    Human robot interaction is a rapidly growing topic of interest in today’s society. The development of real time emotion recognition will further improve the relationship between humans and social robots. However, contemporary real time emotion recognition in unconstrained environments has yet to reach the accuracy levels achieved on controlled static datasets. In this work, we propose a Deep Convolutional Neural Network (CNN), pre-trained as a Stacked Convolutional Autoencoder (SCAE) in a greedy layer-wise unsupervised manner, for emotion recognition from facial expression images taken by a NAO robot. The SCAE model is trained to learn an illumination invariant down-sampled feature vector. The weights of the encoder element are then used to initialize the CNN model, which is fine-tuned for classification. We train the model on a corpus composed of gamma corrected versions of the CK+, JAFFE, FEEDTUM and KDEF datasets. The emotion recognition model produces a state-of-the-art accuracy rate of 99.14% on this corpus. We also show that the proposed training approach significantly improves the CNN’s generalisation ability by over 30% on nonuniform data collected with the NAO robot in unconstrained environments.

    Original languageEnglish
    Title of host publicationNeural Information Processing - 25th International Conference, ICONIP 2018, Proceedings
    PublisherSpringer-Verlag London Ltd
    Number of pages11
    ISBN (Electronic)978-3-030-04221-9
    ISBN (Print)9783030042202
    Publication statusE-pub ahead of print - 17 Nov 2018
    Event25th International Conference on Neural Information Processing, ICONIP 2018 - Siem Reap, Cambodia
    Duration: 13 Dec 201816 Dec 2018

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    Volume11305 LNCS
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349


    Conference25th International Conference on Neural Information Processing, ICONIP 2018
    CitySiem Reap


    • Deep convolutional neural networks
    • Emotion recognition
    • Greedy layer-wise training
    • Social robots
    • Stacked convolutional autoencoders

    ASJC Scopus subject areas

    • Theoretical Computer Science
    • Computer Science(all)


    Dive into the research topics of 'Deep learning for real time facial expression recognition in social robots'. Together they form a unique fingerprint.

    Cite this