Towards Integrating Learning by Demonstration and Learning by Instruction in a Multimodal Robotics

Learning by demonstration and learning by instruction offers a potentially more powerful paradigm than programming robots directly for specific tasks. Learning in humans or primates substantially benefits from demonstration of actions or instruction by language in the appropriate context and there is initial neurocognitive cortical evidence for such processes. Cortical assemblies have been identified in the cortex that activate in response to the performance of motor tasks at a semantic level. This evidence supports that such mirror neuron assemblies are involved in actions, observing actions and communicating actions. Furthermore, neurocognitive evidence supports that cell assemblies are activated in different regions of the brain dependent on the
action type being processed. Based on this neurocognitive evidence we have begun to design a neural robot in the MirrorBot project that is based on multimodal integration and topological organisation of actions using associative memory. As part of these studies in this paper we describe a self-organising model that clusters actions into different locations dependent on the body part they are associated with. In particular, we use actual sensor readings from the MIRA robot to represent semantic features of the action verbs. Furthermore, ongoing work focuses on integration of
motor, vision and language representations for learning from
demonstration and language instruction.