Dual Deep Neural Network Approach to Matching Data in Different Modes

This paper investigates the application of a novel Deep Neural Network (DNN) architecture to the problem of matching data in different modes. Initially one DNN is pre-trained as a feature extractor using several stacked Restricted Boltzmann Machine (RBM) blocks on the entire training data using unsupervised learning. This DNN is duplicated and each net is fine-tuned by training on the data represented in a specific mode using supervised learning. The target of each DNN is linked to the output from the other DNN thus ensuring matching features are learnt which are adjusted to take differing representation into account. These features are used with some distance metric to determine matches. The expected benefit of this approach is utilizing the capability of DNN to learn higher level features which can better capture the information contained in the input data's structure, while ensuring the differences in data representation are accounted for. The architecture is applied to the problem of matching faces and sketches and the results compared to traditional approaches employing Principal Component Analysis (PCA) or Linear Discriminant Analysis (LDA).