KD-Net: Continuous Keystroke Dynamics Based Human Identification from RGB-D Image Sequences

Keystroke dynamics is a soft biometric based on the assumption that humans always type in uniquely characteristic manners. Previous works mainly focus on analyzing the key press or release events. Unlike these methods, we explore a novel visual modality of keystroke dynamics for human identification using a single RGB-D sensor. In order to verify this idea, we create a dataset dubbed KD-MultiModal that contains 243.2K frames of RGB images and depth images, obtained by recording the video of hand typing with a single RGB-D sensor. The dataset comprises RGB-D image sequences of 20 subjects (10 males and 10 females) typing sentences and each subject types around 20 sentences. In the task, only the hand and keyboard region contributes to the person identification, so we also propose methods of extracting Regions of Interest (RoI) for each type of data. Unlike the data of the key press or release, our dataset not only captures the velocity of pressing and releasing different keys and the typing style of specific keys or combinations of keys but also contains rich information on the hand shape and posture. To verify the validity of our proposed data, we adopt deep neural networks to learn distinguishing features from different data representations, including RGB-KD-Net, D-KD-Net and RGBD-KD-Net. Simultaneously, the sequence of point clouds also can be obtained from depth images given the intrinsic parameters of the RGB-D sensor, so we also study the performance of human identification based on the point clouds. Extensive experimental results show that our idea works and the performance of the proposed method based on RGB-D images is the best, which achieved 99.44% accuracy based on the unseen real-world data. To inspire more researchers and facilitate relevant studies, the proposed dataset will be publicly accessible together with the publication of this paper.