Cross-Modal Semantic Enhancement via Co-Training With Supplementary Features and Labels for Multimodal Sentiment Analysis

Multimodal sentiment analysis seeks to recognize and interpret emotional expressions from heterogeneous sources such as text, images, and speech. However, existing methods still face significant limitations in robustness and generalization due to strong modality heterogeneity, alignment challenges, and label sparsity. In this paper, we propose a cross-modal collaborative semantic enhancement method that leverages supplementary features and labels to improve the model's perceptual sensitivity to multimodal sentiment information and structural adaptability. Specifically, we design three composite interaction structures based on Multi-Head Cross Attention to model high-order dependencies across modalities, thereby enabling explicit cross-modal feature fusion and complementarity. To further address the challenges posed by limited supervision, we introduce image captioning and speech recognition as auxiliary tasks, establishing external semantic generation pathways. The intermediate representations produced by these tasks are leveraged as supplementary features to help improve the primary sentiment analysis task, while their predicted labels are utilized as supplementary supervision signals to alleviate the problem of label sparsity. This collaborative framework not only facilitates semantic transfer and structural sharing between the main and auxiliary tasks, but also improves the model's robustness to missing modalities and noise. Extensive comparative and ablation experiments conducted on the public CMU-MOSEI dataset demonstrate that our approach consistently outperforms existing state-of-the-art models.