Trajectory-Aware Deformable Transformer for Contextual Video Prediction

Recent video transformers commonly adopt factorised spatial-temporal attention to reduce computational complexity. However, such factorisation limits temporal modelling by constraining attention to fixed spatial locations across frames, ignoring the motion-dependent spatial context essential for accurate prediction. To address this limitation, we propose Trajectory-Aware Deformable Attention for the video prediction task. This novel mechanism integrates motion-aware sampling into temporal attention by dynamically aligning keys and values along motion paths to restore spatial context. Patch trajectories are estimated in continuous pixel coordinates using a lightweight convolutional flow estimator with forward-backwards consistency checking for occlusion detection. For each query, a set of learnable deformable sample points are placed around the trajectory-predicted position at every source time step, and features are extracted via differentiable bilinear interpolation. The proposed method is trained and evaluated on the KITTI Caltech dataset. It achieves consistent improvements across all metrics, including a 7.6% reduction in MSE and a significant 20.1% reduction in LPIPS, and improvements in PSNR, MAE and SSIM, demonstrating the benefit of motion-aware sampling. These gains are achieved with only a 14.3% increase in FLOPs, demonstrating that it can be integrated into any video transformer at a modest additional cost.