LSA: Localized Semantic Alignment for Enhancing Temporal Consistency in Traffic Video Generation
Mirlan Karimov*, Teodora Spasojevic*, Markus Braun, Julian Wiederer, Vasileios Belagiannis, Marc Pollefeys — IEEE Intelligent Vehicles (IV) 2026
* Indicates Equal Contributions
Abstract
Controllable video generation has emerged as a versatile tool for autonomous driving, enabling realistic synthesis of traffic scenarios. However, existing methods depend on control signals at inference time to guide the generative model towards temporally consistent generation of dynamic objects, limiting their utility as scalable and generalizable data engines. In this work, we propose Localized Semantic Alignment (LSA), a simple yet effective framework for fine-tuning pre-trained video generation models. LSA enhances temporal consistency by aligning semantic features between ground-truth and generated video clips. Specifically, we compare the output of an off-the-shelf feature extraction model between the ground-truth and generated video clips localized around dynamic objects inducing a semantic feature consistency loss. We fine-tune the base model by combining this loss with the standard diffusion loss. The model fine-tuned for a single epoch with our novel loss outperforms the baselines in common video generation evaluation metrics. To further test the temporal consistency in generated videos we adapt two additional metrics from object detection task, namely mAP and mIoU. Extensive experiments on nuScenes and KITTI datasets show the effectiveness of our approach in enhancing temporal consistency in video generation without the need for external control signals during inference and any computational overheads.
Key contributions
- We introduce LSA, a fine-tuning framework that improves the temporal consistency of pre-trained video generation models within a single epoch of fine-tuning.
- We demonstrate that LSA outperforms a two-step conditional video generation method in terms of both standard video quality and detection-based evaluation metrics, while being more computationally efficient.
- We show that SVD fine-tuned with LSA can serve as a drop-in enhancement for conditional video generation, consistently improving the quality and temporal coherence of generated videos.
Method Overview
Overview of LSA (right), our proposed framework for improving temporal consistency in video generation. LSA introduces a semantic feature consistency loss that enforces alignment between the semantic representations of SVD-generated frames x̂ and their corresponding ground-truth frames x0, specifically within dynamic-object regions defined by ground-truth bounding boxes bbgt, promoting appearance consistency and temporally stable localization. Semantic features are extracted with DINOv2, while ground-truth bounding boxes provide spatial supervision. The inference stage of SVD fine-tuned with LSA is identical to the original SVD, hence not requiring bounding boxes. Example input frames are from nuScenes.
Back to publications.