Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence

Siyoon Jin*1Dahyun Chung*1Honggyu An1Sangbeom Lim2Wonjin Nam1Jaewoo Jung1WonJun Moon1Seungryong Kim1
1KAIST AI    2Korea University
*Equal contribution
NeurIPS 2026
arXiv Code

Abstract

Cross-view object correspondence identifies the same object across views. However, existing methods treat this as a frame-level matching problem, necessitating a query mask at every frame. Furthermore, without modeling temporal context, they suffer from severe ambiguity under appearance changes or occlusions. Therefore, we introduce cross-view temporal object correspondence, which requires preserving object identity across both views and time from a query mask. To address this, we propose XTrack, a streamlined framework which repurposes a pretrained video DiT with semantic and spatiotemporal priors for RGB-to-mask latent transport, jointly modeling cross-view matching and temporal tracking. However, naive video DiTs do not guarantee accurate cross-view grounding and often attend to semantically similar yet incorrect objects across views. Thus, we introduce focal cross-view attention alignment, hard negative conditioning, and local context crop strategies to ensure reliable grounding. Finally, beyond these architectural enhancements, we also tackle the lack of dense annotation by introducing a pseudo-label curation pipeline that converts sparse Ego-Exo4D annotations into dense supervision to enable effective video-level training. Our highly competitive performances validate the effectiveness of XTrack as a streamlined framework, ensuring robust temporal consistency and identity preservation.

Motivation

Given a first-frame query mask in one view, the task is to recover temporally coherent masks of the same object in both views. Frame-level matchers must be paired with a tracker: (a) Track-then-Match propagates the query mask and matches it per frame, (b) Match-then-Track matches once and tracks each view independently. Both break under disappearance and viewpoint change. (c) XTrack models cross-view correspondence and temporal dynamics jointly in a single video DiT.

paradigms
Query mask (Exo view, first frame)
Ground truth
XTrack (Ours)
(a) Track-then-Match
✗ temporally inconsistent matching
✗ false positives when the target is absent
ObjectRelator
O-MaMa
LM-EEC
V2-SAM
(b) Match-then-Track
✗ propagates an unreliable initial match
✗ fails when the object reappears
ObjectRelator
O-MaMa
LM-EEC
V2-SAM

Method

architecture
Overall architecture. The ego and exo videos are placed side by side in a single frame and fed to a pretrained video DiT, which transports RGB latents to mask latents for both views at once.
RGB-to-mask flow
  • Flow matching from the RGB latent to the mask latent, keeping the video DiT priors.
Unified ego-exo tokens
  • Ego and exo frames are placed side by side in one input, so cross-view matching and within-view tracking happen in the same attention.
Focal attention alignment
  • Cross-view attention is supervised to land on the query object.
Hard negative conditioning
  • Run the same videos with the correct query mask and with another object's mask; a margin loss keeps the two predictions apart.
Local context crop
  • Object crop tokens with original RoPE positions for small objects.
Two-stage training
  • Image-level cycle-consistency pretraining, then video fine-tuning on curated pseudo-labels.

Pseudo-Label Curation

Ego-Exo4D provides 30 FPS video but object masks only at 1 FPS, which is too sparse to train a video model, and plain SAM2 propagation is unreliable in the ego view. Our curation pipeline propagates the sparse annotations and verifies them across views and within each view, turning the 1 FPS annotations into 30 FPS pseudo-labels. The resulting dense, verified labels are what XTrack is fine-tuned on at the video level.

pseudo-label pipeline
a
Bidirectional propagation
Forward and backward SAM2 between anchors; chunks where the two propagations disagree are selected for verification.
b
Cross-view geometric verification
VGGT similarity to the exo view selects one of the two candidates.
c
Within-view semantic verification
DINOv3 similarity to the first-frame ego anchor discards chunks that are still wrong.

Cross-view geometric verification

Exo query (left), the forward (red) and backward (blue) ego candidates, and the candidate selected by VGGT similarity to the exo view (right).

Within-view semantic verification

Geometric verification only selects between the forward and backward candidates, so it cannot recover a chunk where both candidates are wrong. Semantic verification complements it: the selected candidate is compared with the first-frame ego anchor, and chunks with low similarity are discarded.

Cross-view geometric verification — both candidates miss the ball, so selecting one of them is not enough.
Within-view semantic verification — compared with the first-frame ego anchor (left, green), the candidates have low DINOv3 similarity and the chunk is discarded.

Qualitative results

From a single query mask (left), XTrack predicts the object masks in both the ego and exo videos (right) in one pass. The predicted masks stay on the queried object in both views through viewpoint changes, occlusions and re-appearance, without any per-frame query.

BibTeX

@inproceedings{xtrack2026,
  title     = {Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence},
  author    = {Jin, Siyoon and Chung, Dahyun and An, Honggyu and Lim, Sangbeom and Nam, Wonjin and Jung, Jaewoo and Moon, WonJun and Kim, Seungryong},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}