Trace4D

TL;DR. Trace4D teaches a VLM to first write down where things are at each annotated frame, as a 4D state sequence of 2D boxes and 3D centroids, and then reason over how those states change, checking each change against what is visible in the video before answering.

Abstract

Understanding how objects and their spatial relationships evolve over time remains challenging for vision-language models (VLMs), particularly under changing viewpoints. Recent approaches incorporate explicit 4D information into model responses, but how this information supports the final answer can remain implicit. We present Trace4D, a framework that trains VLMs to use predicted 4D state changes and temporal visual observations as joint evidence for video reasoning. Its 4D-grounded chain-of-thought (CoT) consists of a structured 4D state sequence, temporal visual analysis, and 4D-grounded reasoning that connects state changes to observations of the same targets over matching intervals and explains how they support the answer. From in-the-wild videos, we construct Trace4D-Train with 600K 4D perception examples and approximately 70K grounded question–answer (QA) examples with 4D-grounded CoT supervision. Joint supervised fine-tuning learns state prediction and grounded reasoning together, followed by geometry-aware GRPO that refines responses using answer correctness and reference 4D state agreement. At inference, Trace4D predicts its own 4D states and reasoning from video, without reference annotations or an external geometric model. Experiments on our held-out Trace4D-Bench and the external VLM4D benchmark show improved QA accuracy over the base VLMs and demonstrate the benefit of combining 4D state changes with temporal visual evidence.

Trace4D on held-out clips

Each clip is shown with its question, Trace4D's answer, and the predicted 4D values the reasoning cites. All Trace4D-Bench clips come from source videos excluded from training.

A response that shows its geometry

Every Trace4D answer is generated as y = [s; o; r; a]: the model writes its own 4D state sequence, describes what it sees phase by phase, links the two, and only then answers. The predicted states stay in context, so later reasoning can cite exact positions and their changes.

Two-stage training

Stage 1: Joint SFT. Starting from the base VLM, we train in one mix on 600K perception examples, whose target is the <4d> block alone, and about 70K grounded QA examples, whose target is the full response.

Stage 2: Geometry-aware GRPO. On about 7K QA examples, the policy is rewarded for the correct answer and for predicted states that agree with the reference: position error per timestep (R_pos), the displacement vector between consecutive timesteps (R_Δ), and box IoU (R_bbox). Reference states are used only to compute rewards. At inference the model sees just the video and the question, with no external geometric model.

Trace4D-Bench and VLM4D

Multiple-choice accuracy (%). Open-source models decoded greedily.

Trace4D-Bench, per question type

Trace4D improves on its Qwen3-VL-4B-Instruct base model by 40.33 points and beats every evaluated baseline on all 12 question types.

Citation

@article{trace4d2026,
  title   = {Trace4D: Grounded Spatiotemporal Reasoning via 4D State Sequences},
  author  = {Jang, Wooseok and Kim, Seungryong and Lu, Yan and Ji, Lei},
  journal = {arXiv preprint},
  year    = {2026}
}