Learning Temporally Aware Compact Gaussians for Feed-Forward 4D Reconstruction

Mungyeom Kim^1* Minkyeong Jeon^1* Honggyu An^1* Jaewoo Jung¹ Hyunah Ko¹ Jisang Han¹ Hyeonseo Yu¹ Donghwan Shin¹ Sunghwan Hong² Takuya Narihira³ Kazumi Fukuda³ Yuki Mitsufuji^3,4† Seungryong Kim^1†

¹KAIST AI ²ETH AI Center, ETH Zurich ³SONY AI ⁴Sony Group Corporation

^*Co-first authors ^†Co-advising authors

Paper Code Checkpoints BibTeX

TL;DR

We present T4GS (Temporally Aware Compact 4D Gaussian Splatting), a feed-forward 4D reconstruction framework built upon a compact set of timestamp-conditioned learnable query tokens. Each token aggregates features across the full temporal context and decodes a 3D Gaussian whose position is modulated by the target timestamp, enabling globally coherent motion modeling.

Overview

Dynamic scene reconstruction from monocular video remains a fundamental challenge in computer vision. Existing feed-forward methods predict per-pixel 3D Gaussians for each frame, leading to duplicated Gaussians and a view-dependent bias that hinder learning scene motion. T4GS instead decodes 3D Gaussians from a compact set of timestamp-conditioned learnable query tokens, each aggregating features across the full temporal context.

The resulting temporally coherent Gaussian representation serves as a motion-aware anchor for camera-controllable video generation and supports 4D feature lifting for point tracking and dynamic scene understanding. Across multiple dynamic benchmarks, T4GS achieves strong novel-view synthesis without input camera poses while using significantly fewer Gaussians and remaining robust to large temporal gaps.

Comparison with Previous Works

Pixel-wise 4DGS ^{[1, 2, 3]}: predicts per-pixel Gaussians, which leads to duplicated Gaussians and ghost artifacts at interpolated timestamps, and a view-dependent bias that underuses distant temporal evidence and leaves occluded regions poorly reconstructed.
Our method: uses a compact query bottleneck to aggregate globally across time and views, yielding better motion consistency with fewer duplication and occlusion failures.
[1] MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Second
[2] 4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos
[3] NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos

Failures of pixel-wise feed-forward 4DGS. (a) Duplicated Gaussians from nearby input views cause ghost artifacts. (b, c) Opacity falls off sharply away from the observed timestamp, so regions visible only in temporally distant views remain black in (b). (d) Drift grows with frame distance, confirming that pixel-wise Gaussians cannot model global motion.

Model Architecture

(a) Learnable queries aggregate encoder features from all context frames, with time embeddings of the relative timestamp, and are decoded into 3D Gaussians at the target time. (b) The globally consistent motion of T4GS enables 4D feature lifting, reusing the decoder attention, and camera-controllable video generation, where T4GS renders serve as a motion-aware anchor for a video diffusion model.

Quantitative Results

Citation