Real-Time Infinite World Generation
with Latent 3D Gaussian Memory
1 KAIST AI2 Ewha Womans University3 KAIST Urban AI Institute
TL;DR L3G-World couples L3G-Mem and L3G-Gen to generate consistent worlds over long horizons. Its 3D memory grows adaptively with explored scene extent, while direct rendering into the diffusion latent space enables real-time generation.
L3G-Mem spawns compact Gaussians for newly explored regions and refines existing ones on revisits. Memory grows with scene extent rather than the number of generated frames, allowing GPU memory use to level off during repeated exploration.
Each Gaussian stores VAE latents that render directly into the diffusion latent space, bypassing RGB rendering and VAE re-encoding. Fast memory update and rendering enable real-time generation with few-step distillation.
L3G-Mem updates the persistent 3D memory from each generated video chunk. L3G-Gen then uses that memory to condition subsequent chunk generation. This closed loop preserves a consistent world throughout long-horizon exploration.
Given a single input image, L3G-World autoregressively generates minute-long videos while maintaining scene consistency. At each autoregressive step, L3G-Mem adaptively updates the persistent 3D memory from each generated video chunk, ensuring that the memory grows with the scene extent rather than the generated stream.
Loading videos…
Spatially-aligned memory condition enables video generation along disverse input camera trajectories.
Starting from a single image, L3G-World generates new views along an input camera trajectory while updating a 3D memory.
Loading videos…
A small set of input views initializes a 3D scene memory. L3G-World uses this memory to explore beyond the observed viewpoints.
Loading videos…
L3G-Mem reconstructs image anchors placed at designated camera poses into a shared 3D memory. This memory provides spatial context for generating views around and between the anchored content. For example, two image anchors initialize separate memories that are combined in a common coordinate system, allowing L3G-World to generate a continuous scene connecting the two worlds.
Loading videos…
L3G-World generalizes to a diverse environments with input camera trajectories, using 3D memory to connect newly generated views to earlier observations.
Loading videos…
L3G-Mem reconstructs video streams in real-time, acheiving 47 FPS on a single A100 GPU. It spawns new Gaussians in newly observed regions, while refining existing ones to maintain a consistent 3D representation. The Gaussians also store VAE latents, which can be directly rendered to the target views with efficient rendering pipeline.
From one or a few input images, L3G-World generates new views while updating a persistent 3D Gaussian memory. We optimize the 3D Gaussian memory with the generated views, turning the memory into an explorable 3D reconstruction.
Interactive world models generate visual content conditioned on user actions, yet maintaining spatial consistency over long-horizon exploration in real time remains challenging. Existing approaches rely on either temporal context or explicit 3D memory, both of which scale poorly because memory grows with the length of the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which precludes real-time generation. In this paper, we introduce a Latent 3D Gaussian Memory (L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we propose L3G-World, in which a memory updater and a video diffusion transformer (DiT) alternate autoregressively to generate spatially consistent worlds. The memory updater spawns Gaussians in newly observed regions and refines existing ones, so that the memory grows with scene extent rather than stream length. Each Gaussian further stores a VAE latent feature, allowing the memory to be rendered directly into the DiT's latent space without VAE re-encoding. Experiments show that L3G-World achieves state-of-the-art generation quality and long-horizon spatial consistency, with up to 38.6× fewer 3D primitives, 3.2× faster memory updates, and 359× faster memory rendering relative to the most efficient prior 3D-memory baseline for each metric, enabling real-time world generation at 9 FPS.
L3G-Mem reconstructs compact 3D Gaussian memory from observed or generated views. It spawns Gaussians in newly observed regions and refines existing geometry and features as new views arrive. Each Gaussian stores a VAE feature for direct latent rendering.
L3G-Mem adds Gaussians where the current memory lacks coverage and refines existing Gaussians as new observations arrive. Selective spawning avoids redundant primitives, while residual refinement corrects geometry instead of preserving its errors. Together, they improve reconstruction quality with a compact memory that grows with the explored scene.
Loading reconstruction comparisons…
Each Gaussian stores a VAE latent feature, so the memory renders directly into the video generator’s conditioning space. This removes the RGB-rendering and VAE-encoding round trip. In the L3G-Mem ablation, direct latent rendering reaches 943.6 FPS compared with 2.97 FPS without VAE lifting, while also improving latent reconstruction quality.
L3G-Gen generates video chunks conditioned on text, camera trajectories, and latent renderings of 3D memory. Each generated chunk returns to L3G-Mem to update the scene before the next chunk is generated.
Scroll horizontally for all metrics →
Scroll horizontally for all metrics →
arXiv@misc{choi2026l3gworldrealtimeinfiniteworld,
title={L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory},
author={Jinhyeok Choi and JoungBin Lee and Jongmin Lee and Minkyeong Jeon and Jiyoung Kim and Kaeun Han and Yoonjin Yoon and Seungryong Kim},
year={2026},
eprint={2610.40614},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2610.40614},
}