GRACE : Generation-Aware Latent Compression
for Efficient Video Generation

Abstract

Highly compressed video autoencoders offer an effective way to accelerate video diffusion models, as the Diffusion Transformer (DiT) operates on far fewer tokens. However, such autoencoders are challenging to train, since a higher compression ratio degrades reconstruction quality and recovering it requires more channels, which is known to slow the convergence of the DiT. The compressed latent also differs from the one the DiT was trained on, so the pretrained DiT must be either retrained from scratch or adapted at considerable cost. Compressing the autoencoder the DiT was trained with appears to preserve compatibility, yet optimizing it for reconstruction alone still shifts the latent away from the distribution the DiT has learned. To address this, we propose Generation-Aware Latent Compression for Efficient Video Generation (GRACE), a two-stage framework that compresses a pretrained video autoencoder while keeping it compatible with the pretrained DiT. Specifically, we keep a frozen base latent from the pretrained encoder and learn a residual latent for the information lost under stronger compression, while aligning the compressed latent with the pretrained latent in the feature space of the frozen DiT so that the autoencoder is optimized for generation. We then adapt the DiT with lightweight fine-tuning and asymmetric denoising, where the base is denoised ahead of the residual. GRACE reduces the token count of Wan2.1-I2V-14B by 8× and its latency by 11.1× at 480×832×81, while matching the generation quality of the pretrained pipeline before compression on VBench.

Method

GRACE compresses Wan2.1's latent from f8t4p2 to f16t8p2 (32.8k → 4.3k tokens at 480×832) in two stages. In both, the pretrained pipeline guides its own compression.

Overall architecture. Stage 1 trains the autoencoder against the frozen DiT. Stage 2 adapts the DiT with LoRA, denoising z_base ahead of z_res.

Stage 1

Autoencoder training

Dual-latent representation

A frozen base latent from the pretrained Wan2.1 encoder anchors the compressed latent in the space the DiT already knows. A learned residual latent carries the detail and motion that stronger compression throws away.

Generation-aware alignment

Instead of reconstruction losses alone, the compressed latent is matched to the pretrained one inside the frozen DiT's feature space, so the autoencoder is optimized for generation.

Effect of the dual latent and L_align. (a) single latent, (b) dual latent without L_align, (c) Ours. The latent becomes more uniform and its principal components follow objects across frames.

Stage 2

DiT adaptation

Asymmetric denoising

The DiT's input and output projections are extended to the residual channels, with LoRA on the transformer blocks. The base is denoised ahead of the residual by an offset δ in the same forward pass, so the residual always builds on a more settled base, at no extra sampling cost.

Denoising order. Keeping z_base δ ahead recovers facial detail under large motion.

Qualitative comparison

Same caption and seed for every model, each run with its own prompt expansion. Videos in a row play in sync.

More comparisons, including GRACE-only samples, are in the gallery.

Quantitative results

Ablation

Design components

Each component improves generation on both tasks. Row (VI) keeps the baseline autoencoder but trains the DiT from scratch for the same number of steps.

BibTeX

@misc{kim2026gracegenerationawarelatentcompression,
      title={GRACE: Generation-aware latent compression for efficient video generation}, 
      author={Jiyoung Kim and Paul Hyunbin Cho and Jisu Nam and Donghoon Lee and Hyunsung Go and Yeonkyeong Lee and Hansaem Kim and Seungryong Kim},
      year={2026},
      eprint={2610.10524},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2610.10524}, 
}