DreamX

DreamX

Video Diffusion Model is a Universal Latent Action Learner

Jisoo Kim1*, Chaehyun Kim1*, Junsu Kim1, Jiho Park1,
Seongchan Kim1, Eunbeen Hong1, Soowon Son1,
Fu-En Yang2, Seungryong Kim1
1KAIST2NVIDIA
*Equal contribution
Paper arXiv Code soon Results

DreamX reads compact latent actions out of a pretrained video diffusion model: one motion space for humans and robots alike.

300×
fewer tokens than the video latent, per frame
0.17
LARYBench action MSE
(best baseline 0.38)
57.4
FVD, zero-shot human→robot
(best baseline 170.2)
2.19k h
of human + robot video
from 6 datasets

Overview

One action space for every embodiment

DreamX turns a pretrained video diffusion model into a latent action learner. Its encoder reads compact latent actions out of human and robot videos while the video prior takes care of appearance, so the latents capture motion alone. The same latent actions then drive video transfer, human-to-robot action transfer, and a compact latent policy.

Method

How DreamX works

One pretrained video diffusion backbone is both the encoder and the decoder.

1

Read out motion

Learnable queries read the video features: 4 tokens per frame, ~300× compression.

2

Reconstruct the future

The decoder rebuilds the future from the first frame + latent actions, so the latents carry motion, not appearance.

3

Reuse anywhere

New first frame → video transfer. Robot state → actions. Compact codes → policy.

Text Video Learnable query Noisy video Noisy action ⋯ ⋯ Cosmos Predict 2.5 Self Attention Cross Attention FFN LoRA LoRA Self Attention Cross Attention FFN LoRA LoRA Encoder ℰϕ ⋯ Latent action ⋯ Decoder 𝒟θ ⋯ ℒvideo ℒaction (a) Latent Action Pretraining
Encoder Decoder Video Query Action Attends Masked (b) Attention Mechanism

(a) Queries → latent actions (yellow) → decoder denoises future video and actions.

(b) Rows attend to columns. Hover a group.

Pretraining data

2.19k hours of human and robot video

Six datasets of egocentric human and robot video. Human clips need no action labels.

2.19k hours · 6 datasets

    Quantitative results

    State of the art on LARYBench

    Regressing end-effector trajectories and classifying motions from latent actions. DreamX leads every benchmark.

    Action regression

    Normalized MSE, average of 4 benchmarks (lower is better)

    Action classification

    Accuracy %, average of 3 benchmarks (higher is better)
    Show the full LARYBench table
    Action regression (MSE ↓)Action classification (acc. % ↑)
    MethodCALVINVLABenchRoboCOINAgiBotWorld-βAvg.Atomic robotComp. humanComp. robotAvg.
    Latent actions from 2 frames
    LAPA0.960.950.961.000.9722.2714.6123.6420.17
    UniVLA0.820.740.940.970.8718.6219.0818.5618.75
    villa-X0.860.720.940.970.8715.0017.8029.9020.90
    LAPA-MAGVIT20.590.360.800.830.6533.1259.7029.5340.78
    LAPA-SigLIP20.570.300.860.880.6546.8354.7429.4443.67
    LAPA-DINOv20.550.260.850.860.6358.0455.8634.1949.36
    LAPA-DINOv30.500.250.820.840.6056.2764.1927.0449.17
    Latent actions from video
    MoTo0.430.130.540.680.4441.5755.1156.3451.01
    AdaWorld0.640.260.940.940.6931.5849.8051.4844.29
    VideoWorld20.720.410.820.850.7014.5030.2437.3627.37
    DiLA0.430.090.510.520.3858.5165.4762.7662.25
    DreamX0.150.050.260.220.1768.0372.7268.2669.67
    DreamX + FAE-320.230.070.310.260.2265.9069.4966.0667.15

    Motion transfer · video

    Video transfer

    Encode a source video, decode from a new first frame: the motion carries over, the scene and embodiment stay. No paired data, no extra training.

    Source Video Learnable query Target Frame State ⋯ ⋯ Cosmos Predict 2.5 Self Attention Cross Attention FFN Self Attention Cross Attention FFN Encoder ℰϕ ⋯ Source action ⋯ Decoder 𝒟θ ⋯ Transferred Video Video transfer

    Video transfer. Source latent actions + target first frame → transferred video.

    Any-to-any video transfer

    Across scenes, robots, humans, and beyond manipulation.

    Zero-shot human → robot video transfer

    DreamX follows the human’s motion and keeps the robot and scene intact; AdaWorld blurs, VideoWorld2 leaks the hand.

    Human · source
    DreamX (ours)

    HumanAndRobot benchmark

    MethodFVD ↓LPIPS ↓PSNR ↑SSIM ↑
    AdaWorld170.190.34422.950.846
    VideoWorld2241.920.29922.530.881
    DreamX57.440.17926.520.917

    Motion transfer · action

    Action transfer

    Decode the same latent actions into robot actions from the robot’s initial state. No paired human–robot data.

    Source Video Learnable query Target Frame State ⋯ ⋯ Cosmos Predict 2.5 Self Attention Cross Attention FFN Self Attention Cross Attention FFN Encoder ℰϕ ⋯ Source action ⋯ Decoder 𝒟θ ⋯ Transferred Action Action transfer

    Action transfer. Source latent actions + robot state → executable actions.

    Human → robot, on a real robot

    Open-loop on a bimanual YAM: stroke order, direction changes, and the moving arm are preserved.

    Human demonstration
    Robot execution

    Analysis

    Where do latent actions look?

    Latent-action queries follow the moving hand or gripper, for humans and robots alike.

    Input
    Attention
    Input (left) and one query slot’s attention (right; brighter = higher).

    Control

    A compact latent policy

    Predict compact latent-action codes (2,048-d → 32-d) instead of future video; a DiT head turns them into robot actions.

    Latent action Encoder μψ log σ2ψ ℒKL Compact Code Decoder Reconstructed action ℒrec Text Obs. State Self Cross FFN LoRA LoRA LoRA Action DiT ℒquery ℒaction (b) Latent Policy Model

    Left: compress latent actions into codes. Right: denoise codes from text + observation; the Action DiT predicts actions.

    Citation

    BibTeX

    @article{kim2026dreamx,
  title   = {DreamX: Video Diffusion Model is a Universal Latent Action Learner},
  author  = {Kim, Jisoo and Kim, Chaehyun and Kim, Junsu and Park, Jiho and Kim, Seongchan and Hong, Eunbeen and Son, Soowon and Yang, Fu-En and Kim, Seungryong},
  journal = {arXiv preprint arXiv:2611.05377},
  year    = {2026}
}