Ask4NV

Can Learning to Answer Spatial Questions Improve
Novel-View Generation?

by Jahyeok Koo, Hyeonseo Yu, Seungho Jang, Jaewoo Jung, Jiwon Kang, Seokju Cho, and Seungryong Kim†

KAIST AI  ·  †Corresponding author

Ask4NV learns novel-view generation jointly with 3D language understanding in one unified multimodal model. Spatial question answering and visual grounding supervise the generation transformer through joint self-attention, improving image fidelity and geometric consistency in and out of domain.

Paper Code

arXiv 2026

Learning to answer spatial questions makes a generative model render better novel views. Ask4NV is a framework that trains novel-view generation jointly with 3D language understanding, so that spatial question answering and grounding supervise the generation model.

Motivation

Novel-view generation methods rely on fine-grained geometric cues such as depth-based warping and cross-view correspondence. These cues capture scene structure, but not the high-level scene properties that also determine how a scene should look from a new viewpoint: object identities, attributes, and inter-object relationships. 3D language understanding learns exactly these properties, by answering questions about objects and their spatial relations. We therefore ask whether learning to answer spatial questions can supervise novel-view generation.

Shared scene properties across 3D language understanding and novel-view generation
Novel-view generation
3D language understanding
Shared 3D scene
shared 3D scene with the object the example is about

Questions about object categories, attributes, and spatial relationships address the same scene properties that should be represented in novel views.

1. Does it improve novel-view generation?

If spatial supervision helps, it has to show first as better novel views, and not only on the training distribution. We compare against task-specific novel-view generators and a unified multimodal baseline, in and out of domain.

Finding 1Ask4NV improves novel-view image quality across multiple benchmarks.

Four source views, four target views, on DL3DV and ScanNet (in domain) and RE10K and ScanNet++ (out of domain), against CAT3D, Matrix3D, NVComposer, CAMEO, and Omni-View. PSNR, SSIM, and LPIPS measure image fidelity; camera-pose error and MEt3R measure 3D consistency. Ask4NV has the best PSNR and LPIPS on all four datasets and the best pose accuracy, and the out-of-domain gains show that the improvement generalizes.

Novel-view generation on four benchmarks

Bold best, underline second; † reproduced CAT3D.

Qualitative comparison

2. What does 3D language supervision improve inside the generation transformer?

The understanding objectives are losses on language: answering spatial questions and grounding objects. We ask whether they nonetheless change the spatial representations the generation transformer builds for its VAE latent tokens, measured by cross-view correspondence, which is never supervised during training.

Finding 23D language supervision improves cross-view correspondence in VAE representations.

Lance-FT is the same Lance checkpoint fine-tuned on novel-view generation only. In a two-view setting we take clean VAE tokens after attention at each layer, match every object token to its most similar token in the other view, and score with PCK@1/2/4. Layer 26 gives the strongest correspondence for both models, and Ask4NV outperforms Lance-FT on both datasets at every threshold. Correspondence in these VAE representations improves without any direct correspondence supervision: the benefit of 3D language understanding reaches the spatial representations used for generation.

Cross-view correspondence by layer
Ask4NVLance-FTlayer 26

Correspondence score of VAE-token matches between two views, at every transformer layer. Ask4NV, Lance-FT.

Cross-view correspondence at layer 26
Attention-feature similarity maps at layer 26
Query view · query point in red
query view

For the query point (red) in the query view, the similarity of every location in the other view.

3. Does joint training make generated views closer to the ground truth in a 3D-aware feature space?

We ask whether joint training also brings the generated views closer to the ground truth in a 3D-aware representation space: the feature space of an encoder trained for 3D language understanding, which encodes object identities, attributes, and spatial layout.

Finding 3Joint training improves similarity in a 3D-aware feature space.

The encoder is the SigLIP vision encoder fine-tuned for 3D language understanding inside Video-3D-LLM. We extract features of the generated and ground-truth views at the same target viewpoint and report the mean cosine similarity between spatially corresponding patches. Ask4NV is higher than the generation-only baseline on all four datasets, so joint training produces views that are more closely aligned with the ground truth in the feature space of a 3D language understanding encoder.

Patch-wise feature similarity to the ground truth

Generated vs. ground-truth views in the 3D-tuned SigLIP space of Video-3D-LLM. Mean cosine similarity, higher is better.

BibTeX

@article{koo2026ask4nv,
  title   = {Can Learning to Answer Spatial Questions Improve Novel-View Generation?},
  author  = {Koo, Jahyeok and Yu, Hyeonseo and Jang, Seungho and Jung, Jaewoo and Kang, Jiwon and Cho, Seokju and Kim, Seungryong},
  journal = {arXiv preprint arXiv:2610.20610},
  year    = {2026}
}