ScenePilot: Retrieval-Augmented and Reinforcement-Guided 3D Indoor Scene Generation

A text-driven 3D indoor scene generation framework that combines retrieval-augmented spatial priors, group-wise autoregressive scene growth, and multimodal repair for physically plausible, semantically coherent, and functionally complete layouts.

Jiawei Zhang1 Hongsong Wang1 Pan Zhou2
1Southeast University 2Singapore Management University
Paper Coming Soon Code arXiv Coming Soon
Text-to-Scene 3D Indoor Scene Generation RAG VLM Repair Reinforcement Learning

Comparison with Existing Methods

Comparison between ScenePilot and existing 3D scene generation methods
Method comparison figure:
docs/static/images/method-compare.svg

ScenePilot Pipeline

ScenePilot pipeline overview
Pipeline figure:
docs/static/images/pipeline.svg

Overall workflow. ScenePilot follows a retrieval-augmented Grow-and-Repair process. It first retrieves reusable layout priors from room-, group-, and anchor-level memories, then plans functional object groups and inserts them sequentially into the room.

After each group is inserted, lightweight local repair corrects immediate layout problems. Once the scene is complete, a final multimodal repair stage predicts structured move–rotate–scale actions from rendered views and scene state to improve physical plausibility, object relations, and functional completeness.

Abstract

Text-driven 3D indoor scene generation has advanced from dataset-bound layout modeling to open-vocabulary synthesis with large language and vision-language models. Yet existing methods remain limited: one-pass generators often yield geometrically invalid layouts, heavy post-hoc optimization is costly and unstable, and prompt-only planners lack reusable layout priors for functional grouping and object relations. We propose ScenePilot, a retrieval-augmented Grow-and-Repair framework that formulates scene generation as prior-guided incremental growth with learned rectification. Given a prompt, the Hierarchical Retrieval-Augmented Planning (HRAP) module retrieves room-, group-, and anchor-level layout priors to support functional group planning. A text-driven base generator then inserts object groups sequentially, while the Reinforcement Multimodal Repair (RMR) module performs lightweight local correction after each insertion and a final global repair after completion. To train this policy, we construct SceneReverse-17k, a repair-trajectory dataset built by perturbing high-quality 3D scenes in position, rotation, and scale, then using inverse operations as executable rectification targets. The policy predicts structured move–rotate–scale actions from rendered views, scene state, retrieved priors, and edit history. By combining HRAP with RMR, ScenePilot offers an efficient alternative to one-shot generation and heavy full-scene optimization, improving physical plausibility, functional coherence, and controllability while preserving diversity.

Method Overview

1. Retrieval-Augmented Prior Grounding

ScenePilot builds a prior memory from clean indoor scenes, including room-level compositions, anchor-specific group statistics, and object-anchor relations. Retrieved documents provide layout-aware guidance before scene construction.

2. Group-wise Autoregressive Growth

Instead of generating all objects in one pass, ScenePilot grows the scene by functional groups such as sleeping, seating, dining, working, lighting, and decoration. Each group is inserted with local context and repaired immediately.

3. Multimodal Repair and Optimization

A vision-language repair model observes rendered views and outputs executable actions such as move, rotate, and scale. The repair loop targets physical validity, relation consistency, and functional completeness.

Pipeline

  1. Prior construction: mine clean scenes to form room-level and anchor-centered spatial memory.
  2. Scene planning: retrieve relevant priors and decompose the instruction into functional groups.
  3. Progressive generation: insert each group autoregressively and apply deterministic local repair.
  4. Final visual repair: use rendered observations to produce structured layout edits.
  5. Evaluation: measure physical validity and visual-semantic quality using physics metrics and VLM/user-study scores.

Data Representation

ScenePilot data representation
Data representation figure:
docs/static/images/data representation.svg

Structured scene state. ScenePilot represents each indoor scene as an editable object-level layout with room geometry, object categories, sizes, positions, rotations, and scale attributes.

This representation is compact enough for language-model planning, yet explicit enough for deterministic checking and repair. It also makes the repair policy directly executable: predicted actions modify concrete object fields instead of producing free-form layout descriptions.

SceneReverse-17k Dataset Construction

SceneReverse-17k dataset construction
Dataset construction figure:
docs/static/images/dataset construction.svg

Reverse repair trajectories. SceneReverse-17k is constructed by starting from high-quality indoor scenes and applying controlled perturbations to object position, rotation, and scale.

The inverse transformations are then recorded as executable rectification targets. This turns static clean scenes into step-wise repair supervision, allowing the model to learn how to diagnose and correct layout errors from rendered observations, scene state, retrieved priors, and edit history.

Qualitative Results

Example generated scenes from ScenePilot with their original input prompts.

Generated modern bedroom example
Bedroom Result
docs/static/images/bedroom.svg

Prompt: create a modern bedroom include a single bed, two nightstands, drawer chest, TV stand, floor lamp, pendant lamp.

Generated modern industrial laundry example
Living Room / Laundry Result 1
docs/static/images/livingroom1.svg

Prompt: create a modern industrial laundry include vintage armchair, round dining table, four dining chairs, stainless steel refrigerator, a black washing machines, two pendant lamps, two potted plants, and a row of wine bottles.

Generated modern minimalist living room example
Living Room Result 2
docs/static/images/livingroom2.svg

Prompt: create a modern minimalist livingroom include dark gray three-seat sofa, black leather loveseat, modern armchair, two coffee tables, floor lamp, pendant lamp, and TV stand.

Generated modern living room example
Living Room Result 3
docs/static/images/livingroom3.svg
Fallback: docs/static/images/livinfroom3.svg

Prompt: create a modern living room include gray tufted sofa, striped armchair, round dining table, four dining chairs, floor lamp, chandelier, TV stand, sideboard, refrigerator, and pendant lamp.

Evaluation

We evaluate generated scenes from both physical and visual-semantic perspectives. Physical metrics include out-of-boundary loss, mesh-based collision loss, physical-based loss, and valid scene ratio. Visual-semantic metrics include layout correctness, semantic plausibility, functional completeness, and overall quality.

Metric Group Metrics Goal
Physics OOB ↓, MBL ↓, PBL ↓, VR ↑ Reduce boundary violations and object collisions.
Visual & Semantics LC ↑, SPA ↑, FC ↑, Overall ↑ Improve layout correctness, semantic plausibility, and functional completeness.
Human Evaluation LC ↑, SPA ↑, FC ↑ Measure perceived layout correctness, semantic plausibility, and functional completeness.
Ablation RAG, group repair, final VLM repair Analyze the contribution of each component.

Human Evaluation

Human evaluation comparison on LC, SPA, and FC
Human evaluation figure:
docs/static/images/human evaluation.png

Human preference study. We compare ScenePilot with Reason3D and ReSpace under three human-rated criteria: layout correctness (LC), semantic plausibility (SPA), and functional completeness (FC).

ScenePilot receives the highest scores across all three dimensions, indicating that the retrieved functional priors and grow-and-repair strategy improve not only physical validity, but also perceived semantic coherence and functional completeness.

Higher is better. The y-axis is zoomed to highlight differences among methods.

BibTeX

@article{zhang2026scenepilot,
  title={ScenePilot: Retrieval-Augmented and Reinforcement-Guided 3D Indoor Scene Generation},
  author={Zhang, Jiawei and Wang, Hongsong and Zhou, Pan},
  journal={Preprint},
  year={2026},
  abstract={Text-driven 3D indoor scene generation has advanced from dataset-bound layout modeling to open-vocabulary synthesis with large language and vision-language models. Yet existing methods remain limited: one-pass generators often yield geometrically invalid layouts, heavy post-hoc optimization is costly and unstable, and prompt-only planners lack reusable layout priors for functional grouping and object relations. We propose ScenePilot, a retrieval-augmented Grow-and-Repair framework that formulates scene generation as prior-guided incremental growth with learned rectification. Given a prompt, the Hierarchical Retrieval-Augmented Planning (HRAP) module retrieves room-, group-, and anchor-level layout priors to support functional group planning. A text-driven base generator then inserts object groups sequentially, while the Reinforcement Multimodal Repair (RMR) module performs lightweight local correction after each insertion and a final global repair after completion. To train this policy, we construct SceneReverse-17k, a repair-trajectory dataset built by perturbing high-quality 3D scenes in position, rotation, and scale, then using inverse operations as executable rectification targets. The policy predicts structured move--rotate--scale actions from rendered views, scene state, retrieved priors, and edit history. By combining HRAP with RMR, ScenePilot offers an efficient alternative to one-shot generation and heavy full-scene optimization, improving physical plausibility, functional coherence, and controllability while preserving diversity.},
  url={https://zjw-louie.github.io/ScenePilot/}
}