ACE Robotics and NTU S-Lab's Mundo builds simulated robot training scenes from 1 to 12 photos

ACE Robotics and the S-Lab at Nanyang Technological University (NTU) have unveiled Mundo, a world model that builds an explorable, interactive environment from one to 12 images of a real place. Pandaily, citing Chinese outlet Robot Pro, reported the release on October 31. The team says Mundo can reconstruct a scene, render it from free viewpoints and simulate it over time, so robots can be tested and trained in a digital copy of a specific space. According to the GitHub repository, code and weights will follow by the end of November.

How Mundo works

Mundo takes one to 12 context images and generates new views along a chosen camera path. With a single image, it extends the visible style and structure into unseen views. More images mean less guessing about areas that were never photographed. For large spaces, it can produce camera moves lasting tens of seconds or minutes. It picks the most relevant photos for each segment and overlaps segments to keep them consistent.

The pipeline has several stages:

• Geometry: a 3D foundation model recovers camera poses and dense geometry.
• Object detection: a 3D detector draws an initial bounding box for each major object.
• Scene correction: because detectors miss objects and misjudge sizes, a vision-language model agent edits the scene program. It renders the scene from each input viewpoint, compares the result with the photos and decides what to fix. Numerical optimization fits box position, size and orientation. A separate review agent checks consistency without editing the scene.
• Physics: the scene program records each object's name, pose, size, mass and friction. Scripts apply actions and forces, and a physics engine, currently MuJoCo, computes motion, contact and collisions.
• Rendering: a video diffusion model renders the simulation in the look of the original images.

Video model details

The team fine-tuned a MiniMax reference-to-video model on about 40,000 paired videos at 640 by 480 resolution. It generates 124-frame clips and reuses the last 34 frames of each clip to keep long videos continuous.

Demonstrations

Demos include robots navigating a reconstructed hotel floor, opening a washing machine in a laundry room, putting an apple in a bowl and stacking blocks. They show head-camera and wrist-camera views along with recorded trajectories.

Stated limitations

The team lists the following limits:

• Reconstruction takes 10 to 20 minutes.
• Rendering runs at about 1.5 frames per second on eight A800 GPUs at 20 inference steps.
• Small objects can appear to float.
• There is no long-term memory for scenes that change over long periods.

Context window, pricing and benchmark scores are not reported.

What this means

Mundo separates physical behavior from appearance. Object mass, friction and pose live in an explicit scene program that MuJoCo simulates, while the video diffusion model only renders how the result looks. That design makes the simulated scene editable and lets scripted actions and forces drive the motion.

The stated speeds show where the system stands today. Reconstruction takes 10 to 20 minutes, and rendering at about 1.5 frames per second on eight A800 GPUs is slow compared with typical video frame rates. The floating small objects and the lack of long-term memory for changing scenes also limit what can be simulated.

The code and weights are not yet public, with release promised by the end of November. Until then, the results come from the team's own demos and descriptions, and outside groups cannot reproduce them.