{"id":1147961,"date":"2025-08-20T09:00:00","date_gmt":"2025-08-20T16:00:00","guid":{"rendered":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/?p=1147961"},"modified":"2025-09-09T10:06:53","modified_gmt":"2025-09-09T17:06:53","slug":"mindjourney-enables-ai-to-explore-simulated-3d-worlds-to-improve-spatial-interpretation","status":"publish","type":"post","link":"https:\/\/new-cm-edgedigital.pages.dev\/en-us\/research\/blog\/mindjourney-enables-ai-to-explore-simulated-3d-worlds-to-improve-spatial-interpretation\/","title":{"rendered":"MindJourney enables AI to explore simulated 3D worlds to improve spatial interpretation"},"content":{"rendered":"\n
\"Three<\/figure>\n\n\n\n

A new research framework helps AI agents explore three-dimensional spaces they can\u2019t directly detect. Called MindJourney<\/a>, the approach addresses a key limitation in vision-language models (VLMs), which give AI agents their ability to interpret and describe visual scenes.  <\/p>\n\n\n\n

While VLMs are strong at identifying objects in static images, they struggle to interpret the interactive 3D world behind 2D images. This gap shows up in spatial questions like \u201cIf I sit on the couch that is on my right and face the chairs, will the kitchen be to my right or left?\u201d\u2014tasks that require an agent to interpret its position and movement through space. <\/p>\n\n\n\n

People overcome this challenge by mentally exploring a space, imagining moving through it and combining those mental snapshots to work out where objects are. MindJourney applies the same process to AI agents, letting them roam a virtual space before answering spatial questions. <\/p>\n\n\n\n

How MindJourney navigates 3D space<\/h2>\n\n\n\n

To perform this type of spatial navigation,\u00a0MindJourney\u00a0uses a\u00a0world model<\/em>\u2014in this case,\u00a0a video generation system trained on a large collection of videos captured from a single moving viewpoint, showing actions such as going forward and turning left or right, much like a 3D cinematographer. From this, it learns to predict how a new scene would appear from different perspectives.<\/p>\n\n\n\n

At inference time, the model can generate photo-realistic images of a scene based on possible movements from the agent\u2019s current position. It generates multiple possible views of a scene while the VLM acts as a filter, selecting the constructed perspectives that are most likely to answer the user’s question.<\/p>\n\n\n\n

These are kept and expanded in the next iteration, while less promising paths are discarded. This process, shown in Figure 1, avoids the need to generate and evaluate thousands of possible movement sequences by focusing only on the most informative perspectives.<\/p>\n\n\n\n

\"Figure
Figure 1. Given a spatial reasoning query, MindJourney searches through the imagined 3D space using a world model and improves the VLM’s spatial interpretation through generated observations when encountering new challenges. <\/em><\/figcaption><\/figure>\n\n\n\n

 <\/p>\n\n\n\n

To make its search through a simulated space both effective and efficient, MindJourney uses a spatial beam search<\/em>\u2014an algorithm that prioritizes the most promising paths. It works within a fixed number of steps, each representing a movement. By balancing breadth with depth, spatial beam search enables MindJourney to gather strong supporting evidence. This process is illustrated in Figure 2.<\/p>\n\n\n\n

\"MindJourney
Figure 2. The MindJourney workflow starts with a spatial beam search for a set number of steps before answering the query. The world model interactively generates new observations, while a VLM interprets the generated images, guiding the search throughout the process.<\/figcaption><\/figure>\n\n\n\n

By iterating through simulation, evaluation, and integration, MindJourney can reason about spatial relationships far beyond what any single 2D image can convey, all without the need for additional training. On the Spatial Aptitude Training (SAT) benchmark, it improved the accuracy of VLMs by 8% over their baseline performance.<\/p>\n\n\n\n\t

\n\t\t\n\n\t\t

\n\t\tSpotlight: Event Series<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"Research\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

Microsoft Research Forum<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

Join us for a continuous exchange of ideas about research in the era of general AI. Watch the first four episodes on demand.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tWatch on-demand\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n
\n