How to Use AI Agents to Prepare 3D Scenes for Simulation

PUBLISHER ARTICLE PREVIEW

From the original article

Agentic AI workflows can be used to prepare and validate digital twins for physical AI systems. Agents can inspect 3D scenes, author simulation-relevant data in OpenUSD, add physics properties, render preflight views, and validate the result against simulation-ready (SimReady) requirements. This workflow follows that process from a scene in Blender to a simulation-ready OpenUSD handoff for NVIDIA Isaac Sim or NVIDIA Isaac Lab.

In practice, however, if you’re building agents for robotics, the workflow can often get stuck. It’s tempting to blame the hard part on the policy, the model, or the training loop, but the bottleneck often happens earlier: the robot does not have a simulation-ready world to train in. The extra work required to get a 3D scene into that state is laborious, time consuming, and frequently out of scope for the robotics simulation engineer.

This post walks through an agent workflow for preparing a Blender scene for robotics simulation using NVIDIA Omniverse Libraries. Codex, powered by OpenAI GPT-6 Astra, coordinates the overall task, interprets results, and guides iteration. Specialized subagents built with the Hermes agent harness and deployed through NVIDIA NemoClaw use Omniverse Libraries to inspect the scene, author simulation metadata, configure physics, and render visual preflight views.

Together, these components connect reasoning, tool execution, and validation into a repeatable process for delivering a simulation-ready OpenUSD world.

The scene exists—the assets are there, created by a 3D artist in Blender—but is it usable for simulation? Has all of the following prep work been done?

This prep work is tedious, repetitive, and easy to get wrong. It is also exactly the kind of work agentic systems should help with, if they have the right tools. The point of using NVIDIA Omniverse Libraries in an agent workflow is to integrate the tools agents need to build SimReady worlds.

A general-purpose agent such as Codex by ChatGPT or Claude Cowork by Anthropic can look at a Blender scene and recognize that it needs to be made simulation-ready. Recognition is useful, but it’s not enough. To help a robotics developer, the agent must be able to act on the scene:

A broad request to make a 3D scene ready for simulation becomes a multi-agent engineering workflow. Codex or Claude serves as the main agent, coordinating the overall task of preparing the scene for simulation. NVIDIA NemoClaw provides a reference architecture for building the specialized subagents that perform each job. These subagents can use open source agent harnesses such as Hermes, OpenClaw, or LangChain, configured with different NVIDIA Nemotron models for vision, reasoning, and tool use.

In a configuration using Astra and Hermes, Codex uses Astra to translate the developer’s objective into tasks, identify dependencies, and review results from specialized Hermes subagents deployed through NemoClaw. For example, making an object grabbable requires coordinated updates to its semantic label, rigid-body configuration, and collision geometry. Astra helps connect those requirements across subagents and determine which checks are needed before the workflow proceeds.

NVIDIA Omniverse Libraries provide the tools the subagents call to act on the scene. OpenUSD operations establish the shared scene structure, ovphysx authors and checks physics properties, ovrtx renders visual preflight views, and SimReady validation evaluates the resulting assets against a target simulation profile.

Each subagent owns a specific job and its acceptance criteria. Safe, mechanical issues can be fixed automatically. Decisions that depend on developer intent, such as an uncertain semantic label or physical behavior, are escalated to a human with the relevant context and a proposed next step.

Together, these layers turn the prompt, “Make this scene simulation-ready” into a tool-driven workflow with specialized jobs, a persistent scene state, validation gates, and human review where judgment matters.

Begin by specifying the main objective for the orchestration agent, including the input, desired output, destination, and validation criteria. This gives Codex or Claude enough structure to coordinate the overall task, route work across specialized NemoClaw subagents, and decide when the job is actually complete.

By specifying the main objective, the task changes from simply “make this scene better” to a coordinated agent workflow. Codex or Claude manages the overall request, NemoClaw subagents reason