Interactive world simulator for robot policy training and evaluation

RoboHub News — imagine you want to teach a robot to push an object on a table. The standard recipe in robot learning is to collect hundreds of expert demonstrations on a real robot, train an imitation learning policy on that data, and then evaluate the policy by running it many times on the same real robot.

Both stages (data collection and evaluation) are slow, expensive, and hard to reproduce: hardware breaks, lighting changes, objects drift out of place, and every new task means more hours in the lab. A natural question is whether we can replace some of this real-robot work with a simulator. Classical physics-based simulators are powerful, but building one for a new task means manually modeling geometries, contacts, friction, and deformation, and the resulting simulator often still does not match reality closely enough for policies trained inside it to transfer. In our work, we take a different route. We build an Interactive World Simulator: a learned, action-conditioned video prediction model that, given the current image and a sequence of robot actions, predicts the next frames purely in pixel space, with no physics engine inside. You can plug in a teleoperation device and control the robot through this learned world model for more than 10 minutes at 15 FPS on a single RTX 4090, and the predicted video stays stable and physically plausible. https://aihub.org/wp-content/uploads/2026/07/twitter_1.mp4 The key idea is that, if the simulator is faithful enough, we could unlock two long-standing bottlenecks in robot learning: – Data generation for training becomes cheap, because we can collect demonstrations inside the simulator. – Policy evaluation becomes scalable and reproducible, because we can roll many policies through the simulator under identical conditions. https://aihub.org/wp-content/uploads/2026/07/twitter_2.mp4 What the world simulator can do We trained our world simulator on four manipulation tasks that span very different physical regimes: T pushing (rigid-body contact), rope routing (deformable–rigid interaction with a clip), mug grasping (fine-grained gripper dynamics), and pile sweeping (manipulating piles of objects). All four behaviors are learned from interaction data alone, with no physics priors hard-coded. A few examples of what the model captures: Rope routing: it correctly distinguishes between the rope actually being inserted into the clip and the rope swinging past it without making contact. Crucially, it does not bias toward either outcome — it follows what the actions imply. https://aihub.org/wp-content/uploads/2026/07/twitter_5.mp4 Mug grasping: it captures fine-grained effects such as the mug slipping out of the gripper, or the handle being nudged and rotated. https://aihub.org/wp-content/uploads/2026/07/twitter_6.mp4 Pile sweeping: it can generate video for multiple viewpoints consistently https://aihub.org/wp-content/uploads/2026/07/twitter_8.mp4 You can try this directly on our project page: just open a browser and play with it using your keyboard! How we built the interactive world simulator At a high level, Interactive World Simulator is trained in two stages.