How Rhoda AI is using video data to create generalized AI models for robots

Rhoda pretrains its models on internet-scale video to build a strong prior on motion, physics, and physical interaction. | The company is working with customers in automotive, manufacturing, logistics, and e-commerce to deploy its direct video action, or DVA, model for robotics.

“We’re focused on deploying in the real world, and that means you have to deal with the messiness of the real world. Every run can be different,” Jagdeep Singh, the co-founder and CEO of Rhoda AI, told The Robot Report. “Every configuration can be different, and you need a really robust model. To build a robust model, you need a massive training dataset.” A large training dataset is exactly what allowed large language models (LLMs) like ChatGPT to rapidly progress in recent years. Physical AI has trailed behind those models because the same dataset doesn’t exist for the physical world, noted Singh. “Every word ever put on the internet, from Shakespeare to Reddit, was used to train these AI models. In the physical AI world, you didn’t have an internet-scale dataset,” he said. “People tried to collect their own data by tele-operating robots, and that approach made for interesting lab demonstrations in robots that could fold T-shirts or make coffees and so on, but none of those demonstrations were robust enough to cross over into the real world. The number of autonomous manipulation capable robots that are solving real world problems today is effectively zero.” Rhoda AI is betting on using video to data to close this gap and to create truly generalized robotics. How does a DVA model work? Many physical AI developers are creating vision-language-action (VLA) models, which allow robots to learn from data from images or videos of its surroundings and text instructions. These models, however, still struggle to cope with the variability of the real world, according to Rhoda AI. The company claimed that its DVA model, which it recently extended with mobile DVA, combines manipulation and mobility into one policy. “The way it works is the model will look at a history of video frames and action tokens and language tokens. From that history, or from that context, it will make a prediction in video for the next few 100 milliseconds, the next half second or so,” explained Singh.