How Generalist uses human demonstration data for robot learning
Generalist is built on research in training robots on real-world data. And for robots, that translates into generating lots of data.
It’s also a time to catch up on reading. My beach material last June was the research paper on “Universal Manipulation Interface: In-the-Wild Robot Teaching Without in-the-Wild Robots“ (a.k.a. UMI). The academic findings quickly became the foundation for a $2 billion unicorn robotics startup, Generalist. Essentially, teams at Toyota Research Institute (TRI), Columbia University, and Stanford University created a data-collection platform that uses puppet-like end effectors operated by people, with GoPro cameras capturing real-world tasks such as washing dishes and picking up objects. The resulting demonstrations become training data for robot foundation models, enabling collaborative robots to learn and take over these tasks much more quickly. UMI’s paper demonstration image, illustrating human data capture to full autonomy on co-bots. github.io/” target=”_blank” rel=”noopener”>https://umi-gripper.github.io/ Models quickly generate polices for cobots This past June at Automate, Generalist demonstrated live how its models could quickly create policies for different cobot systems. On one side of the hall, the company demonstrated how it could make Universal Robots (UR) arms fold and build cardboard boxes, while across the McCormick Center, it used Flexiv arms to repair robot vacuums. According to the company‘s X post boasting about the display at the show: “What resonated most wasn’t just that the models could do the tasks — it was that they had the intelligence to recover in real time when things went wrong, and this got people thinking differently about automation.” Generalist uses a range of robotic hands and arms Then, almost a month later, the AI unicorn posted a blog missive titled “Towards Machines with a Thousand Hands.” To better understand the meaning of Generalist’s newest accomplishment of using different arms and grippers, including spatulas, across a plethora of use cases, I interviewed Samantha Castellanos, the company‘s founding mechanical engineer. I wanted to know the process beyond research, especially how it translated into industrial implementations to accelerate adoption and expand usage into new applications. As a starting point, Castellanos shared Generalist’s overall philosophy: “Our main goal is to make the best model in the world. And at the end of the day, I think it’s our model that’s going to be the moat that is going to differentiate us from others.” The mechanical engineer was referring to the already crowded space that has cumulatively raised more than $4 billion, including standouts like Skild AI ($2 billion), Physical Intelligence ($1 billion), Field AI ($300 million), and RLWRLD ($41 million). Data and tools make models better “Everything we do is to make the model better,” she continued. “All of the data collection, all the types of tool collection, different types of interacting with the world.
