Context is king: How Avride uses cloud VLMs as a safety net for delivery robots

Avride has integrated vision-language models into its delivery robots. has built its delivery robots for high level of autonomy.

Every single day, hundreds of them navigate busy city streets entirely on their own, processing complex sensor data locally on their onboard compute units. Our sidewalk robots run with minimal human involvement, reliably handling standard urban maneuvers, pedestrians, and traffic lights on their own. However, efficiently managing the mechanics of navigation – even in challenging conditions like narrow pathways or bad weather – is only one part of the equation. Ensuring a robot behaves appropriately in unusual, sensitive, or high-stakes real-world environments requires a different kind of intelligence. To add a proactive layer of environmental awareness, we have integrated heavy, cloud-based vision-language models (VLMs) into its system as an automated “VLM-watcher.” From object detection to holistic scene understanding Avride’s onboard perception stack is already highly capable. Using a combination of onboard sensors and local neural networks, our delivery robots are designed to detect surrounding agents, including cyclists, children, wheelchairs, and emergency vehicles. However, while our onboard models can identify these individual elements, certain real-world scenarios require a much deeper layer of contextual understanding. Consider how a scenario unfolds on a city street. Encountering a police officer or a firefighter on the sidewalk might hint that something unusual is happening, but basic object detection isn’t enough to grasp the full picture. For instance, distinguishing a police officer walking home after a shift from an active, sensitive crime scene is a highly non-trivial task. It requires a holistic understanding of how multiple elements interact within the frame – interpreting the scene as a whole scenario rather than a mere checklist of detected objects. We want to significantly reduce the likelihood of our delivery robots accidentally entering an active emergency area, crossing a live crime scene, or rolling into unmapped roadwork where fresh, wet cement looks just like a standard grey sidewalk. While onboard models capture the primary entities needed to navigate, a heavy foundation model in the cloud excels at this holistic interpretation, instantly piecing together the deep semantic context of the entire situation. Submit your session idea for the 2026 RoboBusiness How it works: VLMs as cloud guardians It is important to clarify: we do not use VLMs to drive the robot. Using a heavy cloud model to steer in real time would introduce latency and connectivity dependencies that compromise safety.