Much of the autonomous vehicle (AV) space is uncharted territory. However, at Waymo, with more than 200 million miles driven fully autonomously, we’re one of very few companies that can look to our past to illuminate our future. Our experience has led us to ten fundamental truths that shape how we build AI. These truths are validated by our safety data, which shows the Waymo Driver is already making roads safer in the cities where we serve. More than safety being the output, safety is the reason for each insight. Let’s dive into what we’ve learned, starting with the two most heavily debated topics in AV history. 1. Multimodal sensors are indispensable Cameras are incredible, but they aren’t enough. For years, there’s been a debate over whether cameras alone could solve full autonomy. Now, after more than 200 million real-world miles, the data is clear: safe, fully autonomous operations at scale require more. By combining inputs from cameras, lidar, and radar, the Waymo Driver creates a rich, redundant world view that no single sensor can replicate. In addition to redundancy, each sensor brings complementary sensing strengths that bolster safety. Lidar provides the wireframe, capturing 3D geometry with millimeter precision. Cameras provide the semantic overlay with their ability to read street signs and detect colors of traffic lights. Radar serves as a dynamic sentinel, tracking velocity and “seeing” through what obscures cameras, like heavy rain, fog, or dust. 2. HD maps are a powerful “prior” This brings us to the second great AV debate—to map or not to map. At Waymo, we use high-definition (HD) maps to jump-start our validation process, so we can provide a fully autonomous service to riders from our first trip. As we drive, we treat our maps as another input—like our sensors, but acting as a mental memory. It is there as an additional source of information, proving incredibly helpful in poor visibility and complex thoroughfares. This allows the onboard computer to dedicate its real-time processing power to what is dynamic or new, such as a sudden detour or a temporary stop sign. Our AI-driven mapping system ensures these maps are continuously updated, providing the vehicle with a reliable, high-fidelity reference to lean on during complex maneuvers. 3. Fewer, larger models are better In the early days of AV development, the industry relied on specialized modules. For example, one for pedestrian detection, another for car tracking, another to tell when a light turns green. While agile, this modular spaghetti becomes unmaintainable at scale. By consolidating to fewer, high-capacity, specialized foundation models, we are better able to leverage the power of vast datasets and large-scale computation. This strategy lets the data, rather than brittle human priors, determine what is relevant, allowing models with sufficient capacity to develop complex reasoning capabilities. This less-is-more approach allows us to benefit from the same scaling laws that have propelled breakthroughs in large language models, using efficient teacher-student models to optimize onboard compute. However, while fewer models are better, that does not mean we’re consolidating our processing into a single, black box. 4. You can’t build trust with a black box AI is powerful, but it isn’t magic. Pure end-to-end (E2E) neural architectures, where a model takes in raw pixels and directly outputs steering commands, run the risk of black box failures. That means, it’s hard to understand how decision making happens in full E2E systems. While Waymo’s system transforms sensors into driving decisions in real time, we’ve added an independent onboard validation layer. This architectural choice is non-negotiable for safely scaling at L4. This is a separate, AI-based safety system that monitors every trajectory proposed by the Waymo Driver. It checks these plans against hard physics-based constraints and traffic laws by incorporating techniques like Reinforcement Learning and reasoning inspired by Generative AI. If the AI proposes a path that violates a limit or risks a collision, the validation layer acts as a hard backstop. 5. Closed-loop simulation reveals more edge cases If you want to train an AI to safely handle highly dynamic, rare scenarios — like a vehicle suddenly cutting three lanes of traffic on a freeway — it’s really difficult to test in the real world. Simulation allows us to test, validate, and improve our performance for the everyday and the one in a million events we navigate on a weekly basis. However, we’ve learned that simply replaying recorded data, known as open-loop simulation, is insufficient because it’s like stepping into a video replay – the surrounding traffic moves, but it’s completely indifferent to your actions. Large-scale, closed-loop simulation is critical. It provides the most realistic assessment by mimicking real-world cause and effect. In a closed loop, if the Waymo Driver swerves or brakes to avoid that aggressive lane change, the surrounding traffic will react naturally to its actions. This creates a feedback loop, which is essential for understanding complex interactions and for unlocking powerful techniques like Reinforcement Learning. It’s impossible to find every edge case on the road, so a closed-loop simulation allows us to discover and test some of the rarest events before we encounter them on the road. 6. Every great driver needs a great Critic We all dislike back seat drivers, but what if we engineered a helpful one? At Waymo, we built an AI critic to analyze, understand, and detect undesirable driving behaviors both in simulation and on the road. The Waymo Critic allows for a continuous, automated feedback loop that examines the millions of road miles traveled each week (and tens of billions in simulation), allowing our human engineering talent to focus on the most complex edge cases. Without a robust and discerning Critic, the Driver risks “grading its own homework.” The Critic is tuned to catch a range of driving behaviors, from safety and traffic law compliance to how smoothly the car progresses through a turn or how comfortable the braking feels. By combining this powerful Critic with our calibrated driving data, we can precisely measure the Driver’s quality in any scenario. 7. Vision Language Models improve scene reasoning Driving requires complex, chain-of-thought reasoning. When a car encounters a police officer using hand signals at the site of a collision, the Driver must understand the intent of those signals within the context of the surrounding scene. This is where Vision-Language Models (in our case, trained with Gemini) become crucial reasoning partners. They provide high-level semantic “hints” to the Driver, helping it navigate situations that are far outside its standard training data. Our experience shows that while Vision-Language Models (VLMs) excel at high-level reasoning, they are too slow for real-time control and lack sufficient spatial awareness on their own. To bridge this gap, our system adopts a “thinking fast and slow” architecture. It relies on rapid, intuitive processing and sensor fusion for instantaneous, real-time control (thinking fast), while leveraging VLMs for deep, deliberative reasoning (thinking slow). This dual approach allows the Waymo Foundation Model to perceive the world with high fidelity across diverse sensor inputs while simultaneously navigating complex, long-tail scenarios and anticipating future developments. 8. Our holistic approach shows AI is only as effective as the governance that evaluates it Most people think that when you’ve built an autonomous Driver, your work is done. Simply “develop and go” misses the bigger picture. True scalability is only possible if driving, simulation, and evaluation are created with a safety governance to determine their readiness. Our entire strategy is built on this holistic premise. At the heart of our readiness framework is a rigorous, quantitative data engine composed of multiple complementary evaluation methodologies. We augment this empirical core with expert human judgment and proper safety governance — ensuring every deployment decision is grounded in proven metrics. 9. A data flywheel enables continuous improvement A core pillar of our Waymo Values is to always be learning. The Waymo Driver is constantly getting better over time thanks to our automated data flywheel—a virtuous cycle of continuous improvement that accelerates our ability to scale. Driving millions of miles a week, we leverage our automated systems, including our Critic and feedback from riders and communities, to highlight opportunities for refinement. We extract and examine the relevant data, use advanced auto-labelers to categorize it, retrain our models, and validate it through simulation, and then our safety framework. Operationalizing this cycle across exabytes of data is what allows an L4 system to systematically address the long tail of driving scenarios. 10. There is no substitute for fully autonomous experience The final lesson is that there is no substitute for actual fully autonomous miles. Simply improving a driver-assist system (L2) for full autonomy is a false summit. True L4 maturity can only be safely achieved by a purpose-built system, validated on closed courses and hardened by the uncompromising experience of driving without a human in the car. You can run billions of miles in simulation or with human supervision, but an AV system only truly matures when it is solely responsible for the driving task. Full autonomy exposes the system to the true gravity of its decisions and reveals novel situations that humans or simulations might unconsciously smooth over. The Road Ahead As we look toward the billions of miles ahead, these ten lessons remind us that safety is the direct result of rigorous, real-world experience. While these highlights only scratch the surface of what we’ve learned operating at scale, you can take a deeper look at our specific AI approach in this blog post. If you want to help reshape the frontier of machine intelligence, see our job openings. Join us! Article from Waymo.
