Skip to main content

Waymo rules and Apple trails in California self-driving car benchmarks

Before they are let loose on public streets and highways, autonomous vehicles will need to prove their capability and safety. Absent a universal measure of self-driving technology performance, autonomous mode disengagement is the current benchmark. In self-driving operations, disengagement refers to any event when a human driver turns off (disengages) a vehicle’s autonomous mode to take control of the vehicle.

Each year the State of California Department of Motor Vehicles releases reports of self-driving car disengagements for all companies that tested autonomous vehicles on public roads. This year’s reports from 48 companies are included in the Autonomous Vehicle Disengagement Reports 2018.

Based on the most recent California data, Waymo had the lowest disengagement rate — the best performance — of all companies that reported. GM Cruise had the second best performance. Apple self-driving program, however, reported the highest rate of disengagements, trailing all other reporting companies.

California’s DMV doesn’t rate or grade autonomous vehicle company disengagement rates; it just reports the numbers provided. Based on those numbers, Apple’s self-driving test vehicles disengaged every 1.1 miles. GM Cruise reported disengagement every 5,204.9 miles, and Waymo more than doubled GM Cruise’s distance, driving a whopping 11,017 miles between disengagements.

The definition of disengagement is fuzzy. California’s DMV defines disengagement as “deactivation of the autonomous mode when a failure of the autonomous technology is detected or when the safe operation of the vehicle requires that the autonomous vehicle test driver disengage the autonomous mode and take immediate manual control of the vehicle,” according to The Robot Report.

Even if the definition of disengagement was airtight, however, is easy to find fault with the report’s validity when comparing companies. First off, the companies self-report, which opens questions of company interpretation, application, control, and enforcement.

Second, the reported total miles driven in each company’s testing revealed a huge range. Waymo’s test cars recorded 1,271,587 miles and GM Cruise’s autonomous test cars drove 447,621 miles in 2o18. Among other companies reporting, Mercedes-Benz vehicles drove 1,749 miles, Toyota 381 miles, Honda 168 miles, BMW 41 miles, and Telenav only 30 miles — the lowest of all companies. Comparing company technology based on hundreds of thousands and even more than a million miles with companies with just a few thousand miles or less defies logic. Apple didn’t put together a massive driving log, with 79,745 miles in 2018, the third highest miles driven, after Waymo and GM Cruise.

A quick look at the data might suggest that miles alone are the deciding factor in disengagement frequency, but that’s still not the whole story. Driver experience level, the types of roads traveled and the time of day and distance per trip could all affect the results, but the reported data lacks those levels of granularity.

Regardless of the actual validity of comparing the disengagement reports from various companies, distance traveled does affect data reliability — meaning how likely the same vehicles would perform in the same manner with more miles.

Waymo and GM Cruise continue their testing. Apple, however, always secretive about products and programs in development, recently laid off 200 employees from its mysterious Titan self-driving car program.

Editors' Recommendations

Bruce Brown
Digital Trends Contributing Editor Bruce Brown is a member of the Smart Homes and Commerce teams. Bruce uses smart devices…
Cruise autonomous vehicle drives over woman just after she was hit by another car
A Cruise autonomous car.

An autonomous vehicle (AV) operated by Cruise ran over a pedestrian in San Francisco on Monday night just after she’d been hit by another car, the San Francisco Chronicle reported.

According to witnesses, the force of the initial impact knocked the woman into the path of the Cruise robotaxi, leaving her pinned under one of its wheels. The driver in the other car reportedly fled the scene.

Read more
Robotaxi firm Cruise ordered to halve fleet following incidents
A Cruise autonomous car.

Autonomous car company Cruise has been told by regulators to halve its robotaxi fleet in San Francisco following a crash with a fire truck on Thursday in which the driverless car's passenger suffered minor injuries.

The regulator -- the California Department of Motor Vehicles (DMV) -- said that it’s looking into “recent concerning incidents” involving self-driving Cruise cars operating on the city’s public roads.

Read more
An autonomous car in San Francisco got stuck in wet concrete
A Cruise autonomous car.

A self-driving car operated by General Motors-backed Cruise got stuck on Tuesday when it drove into a patch of wet concrete.

The incident happened in San Francisco and occurred just days after California's Public Utilities Commission made a landmark decision when it voted to allow autonomous-car companies Cruise and Waymo to expand their paid ridesharing services in the city to all hours of the day instead of just quieter periods.

Read more