Before they are let loose on public streets and highways, autonomous vehicles will need to prove their capability and safety. Absent a universal measure of self-driving technology performance, autonomous mode disengagement is the current benchmark. In self-driving operations, disengagement refers to any event when a human driver turns off (disengages) a vehicle’s autonomous mode to take control of the vehicle.
Each year the State of California Department of Motor Vehicles releases reports of self-driving car disengagements for all companies that tested autonomous vehicles on public roads. This year’s reports from 48 companies are included in the Autonomous Vehicle Disengagement Reports 2018.
Based on the most recent California data, Waymo had the lowest disengagement rate — the best performance — of all companies that reported. GM Cruise had the second best performance. Apple self-driving program, however, reported the highest rate of disengagements, trailing all other reporting companies.
California’s DMV doesn’t rate or grade autonomous vehicle company disengagement rates; it just reports the numbers provided. Based on those numbers, Apple’s self-driving test vehicles disengaged every 1.1 miles. GM Cruise reported disengagement every 5,204.9 miles, and Waymo more than doubled GM Cruise’s distance, driving a whopping 11,017 miles between disengagements.
The definition of disengagement is fuzzy. California’s DMV defines disengagement as “deactivation of the autonomous mode when a failure of the autonomous technology is detected or when the safe operation of the vehicle requires that the autonomous vehicle test driver disengage the autonomous mode and take immediate manual control of the vehicle,” according to The Robot Report.
Even if the definition of disengagement was airtight, however, is easy to find fault with the report’s validity when comparing companies. First off, the companies self-report, which opens questions of company interpretation, application, control, and enforcement.
Second, the reported total miles driven in each company’s testing revealed a huge range. Waymo’s test cars recorded 1,271,587 miles and GM Cruise’s autonomous test cars drove 447,621 miles in 2o18. Among other companies reporting, Mercedes-Benz vehicles drove 1,749 miles, Toyota 381 miles, Honda 168 miles, BMW 41 miles, and Telenav only 30 miles — the lowest of all companies. Comparing company technology based on hundreds of thousands and even more than a million miles with companies with just a few thousand miles or less defies logic. Apple didn’t put together a massive driving log, with 79,745 miles in 2018, the third highest miles driven, after Waymo and GM Cruise.
A quick look at the data might suggest that miles alone are the deciding factor in disengagement frequency, but that’s still not the whole story. Driver experience level, the types of roads traveled and the time of day and distance per trip could all affect the results, but the reported data lacks those levels of granularity.
Regardless of the actual validity of comparing the disengagement reports from various companies, distance traveled does affect data reliability — meaning how likely the same vehicles would perform in the same manner with more miles.
Waymo and GM Cruise continue their testing. Apple, however, always secretive about products and programs in development, recently laid off 200 employees from its mysterious Titan self-driving car program.
