Should your self-driving car kill you to save a school bus full of kids?

Volvo Autonomous Car
Volvo
It’s the near future and you’re reading this on your way to work in your self-driving car. The human driver of the car in front of yours slams on the brakes. Your car’s reaction time is roughly the speed of light, so it has time to realize that the stopping distance is too short and to see that the lane next to you is empty.

A quick swerve barely interrupts your morning browse of the headlines. The system works.

Now it’s ten years later. Human-driven cars have been banned from the major commuter routes because they’re unsafe at any speed. Wouldn’t you know it, but exactly the same situation comes up. This time, though, your car accelerates and slams itself into a nearby abutment, knowing full well that the safety equipment isn’t going to save you.

Your car murdered you. As it should have.

Once cars are networked, it would be immoral and irresponsible to continue to take self-preservation as the highest value.

In this second scenario, not only are all the cars on the highway autonomous, they are also networked. The cars know one another’s states and plans. They can – and should – be programmed to act in such a way that the overall outcome is the best possible: more humans saved, fewer injured. It’s just like a simulation in which a computer is given some distressing multi-car situation and has to figure out what combined set of actions would be best. But now, it’s real.

Unfortunately for you, the networked cars figured out that to save the busload of children, you had to be sacrificed.

The two scenarios represent programs embodying different moral philosophies, a topic scientists and philosophers are now beginning to notice: The MIT Technology Review cites a study about whether people are ok with their cars making such decisions. (Result: Yes, so long as the respondents are not the ones sacrificed.) This summer a workshop at Stanford considered some of these questions, as did an Oxford University Rhodes Scholar, Ameen Barghi. In fact, I posed some of these questions a year ago). But this process has just begun.

Meanwhile, the problems get complex quickly.

In the first near future scenario, each self-driving car is designed to maximize the safety of its occupants. That’s all the cars can do because they don’t know what any other car is going to do.

2010-Autonomous-Audi-TTS-Pikes-Peak-10
Audi
Audi

The engineers who wrote the first scenario’s car program thought of it as a set of accident-avoidance routines. But it embodies a moral imperative: prioritize preserving the life of this car’s passengers. That’s really all that the designers of the first generation of self-driving cars can do, even though focusing only on one’s own welfare without considering the effect on others is what we would normally call immoral.

But once cars are networked, it would be immoral and irresponsible to continue to take self-preservation as the highest value.  If a human acted that way, we well might sympathize, explaining it as a result of what we think of as genetic wiring. But we also admire those who put themselves at risk for the sake of others, whether they’re medical personnel flocking to ebola sites, teachers who step in front of a gunman entering a classroom, or soldiers who throw themselves on hand grenades. We recognize their ultimate sacrifice while wondering if we would manage to do the same.

Networked self-driving cars can get over weaknesses in the moral decisions made by human drivers.

Self-driving cars will have two moral advantages over us: when networked they can see more of a situation than any individual human can, and they can be hard-wired to steel their nerves when it comes time to make the ultimate sacrifice…of their passengers.

Networked self-driving cars can in these ways get over weaknesses in the moral decisions made by human drivers. But, this will require their human programmers to make moral decisions based on values about which humans will not, and perhaps cannot, agree.

For example, perhaps the networked results show that either of two cars could be sacrificed with equal overall results. One has a twenty-five year old mother in it. The other has a seventy-year old childless man in it. Do we program our cars to always prefer the life of someone young? Of a parent? Do we give extra weight to the life of a medical worker beginning a journey to an ebola-stricken area, or a renown violinist , or a promising scientist or a beloved children’s author? Or, should we simply say that all lives are of equal worth? That may well be the most moral decision, but it is not one we make when deciding which patients will get the next available organ for transplantation. In fact, should we program our cars so that if they have to kill someone, they should do it in the way least likely to damage their transplantable organs? Do we prefer to sacrifice the person who by speeding, or by failing to get her brakes inspected, caused the accident?

autonomous driving
Volvo
Volvo

These same questions arise in every situation in which we require our machines to make decisions. An autonomous killing drone, especially when networked with other drones, can know more about a complex situation than human pilots can. But how are we going to decide what counts as an “acceptable risk” of civilian casualties? It’s entirely plausible that the moral answer is “zero,” even though that is not the answer we give when it’s a human finger on the bomb release trigger. And that, of course, ignores the inevitability of failures of the programming, a risk that has been highlighted recently by Stephen Hawking, Elon Musk, Bill Gates, and quite clearly by the Terminator and Robocop series.

The behavior of programmable machines is an extension of human desires, will, and assumptions. So of course the programs themselves express moral preferences. As more of our lives are wrapped into autonomous machines, we’ll have to take the moral dimension of our programmed devices more seriously. These decisions are too important to be left to the commercial entities that are doing the programming. It’s just not clear who should be settling these difficult questions of morality.

David-shorensteinDavid Weinberger writes about the effect of technology on ideas. He is the author of Small Pieces Loosely Joined and Everything Is Miscellaneous, and is the co-author of The Cluetrain Manifesto. His most recent book, Too Big to Know, about the Internet’s effect on how and what we know.

Dr. Weinberger is a senior researcher at the Berkman Center. He has been a philosophy professor, journalist, strategic marketing consultant to high tech companies, Internet entrepreneur, advisor to several presidential campaigns, and a Franklin Fellow at the US State Department. He was for four years the co-director of the Harvard Library Innovation Lab, focusing on the future of libraries.

The views expressed here are solely those of the author and do not reflect the beliefs of Digital Trends.

Cars

Ford recalls 1.3 million Explorer and F-150 vehicles over safety issues

Ford announced multiple safety recalls on Wednesday, June 12, the largest of which affects 1.2 million Explorer SUVs over an issue with the suspension that has the potential to affect steering control.
Deals

Looking to upgrade? These are the best iPhone deals for June 2019

Apple devices can get expensive, but if you just can't live without iOS, don't despair: We've curated an up-to-date list of all of the absolute best iPhone deals available for May 2019.
Home Theater

The best Apple AirPod accessories can extend your high-end earbuds' life span

It might seem a bit strange to buy an accessory for an accessory, but sometimes you need a little something extra, like a case, cover, or transmitter, to get the most out of your Apple AirPods.
Deals

Before buying an Apple Watch, check out this cheap smartwatch under $100

Not looking to spend hundreds of dollars on a smartwatch? You might want to consider this little-known but well-reviewed cheap smartwatch from Amazfit called the Bip. It's on sale on Amazon, and you can have it in one day if you're a Prime…
Deals

Need a new tablet? Here are the best Apple iPad deals for June 2019

In the wide world of tablets, Apple is still the king. If you're on team Apple and just can't live without iOS, we've curated an up-to-date list of all of the best iPad deals currently available for June 2019.
Cars

Here’s why BMW mechanics now carry smartglasses in their tool chest

BMW technicians in the United States have started wearing smartglasses, and it's not because they're shooting alien ships between oil changes. They use augmented reality technology to access workshop manuals.
Product Review

The last American midsize plug-in hybrid is being squeezed out of existence

The 2019 Ford Fusion Energi plug-in hybrid can’t catch a break. Ford is planning to discontinue production next year, but looming tariffs could spell doom for an otherwise comfortable and very economically sensible sedan.
Cars

Watch as Volkswagen’s ID R racer wins the ultimate EV bragging rights

The Volkswagen ID R has set a lap record for electric cars at Germany's Nürburgring Nordschleife. The electric race car's lap time of 6 minutes, 5.33 seconds beat the previous record by 40 seconds.
Cars

Will Detroit be the ultimate test for Argo A.I.’s self-driving Ford?

Ford-backed Argo A.I. revealed its third-generation autonomous prototype in Detroit. Based on the Ford Fusion Hybrid, the car is fitted with technology that sees better and further, thinks faster, and is more comfortable to ride in.
Cars

Tesla screens may support YouTube with next software update

Tesla CEO Elon Musk announced today at E3 that the infotainment screens will support YouTube video streaming very soon. This most likely lines up with the latest software update that is expected later this year.
Cars

Volvo has created an autonomous truck that looks like a sports car

Volvo’s autonomous truck, called Vera, is about to start work at a Swedish port. The vehicle has a modest top speed of 25 mph and features a striking design that makes it look as if the usual cab has been swapped for a sports car.
Cars

Florida allows autonomous cars to drive on its roads without human supervision

Florida governor Ron DeSantis passed a law that establishes a legal framework for self-driving cars to operate within the state. It allows car and tech companies to test self-driving cars without a human operator behind the wheel.
Cars

New Toyota tech will automatically shut off engines, apply parking brakes

Toyota is launching two new safety features for the 2020 model year. One will automatically shut off a car's engine when stationary, and the other will automatically shift into park and apply the parking brake to prevent rollaways.
Cars

Aston Martin will put its Valkyrie hybrid hypercar to the ultimate test

The Aston Martin Valkyrie will race at the 24 Hours of Le Mans in 2021. Aston Martin is taking advantage of new rules that encourage automakers to bring their fastest hypercars to the legendary French race.