Skip to main content

Digital Trends may earn a commission when you buy through links on our site. Why trust us?

Should your self-driving car kill you to save a school bus full of kids?

Volvo Autonomous Car
It’s the near future and you’re reading this on your way to work in your self-driving car. The human driver of the car in front of yours slams on the brakes. Your car’s reaction time is roughly the speed of light, so it has time to realize that the stopping distance is too short and to see that the lane next to you is empty.

A quick swerve barely interrupts your morning browse of the headlines. The system works.

Now it’s ten years later. Human-driven cars have been banned from the major commuter routes because they’re unsafe at any speed. Wouldn’t you know it, but exactly the same situation comes up. This time, though, your car accelerates and slams itself into a nearby abutment, knowing full well that the safety equipment isn’t going to save you.

Your car murdered you. As it should have.

Once cars are networked, it would be immoral and irresponsible to continue to take self-preservation as the highest value.

In this second scenario, not only are all the cars on the highway autonomous, they are also networked. The cars know one another’s states and plans. They can – and should – be programmed to act in such a way that the overall outcome is the best possible: more humans saved, fewer injured. It’s just like a simulation in which a computer is given some distressing multi-car situation and has to figure out what combined set of actions would be best. But now, it’s real.

Unfortunately for you, the networked cars figured out that to save the busload of children, you had to be sacrificed.

The two scenarios represent programs embodying different moral philosophies, a topic scientists and philosophers are now beginning to notice: The MIT Technology Review cites a study about whether people are ok with their cars making such decisions. (Result: Yes, so long as the respondents are not the ones sacrificed.) This summer a workshop at Stanford considered some of these questions, as did an Oxford University Rhodes Scholar, Ameen Barghi. In fact, I posed some of these questions a year ago). But this process has just begun.

Meanwhile, the problems get complex quickly.

In the first near future scenario, each self-driving car is designed to maximize the safety of its occupants. That’s all the cars can do because they don’t know what any other car is going to do.


The engineers who wrote the first scenario’s car program thought of it as a set of accident-avoidance routines. But it embodies a moral imperative: prioritize preserving the life of this car’s passengers. That’s really all that the designers of the first generation of self-driving cars can do, even though focusing only on one’s own welfare without considering the effect on others is what we would normally call immoral.

But once cars are networked, it would be immoral and irresponsible to continue to take self-preservation as the highest value.  If a human acted that way, we well might sympathize, explaining it as a result of what we think of as genetic wiring. But we also admire those who put themselves at risk for the sake of others, whether they’re medical personnel flocking to ebola sites, teachers who step in front of a gunman entering a classroom, or soldiers who throw themselves on hand grenades. We recognize their ultimate sacrifice while wondering if we would manage to do the same.

Networked self-driving cars can get over weaknesses in the moral decisions made by human drivers.

Self-driving cars will have two moral advantages over us: when networked they can see more of a situation than any individual human can, and they can be hard-wired to steel their nerves when it comes time to make the ultimate sacrifice…of their passengers.

Networked self-driving cars can in these ways get over weaknesses in the moral decisions made by human drivers. But, this will require their human programmers to make moral decisions based on values about which humans will not, and perhaps cannot, agree.

For example, perhaps the networked results show that either of two cars could be sacrificed with equal overall results. One has a twenty-five year old mother in it. The other has a seventy-year old childless man in it. Do we program our cars to always prefer the life of someone young? Of a parent? Do we give extra weight to the life of a medical worker beginning a journey to an ebola-stricken area, or a renown violinist , or a promising scientist or a beloved children’s author? Or, should we simply say that all lives are of equal worth? That may well be the most moral decision, but it is not one we make when deciding which patients will get the next available organ for transplantation. In fact, should we program our cars so that if they have to kill someone, they should do it in the way least likely to damage their transplantable organs? Do we prefer to sacrifice the person who by speeding, or by failing to get her brakes inspected, caused the accident?

autonomous driving

These same questions arise in every situation in which we require our machines to make decisions. An autonomous killing drone, especially when networked with other drones, can know more about a complex situation than human pilots can. But how are we going to decide what counts as an “acceptable risk” of civilian casualties? It’s entirely plausible that the moral answer is “zero,” even though that is not the answer we give when it’s a human finger on the bomb release trigger. And that, of course, ignores the inevitability of failures of the programming, a risk that has been highlighted recently by Stephen Hawking, Elon Musk, Bill Gates, and quite clearly by the Terminator and Robocop series.

The behavior of programmable machines is an extension of human desires, will, and assumptions. So of course the programs themselves express moral preferences. As more of our lives are wrapped into autonomous machines, we’ll have to take the moral dimension of our programmed devices more seriously. These decisions are too important to be left to the commercial entities that are doing the programming. It’s just not clear who should be settling these difficult questions of morality.

David-shorensteinDavid Weinberger writes about the effect of technology on ideas. He is the author of Small Pieces Loosely Joined and Everything Is Miscellaneous, and is the co-author of The Cluetrain Manifesto. His most recent book, Too Big to Know, about the Internet’s effect on how and what we know.

Dr. Weinberger is a senior researcher at the Berkman Center. He has been a philosophy professor, journalist, strategic marketing consultant to high tech companies, Internet entrepreneur, advisor to several presidential campaigns, and a Franklin Fellow at the US State Department. He was for four years the co-director of the Harvard Library Innovation Lab, focusing on the future of libraries.

Editors' Recommendations

David Weinberger
Dr. Weinberger is a senior researcher at the Berkman Center. He has been a philosophy professor, journalist, strategic…
How a big blue van from 1986 paved the way for self-driving cars
Lineup of all 5 Navlab autonomous vehicles.

In 1986, a blue Chevy van often cruised around the streets of Pittsburgh, Pennsylvania near Carnegie Mellon University. To the casual observer, nothing about it appeared out of the ordinary. Most people would pass by it without noticing the camcorder peeking out from its roof, or the fact that there were no hands on the steering wheel.

But if any passerby had stopped to inspect the van and peer into its interior, they would have realized it was no ordinary car. This was the world's first self-driving automobile: A pioneering work of computer science and engineering somehow built in a world where fax machines were still the predominant way to send documents, and most phones still had cords. But despite being stuck in an era where technology hadn't caught up to humanity's imagination quite yet, the van -- and the researchers crammed into it -- helped to lay the groundwork for all the Teslas, Waymos, and self-driving Uber prototypes cruising around our streets in 2022.

Read more
Aptiv’s machine learning-powered radar sees even what you don’t
lyft and aptivs self driving car program has come a long way but not far enough aptiv screen press

Aptiv traveled to CES 2022 to showcase the improvements it has made to its suite of advanced driver assistance systems. It notably leveraged the power of machine learning technology to help its self-driving prototypes detect and classify objects, even those that are out of sight.

Think of a self-driving car as a human being; radars are the eyes and machine learning technology is the brain. Fitting radars to a car's body allows it to scope out the environment it operates in. It can detect that there's a car in front of it, that there's a bike coming the other way, and that there's a traffic light it needs to stop for. These are fairly straightforward tasks that most self-driving prototypes already perform.

Read more
Tesla pulls latest Full Self-Driving beta less than a day after release
The view from a Tesla vehicle.

False collision warnings and other issues have prompted Tesla to pull the latest version of its Full Self-Driving (FSD) beta less than a day after rolling it out for some vehicle owners.

Tesla decided to temporarily roll back to version 10.2 of FSD on Sunday following reports from some drivers of false collision warnings, sudden braking without any apparent reason, and the disappearance of the Autosteer option, among other issues.

Read more