Skip to main content

Statistician raises red flag about reliability of machine learning techniques

Machine learning is everywhere in science and technology: powering facial recognition, picking your recommendations on Netflix, and controlling self-driving cars. But how reliable are machine learning techniques really? A statistician says that the answer is “not very,” arguing that questions of accuracy and reproducability of machine learning have not been fully addressed.

Dr Genevera Allen, associate professor of statistics, computer science, and electrical and computer engineering Rice University in Houston, Texas has discussed this topic at a press briefing and at a scientific conference, the 2019 Annual Meeting of the American Association for the Advancement of Science (AAAS). She warned that researchers in the field of machine learning have spent so much time developing predictive models that they have not devoted enough attention to checking the accuracy of their models, and that the field must develop systems which can assess the accuracy of their own findings.

“The question is, ‘Can we really trust the discoveries that are currently being made using machine-learning techniques applied to large data sets?'” Allen said in a statement. “The answer in many situations is probably, ‘Not without checking,’ but work is underway on next-generation machine-learning systems that will assess the uncertainty and reproducibility of their predictions.”

As an example, recently machine learning has been used to study patients with cancer. To study the disease, scientists use machine learning to identify genetically similar individuals so that drug therapies can then be targeted to these specific genomes. But when comparing across different studies, the clusters identified by machine learning are completely different from each other.

The problem is that machine learning techniques do not have a way to say “I don’t know” or “It’s not clear.” The techniques will generally always produce an answer — in the example of the cancer patients, they will always identify a group in some way — but this answer may not be as certain or accurate as it is believed to be. The techniques are able to find a pattern that exists in the data set, even if only dimly, but the pattern may not hold in the real world.

“There is general recognition of a reproducibility crisis in science right now,” Allen told BBC News. “I would venture to argue that a huge part of that does come from the use of machine learning techniques in science.”

Editors' Recommendations

Georgina Torbet
Georgina is the Digital Trends space writer, covering human space exploration, planetary science, and cosmology. She…
This AI cloned my voice using just three minutes of audio
acapela group voice cloning ad

There's a scene in Mission Impossible 3 that you might recall. In it, our hero Ethan Hunt (Tom Cruise) tackles the movie's villain, holds him at gunpoint, and forces him to read a bizarre series of sentences aloud.

"The pleasure of Busby's company is what I most enjoy," he reluctantly reads. "He put a tack on Miss Yancy's chair, and she called him a horrible boy. At the end of the month, he was flinging two kittens across the width of the room ..."

Read more
Digital Trends’ Top Tech of CES 2023 Awards
Best of CES 2023 Awards Our Top Tech from the Show Feature

Let there be no doubt: CES isn’t just alive in 2023; it’s thriving. Take one glance at the taxi gridlock outside the Las Vegas Convention Center and it’s evident that two quiet COVID years didn’t kill the world’s desire for an overcrowded in-person tech extravaganza -- they just built up a ravenous demand.

From VR to AI, eVTOLs and QD-OLED, the acronyms were flying and fresh technologies populated every corner of the show floor, and even the parking lot. So naturally, we poked, prodded, and tried on everything we could. They weren’t all revolutionary. But they didn’t have to be. We’ve watched enough waves of “game-changing” technologies that never quite arrive to know that sometimes it’s the little tweaks that really count.

Read more
Digital Trends’ Tech For Change CES 2023 Awards
Digital Trends CES 2023 Tech For Change Award Winners Feature

CES is more than just a neon-drenched show-and-tell session for the world’s biggest tech manufacturers. More and more, it’s also a place where companies showcase innovations that could truly make the world a better place — and at CES 2023, this type of tech was on full display. We saw everything from accessibility-minded PS5 controllers to pedal-powered smart desks. But of all the amazing innovations on display this year, these three impressed us the most:

Samsung's Relumino Mode
Across the globe, roughly 300 million people suffer from moderate to severe vision loss, and generally speaking, most TVs don’t take that into account. So in an effort to make television more accessible and enjoyable for those millions of people suffering from impaired vision, Samsung is adding a new picture mode to many of its new TVs.
[CES 2023] Relumino Mode: Innovation for every need | Samsung
Relumino Mode, as it’s called, works by adding a bunch of different visual filters to the picture simultaneously. Outlines of people and objects on screen are highlighted, the contrast and brightness of the overall picture are cranked up, and extra sharpness is applied to everything. The resulting video would likely look strange to people with normal vision, but for folks with low vision, it should look clearer and closer to "normal" than it otherwise would.
Excitingly, since Relumino Mode is ultimately just a clever software trick, this technology could theoretically be pushed out via a software update and installed on millions of existing Samsung TVs -- not just new and recently purchased ones.

Read more