Skip to main content

Forget keywords — this new system lets you search with rudimentary sketches

Could the future of online shopping be as simple as sketching out what you’re looking for and letting a computer figure out the rest? That’s the working theory of two researchers at Queen Mary University of London’s (QMUL) School of Electronic Engineering and Computer Science. They’ve taught a deep learning neural network — an incredibly powerful tool that mimics the way that the human brain works — to recognize hand-drawn sketches and use them to search for real-life products.

The network was “trained” to match sketches to photos based on a data set consisting of around 30,000 sketch-photo comparisons. From these it learned to interpret the way that people depict real objects in hand drawing. Most impressive of all is the fact that the sketches drawn by users don’t even have to be all that detailed — but the more detail users do add, the more accurate the search results become.

Recommended Videos

“I would hope that within six months you’ll be able to log into a shopping site and use this technology.”

Please enable Javascript to view this content

“Sketch is a very flexible tool,” co-developer Yi-Zhe Song, director of QMUL’s SketchX Research Lab, told Digital Trends. “For example, you might see a pair of shoes that you like, but not approve of every detail of them — such as the buckle or the heel. In that case, you can draw the shoe you would like to see — and then let the computer search for the closest possible match.”

But as impressive a computer science project as this is, why would anyone want to spend time drawing pictures instead of typing a quick text-based search in, or snapping a picture as companies like Amazon now allow you to do? Song has an answer.

“Both of those search modalities have problems,” he says. “Text-based search means that you have to try and describe the item you are looking for. This is especially difficult when you want to describe something at length, because retrieval becomes less accurate the more text you type. Photo-based search, on the other hand, lets you take a picture of an item and then find that particular product. It’s very direct, but it is also overly constrained, allowing you to find just one specific product instead of offering other similar items you may also be interested in.”

While Song and his colleague Timothy Hospedales, director of QMUL’s Applied Machine Learning Lab, acknowledge that both of these search methods have their use-cases, and they think the time is right for them to be joined by sketch-based search.

“Today, almost all of us have smartphones, tablets and even smartwatches with touchscreens you can draw on,” Song notes. “This is something that was relatively rare just 10 years ago.”

Song and Hospedales recently showed off their new creation to an audience at the International Conference on Computer Vision and Pattern Recognition in Las Vegas, and are now talking with retailers about incorporating the product into the online marketplace. “I would hope that within six months you’ll be able to log into a shopping site and use this technology,” Hospedales told Digital Trends.

For the sake of easy shopping, we hope so too.

Luke Dormehl
Former Digital Trends Contributor
I'm a UK-based tech writer covering Cool Tech at Digital Trends. I've also written for Fast Company, Wired, the Guardian…
Hyundai Ioniq 5 sets world record for greatest altitude change
hyundai ioniq 5 world record altitude change mk02 detail kv

When the Guinness World Records (GWR) book was launched in 1955, the idea was to compile facts and figures that could finally settle often endless arguments in the U.K.’s many pubs.

It quickly evolved into a yearly compilation of world records, big and small, including last year's largest grilled cheese sandwich in the world.

Read more
Global EV sales expected to rise 30% in 2025, S&P Global says
ev sales up 30 percent 2025 byd sealion 7 1stbanner l

While trade wars, tariffs, and wavering subsidies are very much in the cards for the auto industry in 2025, global sales of electric vehicles (EVs) are still expected to rise substantially next year, according to S&P Global Mobility.

"2025 is shaping up to be ultra-challenging for the auto industry, as key regional demand factors limit demand potential and the new U.S. administration adds fresh uncertainty from day one," says Colin Couchman, executive director of global light vehicle forecasting for S&P Global Mobility.

Read more
Faraday Future could unveil lowest-priced EV yet at CES 2025
Faraday Future FF 91

Given existing tariffs and what’s in store from the Trump administration, you’d be forgiven for thinking the global race toward lower electric vehicle (EV) prices will not reach U.S. shores in 2025.

After all, Chinese manufacturers, who sell the least expensive EVs globally, have shelved plans to enter the U.S. market after 100% tariffs were imposed on China-made EVs in September.

Read more