According to a Piper Jaffray study cited by Forbes earlier this week, the investment firm conducted an extensive test of Siri’s accuracy both on a busy Minneapolis street and within a quiet room. The researchers conducting the study asked Siri 800 questions in each location and checked the accuracy of the responses. According to the results, Siri was able to understand 83 percent of the questions on a noisy street and 89 percent of the questions in the quiet room. When providing answers to those questions, Siri was only 62 percent accurate when on the street and 68 percent accurate when providing an answer within the room. Alternatively, Google’s voice assistant provided accurate results approximately 86 percent of the time when asked the same questions.
Based off these results, Piper Jaffray’s Gene Munster gives Apple a letter grade of a “D” and gives Google’s voice assistant a “B+.” However, Munster is optimistic about future improvements with each revision of Apple’s mobile operating system.
In an email to clients, Munster wrote “In order to become a viable mobile search alternative, Siri must match or surpass Google’s accuracy of B+ and move from a grade D to a B or higher. With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google from 60 percent to 48 percent.”
According to Munster, Apple relies on Google significantly for responses supplied by Siri. The specific breakdown of Siri responses includes sixty percent of the information from Google’s search engine, twenty percent from review site Yelp, fourteen percent from search engine Wolfram Alpha, four percent from Yahoo’s search engine and two percent from Wikipedia. Also breaking down Google’s share, Munster stated “Google provides 100 percent of navigation results, 61 percent of information results, 48 percent of commerce results and 42 percent of local results.”
When iOS 6 is released later this year with the new iPhone, Apple plans to add Fandango for movie tickets and times, Rotten Tomatoes for movie reviews, Open Table for restaurant reservations and Yahoo! Sports for scores and player information.
In addition, Apple is transitioning away from Google Maps and Siri will provide walking or driving directions based on Apple’s new maps software. Munster estimates that Apple will be able to reduce the company’s dependence on Google search results from 60 percent down to approximately 48 percent after iOS 6 is released.
Munster provided a variety of examples of the types of questions that Siri would get incorrect. For instance, when Munster or his staff asked “When did the movie Cinderella come out?“, Siri brought up movie theater listings on Yelp. The statement “I want to go to Lake Superior.” caused Siri to provide directions to a local business with Lake Superior in the title. The question “How do I get from Boston to New York?” was impossible to answer as Siri wasn’t able to provide directions for a place that the user wasn’t currently located. According to Munster, the most common error that Siri made was answering a new question with the response from the previous question.