Google Assistant Takes the Crown, Cortana Sits Last on Smart Speaker IQ Test

Posted on December 21, 2018 by Mehedi Hassan in Amazon, Apple, Google, Microsoft with 15 Comments

Analytics firm Loop Ventures published the annual Smart Speaker IQ Test this week. The research firm put the top smart assistants and speakers head-to-head as part of the new test, grading them on a wide range of queries and commands.

The company asked Google Assistant, Amazon Alexa, Siri, and Cortana the same 800 questions. These questions were categorized by five different topics: Local, Commerce, Navigation, Information, and Command. The personal assistants were tested on their respective smart speakers — that includes the Google Home, Amazon Echo, Apple HomePod, and Harman Kardon Invoke (Cortana).

Google Assistant beat the other assistants in almost all these categories, taking the first place in the test. The assistant answered 87.9% questions correctly, understanding 100% of all the 800 queries. Siri came in second, answering 74.6% of the queries correctly and understanding 99.6% of the queries. Alexa came in third, answering 72.5% of the queries correctly and understanding 99% of the queries.

And then there’s Cortana, coming in last with 63.4% accuracy. Cortana actually understood more queries accurately than Alexa, only getting 5 queries wrong as opposed to Alexa’s 8 misunderstood queries.

In terms of the different categories, Google Assistant took the crown in everything except the Command category where Siri on the HomePod took the crown, which is kind of interesting. What’s even more interesting is that Google Assistant and Siri actually beat Amazon’s Alexa on the Commerce category, which is a big surprise considering the size of Amazon.com.

Overall though, all of these assistants have improved over the last year. Cortana only had a 56% accuracy a year ago, so the new 63.4% is quite an achievement, even though it’s still at the last position. In fact, Siri took a huge step forward over the last year — back in February 2018, Siri was sitting at 52%, and it’s now at a whopping 74.6%, taking over both Alexa and Cortana.

Tagged with , , , ,

Join the discussion!

BECOME A THURROTT MEMBER:

Don't have a login but want to join the conversation? Become a Thurrott Premium or Basic User to participate

Register
Comments (15)

15 responses to “Google Assistant Takes the Crown, Cortana Sits Last on Smart Speaker IQ Test”

  1. jules_wombat

    To be fair to Cortana, there was no Test to check on upcoming appointments in Outlook. I think Cortana is pretty good at looking in MS Outlook. Have a test on that.

  2. JeffFerguson

    This matches my informal testing at home. We have a Google Home Mini and a Sonos One with Alexa and have, at times, asked both devices the same question. Google has been giving me better answers in just about all cases.


    I only wish that Google would play more nicely with Microsoft properties such as my Outlook calendar. It's certainly frustrating to have these islands of information with barriers that prevent cross-device and cross-property adoption.

    • jchampeau

      In reply to JeffFerguson:

      Agreed. I was going to order a Google Home Hub on Black Friday but held off because it won't tell me what's on my (Office 365) calendar when I ask. I do this most mornings when I'm deciding what to wear that day and would hate to give up that nicety.

  3. dcdevito

    This is inevitable. Google cannot be matched in this domain.

  4. Daninbusiness

    What, no Bixby???


    /ducks/

  5. Orin

    In reply to JeffFerguson and jchampeau in the Premium comments, this is annoying. But I've worked around it with Microsoft Flow.


    I followed this guide to setup two flows that keep my Gmail calendar and my Outlook.com calendar synced (I really only use my Outlook.com calendar, but I want Google Home to tell me what's on my calendar).


    https://support.microsoft.com/en-us/help/4132258/synchronizing-events-between-your-outlook-calendar-and-google-calendar


    This works with Office 365 accounts too.


    Be sure to read the limitations in the additional information section. If I create a calendar appointment using Google Home, and then edit it using Outlook, Flow will not sync the changes from Outlook back to Gmail. This stinks, but isn't a deal breaker for me since I really only use Google Home to tell me my appointments and create them in Outlook.

  6. richfrantz

    I imagine Cortana would be better if MS had a mobile platform still.

  7. Brazbit

    I find this baffling. It must be significantly more accurate than Android Auto. Auto is a constant source of frustration for me. Getting the dang thing to activate at all often take 5-6 tries. It can't do 9 out of 10 tasks I ask it to without me having to reword my request several times and even then it often misinterprets something and a song request ends up as a navigation route to Albuquerque or some other random destination.


    There are two things that it is flawless at responding to. I have gotten frustrated and told it that it is useless to which it replied that it is really quite capable once you get to know it and once when I told it to [bleep] off out of frustration after five to seven minutes of trying to get it to play a song, which I had no problem the previous day getting it to play, it said it was sorry that we seemed to be having troubles communicating and to try again at another time.


    With performance like that I have never had any desire to try the assistant in my home.

  8. SvenJ

    I imagine, like human IQ tests, these are skewed by the expectations of the people writing the test. I have access to Amazon, Google, Siri, and Cortana (PC & Invoke). I find Alexa does way better at home control, than Siri or Google. Google answers random stuff better. Cortana works better integrating within my PC, and does a better job at that on my PC than on the Invoke (they aren't the same).

  9. Andi

    Something to keep in mind. Gene Munster is behind this "analytics" firm. He was a notorious Apple evangelist in his previous life so to speak. Take this test with some shovels of salt.

  10. red.radar

    Wow... sort of surprised Siri did so well


    guess you can’t believe everything you read.

  11. Awhispersecho

    Google assistant on my phone is an absolute nightmare. Over half the time is simply says it can't reach Google at the moment. If I try to get it to make a call, It will forget who my contacts are. And that's if it responds at all, half the time it won't respond or it will actually go in to active mode but won't reflect that instead it will do it underneath the home screen or any other open screen so I can't tell if it's listening or not. It's a piece of shit and is literally going to cause me to stroke out.


    I'm sorry, but the experience on WP 3 years ago with Cortana was miles ahead of this. It's 1 of only a couple things Windows Phone did really well but for me it made all the difference in the world. The experience with Google assistant on Android is complete torture for me.

Leave a Reply