Google Discusses the Future of its Assistant

Posted on September 8, 2017 by Paul Thurrott in Android, Cloud, iOS, Mobile, Smart Home with 36 Comments

Google Discusses the Future of its Assistant

Google quietly hosted a developer show in Europe this week, and one of the best things I’ve seen so far is an exciting update about the future of Google Assistant.

That developer show, called Google Developer Days Europe, was held in Poland on Tuesday and Wednesday, and you can find some videos from the keynotes and sessions on YouTube. Among them is the Day 2 Keynote, in which the firm discusses some of its plans for the improving Google Assistant across devices. (I was tipped off to this information by Android Police.) It’s worth watching.

“Google Assistant is one of the core implementations of an AI-first world,” Google senior engineering director Behshad Bezhadi said during his talk. “We feel that we [Bezhadi leads a team of over 100 engineers building Google Assistant] are part of this next revolution.”

After a short discussion of why AI is such a revolution, Bezhadi argues that AI, in some ways, is making smartphones truly smart, by adding conversational, situational, and intra-device capabilities.

“Conversation is really the core of what we think about when we think about Google Assistant because we think that conversation is the most-used interface human beings have used,” he says. “If we solve the problem of conversational understanding, we can expand it to many use cases.”

Bezhadi then showed some demos of where Google Assistant is at today, using a Pixel smartphone. None of this will be particularly surprising to anyone who has interacted with Google Assistant or any other digital personal assistant: You can have short, meaningless conversations, find out about the weather, discover local attractions, and so on. Google Assistant can also be used to learn about much more specific things: He asked about rides at a particular amusement park, and then asked about the height restrictions at a particular ride at that park.

More intriguing, of course, are the Google Assistant features he demoed that are not yet live. Some of these will launch in the next few months, he said, while others will arrive next year.

“We’re trying to understand more complex natural language,” he said, launching into the following question: “What is the name of the movie where Tom Cruise acts in it and he plays pool, and while he plays pool he dances?” Google Assistant responds, correctly, with some information about the movie “The Color of Money.”

That’s impressive, but Bezhadi also shows how Google Assistant can be enhanced by connecting to other services, either from Google or from third parties.

“Be my Vietnamese translator,” he says to Google Assistant, which says it understands and will translate all of his following sentences to Vietnamese. All he has to do to stop is say, “Stop translation.” This, too, is an impressive demo, and it’s not hard to imagine how this functionality would transform international travel.

Bezhadi goes on to control Google Maps’ Street View (“I would like to be on the top of Eiffel Tower now, can you please bring me there?” as well as a third-party service—“Talk to WebMD”—where he asks about the side-effects of aspirin. He asks implicit questions—“How is my team doing?”—and Assistant uses context about his profile, preferences, and query history (and presumably the time of the year) to answer about the right team.

That latter capability is particularly impressive when you consider that you can teach Assistant about yourself and it will get smarter as you do. This capability is already present in a basic form now, but it will get more capable in an update coming next year by adding natural language learning.

“When the weather is more than 25 degrees [Celsius], I can swim in the lake of Zurich,” he tells Google Assistant. Then, he asks whether he can swim in the lake of Zurich this weekend. “No, you can’t,” Google Assistant replies. “The temperature is less than 25 degrees.”

On a related note, the way that Google Assistant handles context is also going to get more sophisticated. Consider contextual voice recognition: He asks about a particular mountain range. And then he asks about the height of one of the mountains in that range by name. Because the mountain range discussion had happened, Google Assistant understood the context of the next question.

“Contextual voice recognition is going to play a big role,” he said. “Because people always try to continue talking about things which are related, location specific, or contextually relevant.”

In another example, Bezhadi asks about “pictures of Thomas,” but because there is no context, it shows him pictures of Thomas the Tank Engine. So he asks about a sports team roster, and is shown a list of players. So he then asks about Thomas again, and this time he is shown pictures of a player on that team who is named Thomas.

Context is also important for having a continuing conversation. He asks about where the Empire State Building is, and gets an answer. Then asks, “I want to see pictures.” And he is shown pictures of that building. And he asks “how tall is it?” he is given the answer for the Empire State Building. He goes on to ask other questions: “Who built it?” “When?” and “What are the Italian restaurants around there?” and “Call the first one.” It’s a conversation.

Context also works visually (or soon will): Through the integration of Google Lens for visual input, Google Assistant will be able to perform some impressive feats. For example, he points Lens at an apple and then asks, “How many calories?” and Google Assistant tells him how many calories are in an apple. He also did a demo with money in which Assistant converted it to a different currency.

Google is also working on improving voice recognition in noisy environments. With music blaring and the audience roaring, Bezhadi asked questions through his phone, and the Assistant answered correctly. (This one is obviously hard to describe.)

Overall, this was an impressive set of demos. And anyone who bets against Google in this area isn’t seeing the big picture. There’s no question whether Google Assistant wins this war. The only question is which assistant ends up in second place.

 

Tagged with

Join the discussion!

BECOME A THURROTT MEMBER:

Don't have a login but want to join the conversation? Become a Thurrott Premium or Basic User to participate

Register
Comments (39)

39 responses to “Google Discusses the Future of its Assistant”

  1. Avatar

    goodbar

    So which assistant will end up in 2nd place? And is this different from the mobile market where there is room for more than 2 assistants?

    • Avatar

      VancouverNinja

      In reply to goodbar:

      Cortina first, Siri or Alexa ( Due to Cortana integration), Google Assistant.

    • Avatar

      jboman32768

      In reply to goodbar:


      My guess is that it will be:

      • 1 Google
      • 2 Alexa
      • 3 Siri
      • 4 Bixby
      • 5 All others (including Cortana)


      Many will say Amazon has a strong lead with Alexa, but that is only a US viewpoint as the Amazon product only available in US, UK and German markets.  Amazon appear to be moving too slowly where as Google Home is already available in many countries, (like here in Australia), as will Apple HomePod be in December likely beating Echo to the wider market.


      I put Alexa at 2nd because ultimately when released to the world, Alexa will be lower cost and have a better home automation appeal than the HomePod due to having the best/widest compatibility and will have access to the data from your Microsoft Account. I suspect HomePod will have good sound quality and be useful with HomeKit certified devices, but not necessarily have the back end AI to match MS and Amazon. Still a good choice if you are a 100% Apple person.


      Bixby deserves a mention because Samsung will start throwing it in with washing machines/TV's etc so it is likely to be the voice assistant you get by accident.


      Cortana is still not available natively on any devices apart from PC's (no XBOX as Kinect is now gone) and Harmon & Kardon (a Samsung company) I would expect will now be more likely to release a Bixby speaker instead of the Cortana version.


      Still the Google offering seems to be quite good and they appear to be showing some progress.



  2. Avatar

    VancouverNinja

    These examples are not impressive at all. Most of these capabilities have been in development in other companies for years. The killer app today for AI is music and home control.


    Microsoft has the best real time translation services in the world today and I doubt Google will be able to exceed what MS is achieving with translation.


    All in all I can see why this announcement was kept low key, nothing big about it at all.

    • Avatar

      Tony Barrett

      In reply to VancouverNinja:

      Whatever MS has doesn't count for squat if they can't win over the consumer and get there products out there. Cortana on the PC is next to useless (and pointless). It's all about mobile or in-home digital assistants you can interact with and control things around your home. Google realised this years ago, but MS just let all this pass them buy... again. MS are paying a dear price for sitting on their Windows cash cow for too long, and are now fighting just to stay relevant. In March this year, Android overtook all versions of Windows as the most widely used platform on the planet, and it's almost certain Microsoft's slide will continue. Paul is just reporting on what's relevant in today's computing marketplace.

  3. Avatar

    Kadren

    Google is seriously up to something in terms of a radical strategy shift. I think Assistant wil lbe an important part of Google going ll mobile. I mean, not just the internet stuff, but serious integration with mobile devices, even at the expence of alternativres. I noticed that they are making little, seemingly unconnected tweaks like Gmail link handling. And now there's the HTC deal.

    I know it's offtopic to this article, but the Assistant's future is probably part of that something they're doing.

  4. Avatar

    nbplopes

    Personally I think we are 75 to 100 years far from a conversational UI. Everything looks highly scripted, not based on native reasoning. Even today in my practice at least, it has not detached itself from command based AI in any context. Imalso feel it will need other hardware, maybe quantum computing, That will take time. Mind you even this is a big step, but it's not has big has tech companies are making it. For instance, map routing is also AI. image recognition is also AI. Now conversational AI its in an entirely different level. They are after your data so that when it's feasible the data will be there. But it will not be any time soon.


    For me, even without conversational AI, there is a lot to squeeze from the "order"/"response"/"basic context". We need to go beyond this assistant paradigm an let user interact with "brands"/apps directly with voice. As I've written in another post.


    "Hey Thurrot, what is hot today on Windows 10"

    "Hey Thurrot, when will Windows 10 fall update be released?"


    "This information is provide by Cashfly" :) Ops that's Windows Weekly :)

  5. Avatar

    SvenJ

    What would be neat would be a 'conversational' UI that doesn't require speaking to your phone. Just as keyboards anticipate your next word/phrase, maybe AI could anticipate follow on questions and present them to be selected. If you search for a sports team for example, maybe it suggests team stats, player stats, or schedule as a follow up. After selection there could be on screen options for narrowing down the query.

    Sometimes 'yelling' at your phone is just not appropriate.

  6. Avatar

    Jules Wombat

    The Google Keynote was interesting and fairly impressive, but not exactly inspiring, just evolutionary steps across a collection of a few different AI services.

    To be fair, Microsoft is working on all this as well with its Chatbots, LUIS services and Cortana. Not yet as good as Google, and will probably never be as good. Its a Google first, Microsoft second race in AI back end services (noting that both Siri and Alexa are reliant upon Microsoft: Bing centric backend services)

    Google will obviously win the AI service delivery race because it is a global focused company, unlike Microsofts US centric aptitude. Nice to see in the keynote Google emphasising the importance of Software Development, and how critical European Software development is towards Google. Microsoft used to be the leading company for supporting Software developers, it is now firmly Google who embraces the software development community.

  7. Avatar

    Bats

    AI is going to be super important and from what I see here, in addition to what I am personally experiencing Google has the upper hand thanks to the Google Assistant. I have dictated Google Docs documents with a lot of success. It just works! However, Google's Voice Recognition works great for me in the car. LOL. I can simply text without typing, while I am driving. That's just one of the great features with Android. You can do anything with it, without even touching your phone.


    Without question the Google is the best, by far, Digital Assistant around. I don't know about Siri, but she is no match for Google Assistant. As for Cortana....forget about it. It's a little better than Siri, but completely stupid. I have Google Home and Cortana (via PC) open all the time and I trigger with hotwords. The Google Assistant always gets my question and tells me the answer and it's usually about weather, facts about actors (because I watch tv), and other things. Cortana always hears me wrong. Like a couple weeks ago, I asked who the actor was who Rhaegar Targarian on Game of Thrones. Google Assistant gave me the name of the actor and some little story about him. I asked Cortana and she gave me the website for Raymour and Flanigan furniture store.

    • Avatar

      SvenJ

      In reply to Bats: Been able to initiate and respond to texts and make calls hands-free with my Windows phones for years. Not selling WP, just noting the capability has been around for them for some time. MS just has no clue how to market. I had augmented reality in City Lens, always on screens, off-line mapping/navigation, and other stuff slowly coming to Android. It's too bad no-one knew.


  8. Avatar

    Angusmatheson

    natural language digital assistants are the next computing platform. Apple saw this first and bought Siri. However, in an interesting stumble...released it too early, and lost the key talent (Not unlike Microsoft buying Danger and then losing Andy Rubin). By blowing their lead, they let everyone be in the running. Google has all the pieces - mobile, search, and home speaker. What is interesting is what does Microsoft do - they have search, but nether of the others. Do they push Cortana? Do they sell Bing (search) to Apple and others who are competing with Google? Can they do both? I agree without either a mobile play or a credible home appliance (I don't think regular people are going to out PCs in their kitchens and living rooms) I can't see Cortana being used much. The Echo has had no competition for 2 years...I wonder how it will sell now that Google, Apple, and third parties sell a connected voice controlled speakers (even a Cortana one!). Unlike previous platform wars this really seems to me wide open. And maybe room for a lot of them (I order socks from Alexa, ask questions from the internet from Google Assistant, play my music and make FaceTime calls from Siri, and Cortana helps me write an email or tell me my appointments?). Or like the other platform wars will it be an all or nothing battle where you are fully committed to one, and just a few will survive?

  9. Avatar

    Mark from CO

    Paul:


    It gets depressing reading one article after another where Microsoft's competition is seen to be lapping and lapping Microsoft, in their great race of technology markets. Here we see a voice assistant service that has a viable strategy attached. Any bets the execution will be there as well?


    I remember, not so long ago, Microsoft having translation demos that wowed everyone. Arguably comparable to, if not in front of Google and other competitors in both translation services and voice assistants. Ahhh, what a few years will do. Now, Cortana is not even in second place.


    Oh, by the way, the article is an example of real life AI applications. What really makes them useful is that they are delivered through a device that real customers use. Oh, yeah... Microsoft gave up its device a couple of years ago. Of course, one will counter that Microsoft has a world class AI effort underway. But I wonder how its AI service(s) gets delivered to a sufficient number of customers (by definition the number has to be very large) without the devices/platforms to make the whole the 'virtuous cycle' viable?


    What has happened in the consumer market the last 3 years has been a disaster for Microsoft. This has been Satya Nadella's Microsoft. I think it is past time to start asking some very hard questions about his leadership.


    Mark from CO

    • Avatar

      arknu

      In reply to Mark from CO:

      Yes, Satya Nadella has been an unmitigated disaster. He comes from the server/cloud side and it shows: He only cares about Azure. But the thing is, there are no longer separate consumer and enterprise markets. People want to use the same services and platforms in both places. Losing your place in the consumer market means losing the enterprise as well in the long term.


      Just as Microsoft devices were actually gaining some traction and with Windows 10 finally being a unified platform (albeit quite a few years late), Nadella decided to pull out. It took the Xbox 10 years to become profitable, Microsoft had the resources to succeed in devices as well, but apparently no longer the will.


      And Microsoft needs to speed up. It is no use showing amazing demos and not delivering a product. HoloLens was amazing, Cortana looks good, but where are the products? Both Apple and Google have overtaken Microsoft.

      Also, Microsoft needs to pay attention to the world outside the US. Too many Microsoft products are either crippled or simply unavailable outside the US. Windows Phone/Mobile was actually doing very well in Europe, but Microsoft apparently never noticed, since they only care about the US.

      • Avatar

        navarac

        In reply to arknu:

        "Also, Microsoft needs to pay attention to the world outside the US. Too many Microsoft products are either crippled or simply unavailable outside the US. Windows Phone/Mobile was actually doing very well in Europe, but Microsoft apparently never noticed, since they only care about the US."

        Very true.

      • Avatar

        BrickPrinter

        I agree with you mostly. But with Hololens I think perhaps I beg to differ. Some of the "enterprise" applications being developed for are going to revolutionize manufacturing, medicine, technical service, design, and education. They seem to have a real lead in those kind of applications. And they have the back end infrastructure to make it happen. Whether or not its little brothers will succeed against the competition remains to be seen. But in really sophisticated things, I think Hololens will be around for a long, long time. just for example

        http://www.maritime-executive.com/article/mixed-reality-technology-readied-for-offshore-vessels

  10. Avatar

    m_p_w_84

    they must have more than 100 people

    working on this

  11. Avatar

    arknu

    Another question is which one gets to the world outside the US first. All these features are nice, but they are worthless as long as they are only available in the US.

  12. Avatar

    Stooks

    Paul you need a "Google" tab at the top of you site. Only question is, does it come before or after "Windows 10" ??

    • Avatar

      Angusmatheson

      In reply to Stooks:

      personal computing is at an inflection point. Personal computing, until 10 years ago, has essentially been about PC running Microsoft Windows. However, since the iPhone was released and the rise of the android smart phone, the world of of personal computing has changed dramarically. There are still people like me, and I suspect Paul, and I suspect you as well who spend their days working on PCs. However there are billions more who "compute" on phones, tablets, home appliances that have nothing to do with Microsoft. Those platforms do many things worse than Windows PCs, but they also do some things much better. I, for one, am delighted that Paul is looking at personal computing - in the PC, but also in these new ways too. Microsft's attempts to transition into this new age makes it by far the most interesting company in tech.

  13. Avatar

    redstar92

    OMG watching Paul drool over Google is getting really old. Almost like he stopped getting kick backs from MSFT and is now getting funded by mountain view. I am surprised that MSFT gives him any insight into their work anymore. I don't remember the last time anything positive about them has come up on this site and was written by Paul.

  14. Avatar

    Darmok N Jalad

    "And anyone who bets against Google in this area isn’t seeing the big picture."

    Oh, I dunno. Google is so good at killing products on the vine that I wouldn't put anything past them. Look no further than the recent news of Google Drive being discontinued to become 2 new separate apps. They don't care what you want, they would rather control what you get for their benefit.

    • Avatar

      Stooks

      In reply to Darmok N Jalad:

      Google is one serious government legislation away from huge earning stumble. With something like 89% of their revenue coming from ads, which come from information gathered from their free apps it could be a company killer if some serious privacy legislation comes down.


      The public is growing increasingly more concerned about privacy. I think we are going to see some serious regulation in the US that will impact Google and others about what is collected and stored.

    • Avatar

      PeteB

      In reply to Darmok N Jalad:

      Except Google drive is not being discontinued . All theyre doing is adding placeholders, something Microsoft has been promising to bring back for years and still can't manage.

      You'll have to find a better example.

      Anyway Paul is absolutely right. Google Assistant is their golden goose, and it blows away everything else in terms of providing expected results. Alexa and Siri are okay, not great. Cortana is dead last - a laughing stock that doesn't work in most countries outside the U.S., gets ignored by desktop users and returns head scratchingly bad results.

      • Avatar

        SvenJ

        In reply to PeteB: I have 'placeholders' back and they are better than they were. Insider build. Pretty sure these happen with the Fall/Autumn/Creators Update Part II. (The better part is iconography which identifies the state of any file. Something I think they could have added initially to mitigate the confusion some people had.)


      • Avatar

        Darmok N Jalad

        In reply to PeteB:

        Not Drive the service, Drive the apps for Mac and PC. And what is replacing that are 2 apps, "File Stream" and "Backup and Sync." Each app has a slightly different purpose, and you can only use one of them at a time. You can't install both at the same time. You get until March 12 to make your choice, as that is when Drive the app goes dead on PC and Mac. Why the change? To push File Stream on enterprise.

        And if you want me to provide "better" examples, look no further than the messenger services: Messenger, Allo, Duo, Hangouts, etc. Then there's Talk, Reader, Picasa, Panoramio, Glass, the craziness of G+, Buzz, Nexus Q. Not a complete list.

        I'm not saying Google doesn't have a superior assistant service, but what Google does with anything they produce can often be befuddling, and often very contrary to what its own users would prefer.

      • Avatar

        david.thunderbird

        In reply to PeteB:OK now I have to agree and that pains me no end.


  15. Avatar

    Eduaam

    RIP iPhone RIP SIRI :V

Leave a Reply