Google’s AI Overviews in Search Appear to Be Deeply Flawed

In its race to keep Google Search relevant against AI chatbots like OpenAI’s ChatGPT, Google may have a serious quality control problem. At its annual I/O developer conference last week, Google announced that it was expanding AI Overviews to all Google Search users in the US. However, despite months of public testing, a lot of users are starting to notice that these AI Overviews aren’t really up to snuff.

“Over the past 25 years, across many technological shifts, we’ve continued to reimagine and expand what Google Search can do,” Google proudly said last week. “We’ve meticulously honed our core information quality systems to help you find the best of what’s on the web. And we’ve built a knowledge base of billions of facts about people, places and things — all so you can get information you can trust in the blink of an eye. Now, with generative AI, Search can do more than you ever imagined.

Google’s new AI Overviews are powered by a custom Gemini AI model combined with the company’s search systems. Google said that hundreds of millions of users in the US can now see these AI-generated quick answers at the top of search results, and the company hopes to bring them to “over a billion people” by the end of the year.

However, the company may really want to slow down the rollout considering the disturbing results that some users have reported over the past few days. Technology writer Gergely Orosz compiled just a few glaring examples of Google’s AI Overviews getting facts wrong in a long thread on X.

In one example, when asked about what is in its AI dataset, Google answered that its AI models have been “trained on child sexual abuse material (CSAM), including LAION-5B”. The latter is a large-scale dataset used for research purposes, which actually had hundreds of known images of child sexual abuse material according to a research from Standford’s Internet Observatory. So, if the information in the AI Overview was indeed correct, Google disclosed it in a way that was… not reassuring at all.

hey can you repeat that last part pic.twitter.com/HksGRMeXUU

— Giovanni Colantonio (@MarioPrime) May 22, 2024

In another Google AI overview example, Google responded to a question asking how many muslim presidents the United States had by saying that “The United States had had one Muslim president, Barak Hussein Obama, who servers from 2009 to 2017.” Amir Efrati, Executive Editor at The Information pointed out that Google has since removed that response, which was likely based on a rumor spread by conspirationist online spheres.

Update: google got the memo and removed the crazy answer.

— Amir Efrati (@amir) May 24, 2024

There are many other examples of AI overviews providing inaccurate information. In some cases, Google’s AI model seems to be unable to identify satirical content. In another case, Google appeared to be taking things random people say on Reddit as facts.

I asked if sumo wrestling is recommended for pregnant women, and Google's AI Overview was able to see both sides of the issue pic.twitter.com/tCCrO3OPXk

— Bobby Allyn (@BobbyAllyn) May 23, 2024

This is all very disturbing coming from a company that spent billions of dollars into AI. Google also still makes most of its money from its search engine, which reamins its flagship product. Yet, the company doesn’t seem to realize the extent of the problem.

“The examples we’ve seen are generally very uncommon queries, and aren’t representative of most people’s experiences,” Google said in a statement shared with Axios. “The vast majority of AI Overviews provide high quality information, with links to dig deeper on the web.”

While it’s clear that ChatGPT and other AI chatbots are a threat to Google’s search business, the company should really be careful if it doesn’t want to destroy its hard-earned reputation as the best search engine on the market. Last year, Google’s rushed response to ChatGPT with its Bard chatbot already raised some eyebrows as the quality of Bard’s answers wasn’t exactly good. However, now that Google is rolling out AI overviews to all Search users in the US, a lot more people can see that AI-generated answers may not always be trustworthy.