Generative AI isn’t just for creating marketing content. It’s deeply disrupting online search.
This shakeup has led people to some drastic assumptions:
Are these assumptions true? If so, are they helpful and predictive of larger future marketing trends?
Rather than attempting to predict the future, I ran an experiment to test the current impact of AI on online search results.
For the longest time, the algorithms behind Google's ranking factors were a mystery.
Much of that mystery was uncovered with the massive Google Search algorithm leak in May 2024. AI presents a newer and more complex mystery for online marketers.
AI algorithms are complex black boxes even the developers who design them don’t fully understand (which strangely might increase trust).
We’re all wondering countless questions like:
Answers to these questions will hopefully become clearer over time. In the meantime, I set out to conduct an experiment to test the breadth and quality of search using the most popular available tools.
My goal was to objectively collect data across a wide spectrum of tools and compare the results of traditional search engines to AI chatbots, voice assistants, and popular social media channels.
Here was my hypothesis:
"Search engines will still better at helping find basic information, while AI tools will be more helpful with processing complicated queries, such as analyzing complex opinions and performing specific tasks."
As for social media, I’m still unsure how or why people use them for searches, so I didn’t expect those platforms to perform well by comparison.
(Wanna skip the process details? Jump to the key results and observations here.)
I asked 26 questions on 20 different platforms for over 500 individual queries.
Each query fell into one of these six categories:
As I queried, I captured the responses verbatim into a spreadsheet, noted the sources they cited, calculated some statistics (like word count and response time), and added my own observations about the quality or format of each response.
For consistency, I asked the same 26 questions of each platform, knowing that certain types of platforms were better suited than others to give certain answers.
Sometimes, chatbots or voice assistants didn’t know how to respond, but that’s part of what made this experiment valuable.
While there are certainly examples of hallucinations and mistakes (honestly, I tried to trigger some), these tools have become pretty reliable.
Both had limitations because there were some questions AI tools didn’t know how to answer or wouldn’t commit to. However, this also shows an awareness of where their information was insufficient or where it wasn’t wise to speculate.
Part of that is likely because Bing powers Yahoo search, but also because their algorithms have been optimized to the point of similarity.
The AI chatbots also responded similarly to several of the queries. This may be because they’re trained on similar data sets, but there’s no way to tell.
They were decent at summarizing information and mostly varied in their lengths and formats of responses.
The AI chatbots were surprisingly good at forming arguments for opinion queries, but all stopped short of making a final decision.
They preferred to summarize information, which wasn’t as helpful as a decent article that might be found on a search engine.
Regarding specific actions (like “tell me a joke”), AI bots and voice assistants were better than search engines. They were more direct and took action more like a person would.
However, they were woefully out of their depth with more specific information (e.g., a local business or individual person), which is where search engines can still be helpful.
There’s plenty of talk about how younger generations are abandoning traditional search engines in favor of social media platforms.
Before this experiment, I didn’t understand that; and after the experiment, I still don’t get it.
Social media just doesn’t seem helpful for answering questions typically sent to Google.
Millennials like me grew up using search engines, so we’ve adapted our queries to that format.
Younger users are more likely to be adapting how they search based on the platforms they use. With algorithms predicting what content we prefer, they may decrease the desire to search altogether.
YouTube and TikTok certainly had plenty of results for each search. Some of their video results were relevant, but few answered the specific question.
I didn’t bother testing out searches on Facebook or Instagram because they proved even less useful. The exception was Quora, which was built to answer people’s questions.
What does this all mean for you as a marketer or business owner? What is the short-term takeaway for you to remain relevant in online searches? Here are a few final thoughts: