A Brief History of Search & SEO

Tracing the history of SEO is kind of like trying to trace the history of the handshake. We all know it exists, and we know it’s an important part of business. But we don’t spend a ton of time thinking about its origins -- we’re mostly concerned with how we use it day-to-day.

But unlike the handshake, SEO is fairly young, and changes frequently. Quite appropriately, it appears to be a millennial -- its birth is predicted to fall somewhere around 1991.

And in its relatively short life, it’s matured and evolved rather quickly -- just look at how many changes Google’s algorithm alone has gone through.

So where did SEO begin, and how did it become so darn important? Join us, as we step back in time and try to figure this out -- as it turns out, it’s quite a story.

But First, a Look Back at Search Engines

Source: Wayback Machine

The first idea for creating a common archive for all the world’s data came to fruition in 1945. That July, Dr. Vannevar Bush -- then director of the now-defunct Office of Scientific Research and Development -- published a piece in The Atlantic proposing a “collection of data and observations, the extraction of parallel material from the existing record, and the final insertion of new material into the general body of the common record.” In other words, we believe, today’s Google.

Several decades later, in 1990, McGill University student Alan Emtage created Archie, which some say was the very first search engine -- though that remains up for debate, according to research from Bill Slawski, president and founder of SEO by the Sea. However, Archie was what Slawski called the “best way to find information from other servers around the internet at the time,” and is actually still (very primitive) operation.

The next decade saw several pivotal developments, with the more commercial versions of search engines we might recognize today taking shape.

• February 1993: Six Stanford students create Architext, which would later become the search engine Excite. Some, like Search Engine Land (SEL), say that Excite “revolutionized how information was cataloged,” making it easier to find information “by sorting results based on keywords found within content and backend optimization.”
• June 1993: Matthew Gray debuts World Wide Web Wanderer, which later became known as Wandex.
• October 1993: Martijn Koster introduces ALIWEB, which allows site owners to submit their own pages (unbeknownst, sadly, to many site owners).
• December 1993: At least three “bot-fed” search engines exist -- JumpStation, RBSE spider and World Wide Web Worm -- which likely means they were powered by web robots to crawl both servers and site content to produce results.
• 1994: Alta Vista, Infoseek, Lycos, and Yahoo search engines all come to fruition.
• 1996: Google co-founders Larry Page and Sergey Brin begin building a search engine that they initially call BackRub.
• April 1997:AskJeeves is introduced, later becoming Ask.com.
• September 1997: Google.com is registered as a domain name.

It’s worth noting that nearly twelve years later, in June 2009, Microsoft released Bing -- its previous editions were also known as Live Search, Windows Live Search, and MSN Search.

But here’s where SEO itself comes in. As search engines became more mainstream and widely used, site owners started to get wise. As SEO community Moz puts it, “It was discovered that by taking some rather simple actions, search engine results could be manipulated and money could be made from the internet.”

Those results, though, weren’t exactly quality ones. And that, dear readers, is where the SEO story begins.

A Brief History of Search & SEO

The ‘90s

Source: The Daily Dot

With search engines becoming household names and more families becoming connected to the Internet, finding information came with greater ease. The problem, as noted above, was the quality of that information.

While search engine results matched words from user queries, it was usually limited to just that, as an overwhelming amount of site owners took to keyword stuffing -- repeating keywords over and over again in the text -- to improve rankings (for which there was no criteria), drive traffic to their pages and produce attractive numbers for potential advertisers.

There was also a bit of collusion going on. In addition to the keyword stuffing, people were using excessive and “spammy backlinks,” according to SEL, to improve their authorities. Not only were there no ranking criteria at the time -- but by the time search engines fixed algorithms accordingly, there were already new black hat SEO practices taking place that the fixes didn’t address.

But then, two kids at Stanford got an idea.

Source: Stanford InfoLab

When Page and Brin set out to create Google, that was one of the problems they wanted to solve. In 1998, the pair published a paper at Stanford titled “The Anatomy of a Large-Scale Hypertextual Web Search Engine,” where they wrote:

...the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users.”

It was in that same paper that Page and Brin first mentioned PageRank, the technology that Google uses to help rank search results based on quality, and not keywords alone. Some might say that thesis cleared the path for SEO as we know it today.

The Early 2000s

Source: Wayback Machine

The early 2000s saw the beginning of the Google takeover. In the process of making search engine technology less advertising-centric, Google began to provide guidelines for white hat SEO -- the kind that the “good guys” stick to -- to help webmasters rank without any of the common fishy behavior from the 90s.

2000-2002

But according to Moz, the guidelines didn’t yet have an actual impact on ranking, so people didn’t bother following them. That’s partially because PageRank was based on the number of inbound links to a given page -- the more of those, the higher the ranking. But there wasn’t yet a way to measure the authenticity of those links -- for the early part of the 2000s, Marketing Technology Blog says it was still possible to use these backlinking techniques to rank pages that weren't even related to search criteria.

But in 2001, Brin and Page appeared on "Charlie Rose," when the host asked them, "Why does it work so well?" As part of his answer, Brin emphasized that -- at the time -- Google was a search engine and nothing else, and was looking at "the web as a whole, and not just which words occur on each page." It set the tone for some of the initial major algorithm updates that would begin to more closely examine those words. Have a look at the full interview:

Source: Charlie Rose

2003-2004

This approach to the web being about more than just words really began taking shape in November 2003, with the “Florida” update to Google’s algorithm. Enough sites lost their ranking for Search Engine Watch to call the response to Florida a massive “outcry,” but careful to note that many sites benefitted from the change, too. It was the first major instance of sites receiving penalties for things like keyword stuffing, signaling Google’s emphasis on solving for the user first -- mainly with quality content.

In 2004, one of the more primitive versions of Google's voice search existed, in what the New York Times called a half-finished experiment. And while the technology was somewhat infantile at the time -- just check out what the instructions looked like at first -- it was also a signal to the future importance of mobile in SEO. (Stay tuned -- more on that later.)

Source: Wayback Machine

2005: A big year for SEO

One of the biggest years in the search engine world was 2005. That January, Google united with Yahoo and MSN for the Nofollow Attribute, which was created in part to decrease the amount of spammy links and comments on websites, especially blogs. Then, in June, Google debuted personalized search, which used someone’s search and browsing history to make results more relevant.

That November, Google Analytics launched, which is still used today to measure traffic and campaign ROI. Check out its baby photo:

Source: Wayback Machine

2009: SEO shakeups

In 2009, the search engine world saw a bit of a shakeup. Bing premiered that June, with Microsoft aggressively marketing it as the search engine that would produce noticeably better results than Google. But as SEL predicted, it was no “Google-killer,” nor did its advice for optimizing content significantly contrast Google’s. In fact, according to Search Engine Journal, the only noticeable difference was Bing’s tendency to give priority to keywords in URLs, as well as favoring capitalized words and “pages from large sites.”

That same year, in August, Google provided a preview of the Caffeine algorithm change, requesting the public’s help to test the “next-generation infrastructure” that Moz says was “designed to speed crawling, expand the index, and integrate indexation and ranking in nearly real-time.”

Caffeine wasn’t fully introduced until nearly a year later -- when it also improved the search engine’s speed -- but in December of 2009, a tangible real-time search was released, with Google search results including things like tweets and breaking news. It was a move that confirmed SEO wasn’t just for webmasters anymore -- from that moment forward, journalists, web copywriters and even social community managers would have to optimize content for search engines.

Here's Matt Cutts, Google's head of webspam, discussing Caffeine in August 2009:

Source: Wayback Machine // WebProNews

2010-Present

Source: Wayback Machine // Google

When you’re typing in a search query into Google, it’s kind of fun to see what its suggestions are. That’s thanks to the Google Instant technology, which rolled out in September 2010. At first, Moz says, it made SEOs “combust,” until they realized that it didn’t really have any result on ranking.

But Google Instant, along with the evolution of SEO from 2010 on, was just another phase of the search engine’s mission to solve for the user -- despite some controversy along the way around pages whose rankings were actually improved by negative online reviews. The algorithm, Google said, was eventually adjusted to penalize sites using such tactics.

More on Google Instant, circa 2010:

That year also saw a growing importance of social media content in SEO. In December 2010, both Google and Bing added "social signals," which first displayed any written Facebook posts, for example, from your own network that matched your query. But it also began to give PageRank to Twitter profiles that were linked to with some frequency. The importance of Twitter in SEO didn't end there -- stay tuned.

2011: The year of the panda

The trend of punishing sites for unfairly gaming Google’s algorithm would continue. Some of these incidents were more public than others, including one with Overstock.com in 2011. At the time, according to Wall Street Journal, domains ending with .edu generally had a higher authority in Google’s eyes. Overstock used that to its advantage by asking educational institutions to link to its site -- and use keywords like “vacuum cleaners” and “bunk beds” -- offering discounts for students and faculty in return. Those inbound links would improve Overstock’s rankings for queries with such keywords, until Overstock discontinued the practice in 2011 and Google penalizing them soon after.

It was also the year of Panda, which first rolled out that February -- the algorithm update that cracked down on content farms. Those were sites with huge quantities of frequently updated, low-quality content that was written with the sole purpose of driving search engine results. They also tend to have a high ad-to-content ratios, which Panda was trained to sniff out.

Panda itself has undergone several updates -- so many that in its timeline of changes to Google’s algorithm, Moz declined to list any that weren’t major after 2011. Even with that exclusion, the timeline still lists twenty-eight panda updates -- for most of which the impact was difficult to measure -- through July of 2015.

2012: Along came a penguin

In April 2012, Google took what it called “another step to reward high-quality sites” with the first of many Penguin updates -- and, in the process of announcing it, acknowledged Bing’s month-earlier blog post on the changing face of SEO. Penguin targeted sites that more subtly used non-white hat SEO tactics; for example, those with content that might be mostly informative, but was also sprinkled with spammy hyperlinks that had nothing to do with the page’s H1, like in this example:

Source: Google

It's worth noting that 2012 also saw a throwback to Google's original anti-ad-heavy thesis with the "Above The Fold" update, which began to lower the rankings of sites with heavy ad-space above the "fold," or the top half of the page.

Eventually, Google would go beyond targeting spammy content itself. The Payday Loan algorithm update -- which was hinted at in June 2013 and officially rolled out the following May -- actually focused more on queries that were more likely to produce spammy results. Those were typically searches for things like, well, payday loans, and other things that might make your mother blush. Google adjusted its ranking system to help keep spam out of those results, and while it didn’t necessarily impact the SEO efforts of legitimate sites, it displayed efforts to keep search results authentic.

Google goes local

Keeping with the tradition of animal-named algorithm updates, Google released "Pigeon" (dubbed so by SEL) in 2014, which carried quite an impact on local search results. At the time, it seems to have been designed to improve Maps queries, which began to be treated with some of the same technology that was applied to its other search functions, like "Knowledge Graph, spelling correction, synonyms". Local searches were going to become a big deal -- and it will only continue to do so, as you'll see in a bit.

Then, in 2015...

The biggest post-2010 SEO announcement might have been Google’s mobile update of April 2015, when non-mobile-friendly websites would start getting lower rankings. That meant SEO was no longer about keywords and content -- responsive design mattered, too.

Google announced that change in advance, in February 2015, with a mobile-friendly test that allowed webmasters to view potential issues and make changes before the rollout. It wasn’t the last of Google’s mobile updates -- in August 2016, it announced a crackdown on mobile pop-ups.

What’s Next?

It might be hard to believe, but it looks like even more change is on the horizon.

To mobile and beyond

As mobile usage is on the rise -- 51% percent of digital media is consumed that way, versus 42% on desktop -- it makes sense that SEO will continue leaning in that direction.

That's already apparent with Google’s favorability toward a mobile-friendly user experience. We predict that a future wave of SEO will largely pertain to voice search. That has its own complex history and is on the rise -- 20% of Google searches are currently done by voice, as are 25% of Bing’s. And it's compounded by the rise of such voice-powered digital personal assistants, like Amazon’s Alexa.

While there might not be a clear-cut way to optimize for voice searchyet -- largely due to a lack of analytics in that area -- we anticipate that those resources will become available, creating yet another critical pillar of SEO.

Going local

But that brings up the issue of localization in SEO, and optimizing results to be regionally relevant. That’s especially true in the realm of voice search -- Yelp and other business aggregators are used to answer voice queries about what’s nearby, for example. That’s an SEO opportunity for local businesses, by making sure their listings are “comprehensive, accurate and optimized to be referenced” on a third party site.

Getting Social

While the 2009 introduction of Google's real-time search had some social ramifications, social media is becoming a more pivotal piece of SEO strategy. When the search engine began indexing tweets in 2011, for example, it hinted toward a future in which users seek information on social media in the same way that they do via search. In fact, this indexing might be Google's version of future-proofing -- if you can imagine it -- for a time when people no longer use search engines the way we do now.

For example, type in the name of any celebrity -- say, Charlie Rose, whose video we shared earlier. The first page of search results for his name includes his Facebook and Twitter profiles. Plus, check out the biographical sidebar to the right -- there are social icons with links to his various networks there, too. When users search for a person, that's one of the first things they want to see.

Source: Google

In any case, it’s clear why SEO has become a full-time job. Its history will only continue evolving. Executing it well requires a high level of skill, ethics, and upkeep on technology.

But we know that, sometimes, it’s not possible to have a single person dedicated to it, which is why we continue to create the best SEO learning resources we can. Check out some of our favorites:

• How to Increase Your Website SEO Traffic in 30 Days
• HubSpot SEO
• How to Learn SEO: 10 of the Best Resources to Bookmark

What are your favorite pieces of SEO history? Let us know in the comments.