Trying to perform keyword research to inform your SEO strategy is tough these days. Why? Because of that huge chunk of organic search traffic swept under the “(Not Provided)” carpet in your analytics.
And it’s getting crowded under that carpet! When Google rolled out encrypted search (more on this later, if you need a brushup on the history of Google encrypted search) a Google spokesperson claimed this change would affect “a single-digit percentage of all Google searches.” But in the past few months, we’ve seen over 50% of hubspot.com’s organic search traffic fall into this bin. That’s a ton of missing insight into what keywords are driving organic traffic.
A lot of SEO has traditionally focused around keywords: choosing keywords, creating content for keywords, building links for keywords, ranking for keywords, and tracking traffic from keywords. But as encrypted search rolls out to more and more internet users, it’s time to start re-thinking how we optimize and track our SEO efforts. Well, there's no time like the present, right? So let's talk about the changing SEO landscape as it relates to keywords, well, right now.
What's Actually Going On With Encrypted Search
For the uninitiated, here’s a brief primer on what you need to know about encrypted search. I’ll explain why it’s not a deliberate choice by search engines to take away our marketing analytics, but rather a side effect of using more secure technologies.
HTTP and the Misspelled Header
Whenever you click a link, visit a bookmark, or generally browse the web, the web browser that you’re using automatically sends certain information to every page you visit. In nerd-speak, these are known as “Headers,” and there are all sorts of different Headers that can be sent by your browser. Some common Headers you might have heard of include “Cookie,” “User Agent,” and “Referer.” (Fun fact: the word “referrer” was accidentally misspelled in the original HTTP specification, and the misspelling has been adopted as the official title for this header for the sake of consistency).
Whenever you navigate from page A to page B, your browser usually sends along the “Referer” header to page B, telling it the full URL of page A. That sentence is worth re-reading, because it’s important.
Re-read it? Cool.
This is the core foundation that underlies modern web analytics. It lets site owners know that, hey, the visitor who just landed on your Pricing page came from twitter.com. Or, hey, 21% of the people who visit your landing page also visit your Thank You page. By passing along the referrer’s information, site owners can see where their visitors are coming from, and where they’re going.
Using Referrers to Track Regular Web Traffic
So this brings us to organic search. In the past, when a visitor clicked on a link in the search results and visited your site, your site would see the full URL from the search engine results page from whence that visitor came. Your analytics software would see that the referring URL contains “google” or “bing” -- and no paid search tracking tokens -- and thus labeled that visit as “organic search.” It would then parse out the keywords from the referring URL to see what they searched for.
The whole process might look something like this:
1) Visitor conducts search on www.google.com for “red shoes”
2) Visitor is taken to http://www.google.com/search?q=red+shoes
3) Visitor clicks on an organic result and reaches your site
4) When the visitor’s browser requests your page, it passes along the full URL of the Google search engine results page as the “Referer” header.
5) Your site’s analytics parses that referrer URL to deduce that: 1) The visitor came from organic search, and 2) The visitor was searching for "red shoes."
Secure HTTP (HTTPS) Throws a Wrench in the Mix
The reason we lose information with encrypted search has to do with how HTTPS works, not anything Google is doing (note the “S” at the end, indicating “secure HTTP”). When browsing a site over HTTPS, your browser strips out the “Referer” header that’s the lynchpin of the entire analytics process.
If you go from page A to page B, but page A is encrypted (uses HTTPS instead of regular old HTTP), then the browser only sends the domain name -- https://www.google.com -- as the referrer, instead of the full URL. As a result, your analytics has no way of knowing what keywords the visitor used to find your site.
Not If, But When
The day when the overwhelming majority of organic search traffic comes through as encrypted is coming soon. The major search engines started slowly rolling out encryption two years ago, and those experiments were largely deemed a success; as a result, we’re seeing wider rollouts of the technology. And while it may annoy marketers, encrypted search offers a number of benefits to the people that use search engines every day -- yes, that means you and I -- including increased privacy and security.
When it was first announced, encrypted search was only offered to logged-in Google users. But over the past few years, encrypted search has become the default type of search in all sorts of different places, from Firefox 14, to Google Chrome, and even Safari in iOS 6. Non-Google users can also now use encrypted search by visiting encrypted.google.com to perform their searches. In fact, it wouldn’t surprise me, personally, to see regular ol' www.google.com automatically redirect visitors to a secure version sometime this year. As the days of more and more encrypted search options inches closer, tracking individual keywords and phrases will become less and less useful -- and frankly, just less and less possible.
As I mentioned earlier, at HubSpot, about 55% of the organic search we get each month is encrypted, and we’ve seen that percentage steadily rising by about 4 percentage points each month. Tracking how individual keywords perform for us is quickly losing value.
So, Where Does This Leave Marketers?
As HTTPS becomes increasingly common, we’ll see the value of keyword research drop off even further. Optimizing your website will be less about tracking keywords and rankings, and more about capturing visitors once they've landed on your website.
At HubSpot, the SEO team has recently renamed ourselves simply the "Optimization" team, electing to drop the "Search Engine" part from our name. We're recognizing that optimization is about more than just search engine rankings, and requires a more holistic view of the top of the marketing funnel, including where people are landing on your website, and how easily they're converting. That means instead of spending our day doing keyword research, and optimization around the results of the research, we're doing things like:
- Identifying and promoting our highest converting offers
- Breathing new life into classic, high-performing content that didn't get the love it deserved when it was first published
- Identifying low- and high-converting landing pages, seeing how they align with the rest of our marketing strategies, and ensuring they're being promoted in the right ways to help them perform their best
All of these things help us optimize the experience for our site visitors -- giving them fresh, relevant content that's aligned with what they're searching for -- which then translates into more search engine traffic.
Keyword research is still important in order to come up with high-value terms to create content around, but don't expect to track the results the way you used to for much longer. SEO is changing. Is your organization keeping up?
How have you been impacted by SSL encryption? Has your SEO strategy changed as a result?
Image credit: woodleywonderworks
Inbound Marketing Blog Feed
John Rode 4:46 PM on January 17, 2013
I wonder is this means the keyword research tools will still be useful. I'm thinking mostly of the Google Keyword Tool. Will we still know volumes and general levels of competition?
Tatiana 5:46 PM on January 17, 2013
Hartley, you've wrote that you are "Breathing new life into classic, high-performing content that didn't get the love it deserved when it was first published". Does that mean that you are going to redo the already created and posted on the site content that didn't get the deserved attention?
Rebecca Haden 6:48 PM on January 17, 2013
Very interesting article! I think it's worth noting that the Queries report at Google Analytics gives you lots of keyword data. You just have to have Webmaster Tools integrated with Analytics to get to that report.
Mick 7:01 PM on January 17, 2013
Keywords will still be what drives search. Just because you might not be able to track an individual keyword on a specific page doesn't mean that you can't still optimize that page. Google will still tell you what keywords are being searched. It's therefore logical to assume that these keywords are still driving the traffic to your page, what else would be?
You don't need to see it to know that it's there.
Yeremi 7:04 PM on January 17, 2013
With the degree of changes being introduced into search by Google, I would not be surprised if their own algorithm gets too complex for them. I have sought to position my jobs ite rank for the keyword Jobs in Nigeria, but funny enough, all efforts prove to be with sparse results...
Yeremi 7:06 PM on January 17, 2013
With the degree of changes being introduced into search by Google, I would not be surprised if their own algorithm gets too complex for them.
I have sought to position my jobs site rank for the keyword Jobs in Nigeria, but funny enough, all efforts prove to be with sparse results...
Michael 7:12 PM on January 17, 2013
I couldn't agree more with Mick - Keywords will always drive search. Social Media is also becoming increasingly search-centric.
Now Conversion, that's always been the challenge!
Guido E. Zecckine III 7:13 PM on January 17, 2013
Great read. Analytics is still a major tool for marketers, even with SSL encryption. You can still track where visitors go on your own sites, unless, like stated, the page is secure.
Having a good marketing funnel in place, and calls to action, along with great graphics / videos, and relevant content that converts is the winning formula I think.
You should never really design JUST for the search engines anyway. I design for the end user, and it works itself out in the wash somehow.
Google isn't the only game in town! It does help though, knowing what keywords sent the visiting traffic to your site.
Wouldn't it be nice if the referring site was ALWAYS one of your own sites!?
Thanks for the great read, I enjoyed it.
P/S... where did I come from? Do you know? :)
Richard Robbins 7:44 PM on January 17, 2013
I have seen the % of referral traffic from (not provided) nearly double in the past year alone. I have been re-thinking how to determine which phrases to target. With search engine personalization, this become even harder.
The last two years of Internet marketing have brought bigger changes than I've seen in my past twelve years of marketing websites. It's not a bad thing, though. It keeps us on our toes, always honing our marketing skills.
Alex 9:15 PM on January 17, 2013
I couldn't disagree more. As others have mentioned above me, keywords are still what drives that search traffic to your website, and less insight into what drove traffic doesn't mean it is not still driving traffic. On-page keyword optimizations will still be responsible for high rankings which will in turn drive traffic.
Google will still provide monthly search volumes, and not provided doesn't apply to paid search, where you can still get insights that will drive organic strategy.
Additionally, there are many ways to drill down, if you have a proper keyword mapping strategy, you can understand which landing pages drove the most not-provided search traffic, in which case you'll know which keyword themes may have played a role in organic traffic.
The not provided issue is a tough one, but giving up on keyword research would be a huge mistake.
Nina 9:57 PM on January 17, 2013
Thanks for this, after all said and done, "breathing new life into your old content" is to reinterate the "keywords" in those write ups, with hope that the keywords will drive traffic to your site
Wsi Digimedia Marketing 12:29 AM on January 18, 2013
It is not free but you can do a keyword research using adwords (PPC) and testing keywords... What do you tjhink?
Christina 3:24 AM on January 18, 2013
For a number of years I have not depended on keyword tools to find the best words to use. I have other methods of choosing keywords. I only use the tools to confirm I am moving in the right direction.
Uncle Demotivator 3:37 AM on January 18, 2013
You've lost me here - check out how AdWords still get this data. Unless You already know it and You're just doing PR for Google.
Colin 4:13 AM on January 18, 2013
Great post...informative and thought-provoking. Well, however adWords gets referral data, Google's in the business of making search better for their users. I get the sense they don't really want marketers and SEO's to have a ton of precise data.
Igor Mateski 5:45 AM on January 18, 2013
Actually, the (not provided) mystery can be solved by linking up Analytics with Webmaster tools, and then use the Webmaster Tools to look through the Search Queries. It can be cumbersome and there will be some intuitive guesswork, but in Analytics you can list the Keywords with corresponding Landing Pages as secondary dimension, and compare them with Webmaster search queries list. So the (not provided) isn't a total black box.
Mike Campolattano 9:07 AM on January 18, 2013
For our team "Optimization" has become more about creating a great journey for the visitor and telling a better story and ranking is more of a byproduct of those efforts. So instead of "I want to rank for toys" it's "Hey we rank for toys..hooray". We have changed our reporting models so that we are focused on more true core business metrics and ROI for our clients, and have been weaning them off of the whole "I have to Rank" idea. Because in the end if you tell me you rank #1 for a specific term my first question is "To who and where?" Because search is tailored anyway now for the individual, so you may not be number #1 to me and my location. I am excited about where search is going.
Aidan Lawlor 9:42 AM on January 18, 2013
I guess it gets business's to start thinking of the user first and foremost and what keywords those users are using when searching. Definitely a positive move for users. For marketers, no.
Nate Goodman 9:51 AM on January 18, 2013
This definitely throws a wrench into the works. It will impact all the marketing automation software products out there that offer tracking of paid search campaigns as well.
Hartley Brody 11:19 AM on January 18, 2013
Hey guys,
Love the discussion that's going on. Wanted to address a few points that some people have brought up:
1. Keyword research is still a good way to source content and make sure you're talking about things using the right words. Free tools like Google's Keyword tool are great for this, and will hopefully continue to be effective. I was referring to the fact that it's getting harder to track the keywords that are sending traffic to your website.
2. Some of you pointed out that using AdWords or other paid forms of search still allows you to see the keywords that are sending traffic. That's because, for paid results, Google first redirects the searcher through a non-secure page that has the search query in the URL, and then *that* page sends them to the final landing page. Next time you click on a paid search ad, watch the URL in your browser change twice, and you'll see the interstitial page very briefly.
3. It's great to see people talking about "user experience" in the context of SEO, and how the old 'rank #1' goal is largely meaningless these days. Whenever algorithms evolve, we can always squabble over the details and what it means for SEO, but the point is always the same -- search engines want to deliver the best search experience to their users. The more we think about optimizing our websites for the user experience and providing relevant content, the better we'll do as SEOs.
Tom Parker 8:11 AM on January 19, 2013
@uncle demotivator Adwords referrer data is stored in a cookie, it doesn't get sent through the http header.
Dan Tyre 10:50 AM on January 19, 2013
Hartley, this is seminal information, explained simply, precisely and effectively that will influence the way that we recommend how to manage lead generation for our customers & prospects. Conversions are always a challenge, but developing quality content without keyword data will make it a bit harder
Uncle Demotivator 6:54 PM on January 19, 2013
@Tom Parker
So? It doesn't justify why Google didn't do the same for Analytics (organic search data).
Jonathan Thompson 4:01 PM on January 20, 2013
Thank you for taking the time to write this article. I found it informative and useful. I think the main takeaway for me was that your SEO team changed dropped the S and the E.
Dominic Reyes 6:01 PM on January 20, 2013
If we are going to be slowly losing keyword research in out Hubspot accounts, and focus on landing page performance: Will Hubspot be replacing it with the ability to A/B test landing pages, currently only available to Enterprise Accounts?
Bryan Teague 10:37 AM on January 21, 2013
And what about webmaster tools? There is analytical data available though those tools as well that is independent of google analytics. This is a big help in filling in the gap of the (not provided) reporting.
I don't believe that it captures everything, but, at least anecdotally, the webmaster tools do report on a larger percentage of encrypted search traffic... Logged in users who do not opt out of search reporting, and non-logged in users who use the encrypted search.
By utilizing both tools, keyword optimization is still worth while, and will provide good information, albeit still incomplete for keyword targeting.
I do think that there is, sometimes, to much focus on keywords and not enough focus on providing rich content on your top service offerings, while not neglecting the full offering of a company.
Suzanne Mannion 10:50 PM on January 22, 2013
Very helpful article - thanks! Our team is keeping this in mind while also considering other potentially "hot topic" marketing efforts that may prove beneficial in 2013. One I'm hoping takes off is integrating with non-traditional departments - we can learn from each other http://www.newsmakergroup.com/blog/the-world-is-changing-and-pr-is-keeping-up-strategies-to-focus-on-in-2013/