Indexing as many pages on your website as possible is tempting for marketers who are trying to boost their search engine authority. While it’s true that publishing more pages that are relevant for a particular keyword (assuming they’re also high quality) will improve your ranking for that keyword, sometimes there’s just more value in keeping certain pages on your website out of a search engine’s index.
Stay with us, folks. This post will walk you through why you might want to remove certain web pages from the SERPS (search engine results pages), and exactly how to go about doing it.
Why You'd Want to Keep Web Pages Out of Search Results
While there may be a number of reasons you'd want to make sure search engines aren't indexing certain pages on your website, one of the most obvious use cases for marketers is thank-you pages. In other words, the page a visitor lands on after converting on one of your landing pages -- and usually where they get access to whatever offer that landing page promised, such as a link to an ebook PDF.
And any marketer who understands the value of landing pages can understood why having thank-you pages ranking in search is no bueno. After all, you want to be able to capture those visitors as leads first before they can access your offers, don't you? So if your thank-you pages are easily discoverable through a simple Google search, you may be leaving valuable leads on the table. Visitors will have easy access to your lead generating offers without having to provide you with their information to pass through your "gate" -- in other words, the lead-capture form on your landing page.
What's worse -- you may even find that some of your highest ranking pages for some of your long-tail keywords might be your thank-you pages, which means you could be inviting hundreds of potential leads to bypass your lead-capture forms. This is pretty compelling example of why you'd want to remove some of your web pages from the SERPS if I've ever heard one.
So how do you go about "un-indexing" certain pages from search engines? In one of two ways.
Two Ways to Un-Index a Web Page From Search Engines
Option #1: Via a Robots.txt File
The first way to remove a page from search engine results is by adding a robots.txt file to your site. The advantage of using this method is that you can get more control over what you are allowing bots to index. This means you can proactively keep unwanted content out of the search results.
Within a robots.txt file, you can specify whether you’d like to block bots from a single page, a whole directory, or even just a single image or file. There’s also an option to prevent your site from being crawled but still enabling AdSense ads to work. That being said, of the two options available to you, this one requires the most technical kung fu. To learn about how to create a robots.txt file, refer to this article from Google Webmaster Tools.
But if you don’t need all the control of a robots.txt file and just want to get straight to the point, use my favorite -- the second option.
Option #2: Via a “noindex” Meta Tag
Using a meta tag to prevent a page from appearing in search engine results is both effective and easy. It requires only a tiny bit of technical know-how, and it's actually just a copy/paste job if you’re using the right content management system. Here's how to do it ...
First, copy this tag:
<META NAME="robots" CONTENT="noindex">
Next, paste the full tag into a new line within the <head> section of your page’s HTML (known as the page’s header). The screenshots below will walk you through it.
This signifies the beginning of your header:
And here is the meta tag within the header:
And this signifies the end of the header:
Boom! That’s it. This tag tells a search engine to turn around and go away, leaving the page out of any search results.
Instructions for HubSpot Customers
If you’re a HubSpot customer, the "noindex" meta tag option is even easier to implement. Say you’re trying to add this tag to a thank-you page. Simply go to that page's options tab and paste the above tag into the "Head Section HTML" field.
Ta da! You’ve just magically erased your page from search engine results, and now you can start capturing more of those lost leads again. One thing to keep in mind is that the results may not be instantaneous. Your changes won’t kick in until the next time a search engine crawls your page, and depending on how often you typically publish new pages to your website (the more frequently you publish content, the more often search engines will crawl your site), this might take a few weeks. But you can sleep a little easier knowing you’ve ultimately made your website a better place for your marketing.
In what other instances might you want to un-index a web page from search engines? Share in the comments below!
Image Credit: brad montgomery