How to Tell Google Not to Index a Page in Search Results [Quick Tip]

    by Elliot Miller

    Date

    March 30, 2013 at 9:00 AM

    lego-robot-with-message.jpgIndexing as many pages on your website as possible is tempting for marketers who are trying to boost their search engine authority. While it’s true that publishing more pages that are relevant for a particular keyword (assuming they’re also high quality) will improve your ranking for that keyword, sometimes there’s just more value in keeping certain pages on your website out of a search engine’s index.

    Say what?! 

    Stay with us, folks. This post will walk you through why you might want to remove certain web pages from the SERPS (search engine results pages), and exactly how to go about doing it.

    Why You'd Want to Keep Web Pages Out of Search Results

    While there may be a number of reasons you'd want to make sure search engines aren't indexing certain pages on your website, one of the most obvious use cases for marketers is thank-you pages. In other words, the page a visitor lands on after converting on one of your landing pages -- and usually where they get access to whatever offer that landing page promised, such as a link to an ebook PDF.

    And any marketer who understands the value of landing pages can understood why having thank-you pages ranking in search is no bueno. After all, you want to be able to capture those visitors as leads first before they can access your offers, don't you? So if your thank-you pages are easily discoverable through a simple Google search, you may be leaving valuable leads on the table. Visitors will have easy access to your lead generating offers without having to provide you with their information to pass through your "gate" -- in other words, the lead-capture form on your landing page.

    What's worse -- you may even find that some of your highest ranking pages for some of your long-tail keywords might be your thank-you pages, which means you could be inviting hundreds of potential leads to bypass your lead-capture forms. This is pretty compelling example of why you'd want to remove some of your web pages from the SERPS if I've ever heard one.

    So how do you go about "un-indexing" certain pages from search engines? In one of two ways.

    Two Ways to Un-Index a Web Page From Search Engines

    Option #1: Via a Robots.txt File

    The first way to remove a page from search engine results is by adding a robots.txt file to your site. The advantage of using this method is that you can get more control over what you are allowing bots to index. This means you can proactively keep unwanted content out of the search results.

    Within a robots.txt file, you can specify whether you’d like to block bots from a single page, a whole directory, or even just a single image or file. There’s also an option to prevent your site from being crawled but still enabling AdSense ads to work. That being said, of the two options available to you, this one requires the most technical kung fu. To learn about how to create a robots.txt file, refer to this article from Google Webmaster Tools.

    But if you don’t need all the control of a robots.txt file and just want to get straight to the point, use my favorite -- the second option.

    Option #2: Via a “noindex” Meta Tag

    Using a meta tag to prevent a page from appearing in search engine results is both effective and easy. It requires only a tiny bit of technical know-how, and it's actually just a copy/paste job if you’re using the right content management system. Here's how to do it ...

    First, copy this tag:

    <META NAME="robots" CONTENT="noindex">

    Next, paste the full tag into a new line within the <head> section of your page’s HTML (known as the page’s header). The screenshots below will walk you through it.

    This signifies the beginning of your header: 

     

    header_start_capture

     

    And here is the meta tag within the header:

     

    meta_tag_in_heaer

     

    And this signifies the end of the header:

     end_of_header_capture

     

    Boom! That’s it. This tag tells a search engine to turn around and go away, leaving the page out of any search results.

    Instructions for HubSpot Customers

    If you’re a HubSpot customer, the "noindex" meta tag option is even easier to implement. Say you’re trying to add this tag to a thank-you page. Simply go to that page's options tab and paste the above tag into the "Head Section HTML" field.

    hubspot-meta-tag-screenshot 

    Ta da! You’ve just magically erased your page from search engine results, and now you can start capturing more of those lost leads again. One thing to keep in mind is that the results may not be instantaneous. Your changes won’t kick in until the next time a search engine crawls your page, and depending on how often you typically publish new pages to your website (the more frequently you publish content, the more often search engines will crawl your site), this might take a few weeks. But you can sleep a little easier knowing you’ve ultimately made your website a better place for your marketing.

    In what other instances might you want to un-index a web page from search engines? Share in the comments below!

    Image Credit: brad montgomery


    SUBSCRIBE TO HUBSPOT'S MARKETING BLOG

    Join 280,000+ fellow marketers! Get HubSpot's latest marketing articles in your inbox. Enter your email:

    Search Inbound Hub

    Subscribe to Marketing Articles by Email

    Subscribe by RSS

    Follow HubSpot

    Call Us: 1-888-HUBSPOT