So, you've developed an amazing site. You've put together some seriously valuable content, iterated on the design, and gathered a ton of feedback. Now, you're finally feeling ready to share it with the world.
And as soon as you publish your new site for the world to see, all of your content will immediately start showing up in Google, right?
Well, maybe. It takes a little more than just hitting the publish button. To get your site listed on search results, Google needs to "crawl" and "index" your content. This happens automatically over periods of days or weeks, but if you're a site owner, you can manually submit your site to Google and accelerate this process.
There are two ways to do this. But first, let's briefly explore how Google crawls and indexes your website content.
How Google Finds Your Content
Google, in its own words, uses a huge set of computers to crawl billions of pages on the web. This crawler, called the Googlebot, essentially begins with a list of web page URLs generated from previous crawls and then augments those pages with sitemap data provided within Google Search Console. During the crawling process, the Googlebot -- also known as a "spider" -- looks for new sites, updates to existing pages, and any broken links.
If new pages are within your sitemap, Google will discover them and crawl the content and then potentially list the page within search results based on its evaluation of 200+ criteria.
Once the crawling process is complete, all of the results are fed into Google's index, and any new sites or updated content will be listed accordingly. During the processing of results, Google looks at information on your page such as title tags, meta description, alt tags, and more. If you have dynamic content on a page, the Googlebot may not be able to read it and will crawl the default version -- it's recommended that your default version is optimized for search.
As a result of Google's crawling, you may never need to submit your website as it will be discovered automatically. The downside to this approach has always been that it's reliant on Google's timeframe to crawl and index your site content, which may not happen as quickly as you would like.
Want to check if a particular site is listed in Google? Just begin your search with "Site:Sitename.com." For example, here's what is displayed in Google for HubSpot.com:
If no content is indexed yet for a site, Google will let you know that your search did not match any results.
If no content is found, your next step should be to create a sitemap that can be submitted to Google.
(HubSpot customers: Your sitemap is automatically generated and maintained. Simply go to yourdomain.com/sitemap.xml to see it.)
How Long Does It Take for Google to Index Content?
My former colleague Casey Henry wondered this very question and ran a test to see how long it took Yahoo! and Google to crawl and index content. The results? Well, they were staggering.
When publishing content without manually submitting an updated sitemap, Henry found that it took Google 1,375 minutes to crawl, while Yahoo took 1,773 minutes. To put those numbers into perspective, that's roughly a full-day just to crawl your content.
On the other hand, if you're launching a new website, or adding a number of new pages to an existing domain, it may be worth submitting an updated sitemap. According to the same study, Henry found that after submitting an updated sitemap, the average time it took for a bot to visit the page was 14 minutes, compared to Yahoo!'s 245 minutes. In other words, your new page can start generating organic traffic and conversions on the same day.
To submit your website to Google, you can either add an updated sitemap to your Google account, or submit an indexing request for the desired URL through Fetch as Google. Both processes require site owners to register with Google Search Console.
Here are the specifics of each option:
If you have a brand new site ...
If you're launching a website for the first time, you should first verify you own the site within Google Search Console. Then, submit it here -- select the "submit a sitemap" option once you land on this page.
If you have an existing site and are launching new pages ...
With a web domain already launched, you can still submit new pages for Google to index and rank them accordingly. Anyone used to be able to do this with a page they wanted crawled, whether they owned the page or not. Now, just as you would when launching a brand new website, you must be the URL's owner to ask Google to re-crawl it.
If you own a webpage you'd like Google to re-crawl, you can do a few different things:
Submit an Updated Sitemap
First, you can submit an updated sitemap to ensure it gets listed as quickly as possible. To submit an updated sitemap, log in to Google Search Console and select "Add a Property."
Once you're there, you can submit your updated sitemap for Google so it can begin crawling it as soon as possible.
Fetch as Google
Fetch as Google allows you to view webpages on your site as Google sees them. As a site owner, you can also use this tool to reindex individual URLs once you've fetched them. To do this, log into Google Search Console and select the property you have currently listed with Google.
Along the lefthand sidebar, select Crawl > Fetch as Google, as shown below:
This will pull up a table where you can enter a URL path following your domain name, and "Fetch" this particular webpage on your website. See what this looks like below:
As you can see above, two sample Fetch requests have been listed below the Fetch bar. The status of these fetches is "Partial," but once these fetches are complete, Google will assess their eligibility for reindexing. If the URL you've fetched qualifies, you'll see a "Request Indexing" option appear where the red box is in the screenshot above. Select this option, and Google will begin the process of re-crawling and reindexing this webpage.
As stated earlier in this article, indexing requests can take anywhere from a day to a couple of weeks to complete. So, check your Fetch as Google table periodically to see the status of your active indexing requests.
Based on the above instructions, you may be wondering if you need to submit an updated sitemap or indexing request every time you publish a new page. If you're updating critically important content that you'd like Google to recognize quickly, you certainly could do so. But keep in mind Google also re-crawls pages on its own, and it's ok to let this process work in the background as you regularly create and update your content.
Originally published Jul 30, 2018 8:00:00 PM, updated December 04 2020