Fetch as Googlebot: Know How Googlebot Looks at Your Site

Google Webmaster Tools has this “labs” section that stores new features in their early development or beta stage until fully developed. One tool that has just been taken out of the labs and into the world is Fetch as Googlebot, which allows you to see how Google views your website when it’s being crawled.

What’s a Googlebot?

Googlebot is Google’s search bot software used to collect documents from the Internet to create a searchable index for the search engine. The process of discovering updated and new websites is called “crawling.” Also called spider or crawler, Googlebot uses computer programs to find out which sites to crawl, how many pages in a site to fetch, and how often a site should be crawled.

Why is Fetch as Googlebot useful?

Fetch as Googlebot allows you to eliminate the guesswork whenever you need to analyze how Google reads your site. Since the tool identifies content that can and cannot be crawled, you can easily pinpoint which pages need improvement. Troubleshooting, therefore, will be done more quickly.

The new tool also looks at your malware details, and then shows you snippets of malicious code. This is great because you’re now able to know if your site has been hacked with undesired and malicious content.

For example, let’s say you’re the owner of example.com. After looking up your website on Google search, you’re surprised to find that your site shows up for spammy keywords like “male enhancement” or other terms you never included in the source code. If your site is linked to Webmaster Tools, you can use Fetch as Googlebot to fully understand what Google is seeing on your site exactly. The tool lets you see the fetched page’s content, including the word “male enhancement” and such spammy terms. Things like this happen when someone hacks into your website and inserts malicious content, hiding it so that it’s visible to Googlebot only.

Ultimately, Fetch as Googlebot lets you know if the server is returning something else to Googlebot than what normal users see. However, know that the tool doesn’t necessarily show content that Google extracts from your website. For example, if a page contains JavaScript, what you’ll see is the raw JavaScript code, and not the rendered view that appears on the browser.

How to Use

Using Fetch as Googlebot is very simple. On your Webmaster Tools homepage, choose a specific site and then click the Fetch as Googlebot link under Diagnostics. Then just indicate the URL in the text box and click the Fetch button. It may take awhile for Googlebot to return the results since the pages are fetched in real time. To see the progress, click the Refresh button.