Googlebot is the indexing program of Google that visits your site to fetch the content to determine your search engine rankings. With a popular website, tens of thousands of pages can be a problem in that Googlebot may visit more than you want because it’s expending your precious bandwidth and even crashing your server. Every crawler bot visit is no different than a user one and your site has to perform all necessary actions and logics to render the web page to the search engine crawler bot such as Googlebot including searching through a database with potentially millions of records which could take a while. Imagine, the Googlebot pays 300,000 visits a month to your site. That’d be a substantial expenditure in bandwidth and server computing resources.
For example, one of my sites have experienced 338,768 hits from just one IP of Googlebot last month. I have no idea why Googlebot is so fanatic about this site because it rarely have any backlinks and Google is neither sending any significant traffic. But one thing is for sure, that this site is creating some serious trouble on my hosting bills because of Gooblebot. While we cannot totally deny Googlebot from visiting our websites, we can do something to slow it down a little bit.
There are 2 things you can do:
- Visit Google Webmaster Central: http://www.google.com/webmaster, sign in to webmaster tools, (add if you haven’t, and) select the site in question, select Site configuration, select Settings, select Set custom crawl rate of Crawl rate and adjust the scroll bar to slow the crawl rate down.
- Create robots.txt and place it at the root of your website, put these 2 lines inside:
User-agent: * Crawl-delay: 20
Wherein the value of Crawl-delay is the time in seconds that the search engine bot should wait between requests. 20 is a very slow option. Most search engines such as Google wait less than 1 second to fetch a moderately popular website.
Some argue that Google never respects the Crawl-delay option of robots.txt and the only way to decrease the visiting frequency of Googlebot is to adjust the scroll bar in Google Webmaster Central.