How to Save Googlebot the Trouble of Crawling Infinite Space!

Over at the Google Webmaster Central Blog, Google has put up an interesting post about how Googlebot can get stuck in the loop of 'infinite space' and what measures can the Webmasters take to avoid such a glitch.

Infinite Space as defined by Google: “These are very large numbers of links that usually provide little or no new content for Googlebot to index. If this happens on your site, crawling those URLs may use unnecessary bandwidth, and could result in Googlebot failing to completely index the real content on your site.”

However, with the increase in such incidents, Google has now started to inform the Webmasters via the Webmaster Tools message center, whenever such an anomaly is discovered. Because the issue of infinite space can cause inconvenience to the Webmasters, therefore it is imperative that they get their websites verified with Webmaster Tools. Frequent check of the Webmaster Tools message center would also serve as an added advantage.

Google Webmaster Tools Message Center:

Example of an Infinite Space:

Google has taken the example of a calendar with a 'next month' link. As per Google, “It may be possible to keep following those "Next Month" links forever! Of course, that's not what you want Googlebot to do. Googlebot is smart enough to figure out some of those on its own, but there are a lot of ways to create an infinite space and we may not detect all of them.”

How to Avoid/Correct the Infinite Space Issue:

However, while explaining the problem of Infinite Space, Google has also provided some tips as to how Webmasters can avoid and correct this anomaly.

1. Go through the Webmaster Tools Help article, where you can know more about the steps that can be taken to avoid infinite space.
2. Eliminate whole categories of dynamically generated links using the robots.txt file. You can learn more about the usage of the robots.txt file for the elimination of infinite space here.
3. You can also block these useless links with a "nofollow" link attribute. You can learn more about the usage of the “nofollow” link attribute here.

Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!