Y! Search Crawler Updated to Slurp 3.0!

Yahoo! announced the latest update to its Search Crawler called Yahoo Slurp 3.0. Although the Slurp 3.0 will recognize the same parameters and all robots.txt directives for Yahoo! Slurp, still it will be registered as Slurp 3.0 in the user web logs.

As this new software is being slowly introduced, users would have an introduction to the following changes:

The crawlers will originate from the crawl.yahoo.net.domain as usual, but they would start crawling from a different and much smaller set of IP addresses. As before, reverse DNS checks performed to identify Yahoo! crawlers will still be working. According to Yahoo!,“Please note that if you're using IP-based recognition of our crawlers, you might see a drop in crawl/coverage from Yahoo! We strongly recommend that you move to 'reverse DNS based identification of Yahoo! Slurp if you're using any other method to avoid this problem. The current set of IPs will disappear from your web logs in the next several weeks.”

As mentioned above, the new user-agent Yahoo! Slurp 3.0 will follow existing robots.txt directives for 'Slurp' or 'Yahoo! Slurp. However, directives specific to Yahoo! Slurp 2.0 will not be recognized or followed by the new crawler. Therefore, it is recommended by Yahoo! that the user specifies the shorter version of User-agent: Slurp.

These new changes will be only applicable to the main Yahoo! Web Search Crawlers and not would not affect specific search crawlers. For example, Yahoo! Slurp China and others.

Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!