Problem In Yahoo!'s Crawler?

Seems like there is a big problem with Yahoo!'s crawler. Reports coming from Search Engine Watch Forums and  WebmasterWorld Forums indicate that Yahoo! Slurp is indexing pages which should not be indexed. It might be that some specific bots are not conforming to robots.txt file. The pages are being indexed at rates which can be very harmful. A member of  Search Engine Watch Forums has posted an example:

“Host: 72.30.216.22
/suspended.page/
Http Code: 404 Date: Jan 17 01:12:31 Http Version: HTTP/1.0 Size in Bytes: –
Referer: –
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 74.6.67.78
/suspended.page/
Http Code: 404 Date: Jan 17 01:11:24 Http Version: HTTP/1.0 Size in Bytes: –
Referer: –
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 74.6.74.155
/suspended.page/
Http Code: 404 Date: Jan 17 01:08:45 Http Version: HTTP/1.0 Size in Bytes: –
Referer: –
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 74.6.71.43
/suspended.page/”

Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!