Googlebots Can Trip Bugs In ASP 2.0?

In a post called, "ASP.NET 2.0 Mozilla Browser Detection Hole," Brendan Kowitz writes that 'there is something drastically wrong with the way search engines have been indexing my ASP.NET 2.0 blog.'

Back in March 2006, Googlebot was altered and it sparked off a bug in ASP.NET webservers. With this, sites wouldn't be crawled properly. In its place, it would receive 500 Internal Server Errors, through which they considered certain webpages as unavailable, even if they are.

Brendan is not alone in this dilemma, many users is Webmaster World are also expressing over the bug, which they feel is Google's fault. Tedster offers a solution:

• "I've been researching this for several clients on the .NET 2.0 platform, and those who use Helicon's ISAPI Rewrite for url rewriting do not seem to have a problem with googlebot's US string. I haven't zeroed in yet on which solutions do trigger the 500 error problem, or why. A number of .NET/IIS rewriting solutions are discussed on Scott Gu' blog (he's a developer for Microsoft) and only some of them have the problem.
• I'd sugggest that anyone using .NET 2.0 install Firefox and the UserAgentSwitcher add-on so they can try a page from their site as Googlebot and discover if they get a 500 error. It's a quick health check-up that is well worth the little bit of time it takes. I also note that mikey158's problem "fixed itself" with no apparent changes on his server – so possibly the Google crawl team is taking some action here when they see too many 500 http errors. All I can imagine them doing, however, is re-crawling with a different user-agent string, one that does not include "Mozilla"."

Discussion continues at Webmaster World.

Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!