How To Verify The GoogleBot Crawling Your Site Is Not A Fake

The Googlebot that just visited your website might not have been from Google. There are rogue bots out there which imitate the Googlebot to steal content. Barry Schwartz has compiled few resources which elaborate upon ways to verify the authenticity of a bot.
    
Included in the post are some past posts on the issue, a post by Matt Cutts and discussions at Cre8asite Forums. Matt Cutts has explained how to verify the bot with reverse DNS.

Telling webmasters to use DNS to verify on a case-by-case basis seems like the best way to go. I think the recommended technique would be to do a reverse DNS lookup, verify that the name is in the googlebot.com domain, and then do a corresponding forward DNS->IP lookup using that googlebot.com name; eg:

I don't think just doing a reverse DNS lookup is sufficient, because a spoofer could set up reverse DNS to point to crawl-a-b-c-d.googlebot.com.

Barry also advises to use CrawlWall to automate the process. 
 

Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!