Big Brother DreamHost Demands Obstructing Google Bot!

Search Engine Journal has an interesting report on DreamHost, a Web hosting & Server company who has asked its Client sites to block Googlebots. The news came known when a letter DreamHost sent one of its clients

Zoso emerged.

Stated in the letter from DreamHost:

“This email is to inform you that a few of your sites were getting hammered by Google bot. This was causing a heavy load on the webserver, and in turn affecting other customers on your shared server. In order to maintain stability on the webserver, I was forced to block Google bot via the .htaccess file.

[Limit GET HEAD POST

order allow,deny

deny from 66.249

allow from all]

You also want to consider making your files be unsearchable by robots and crawlers, as that usually contributes to high number of hits. If they hit a dynamic file, like php, it can cause high memory usage and consequently high load…”

Readers commented on varying degrees. One said “This is a host everyone dreams of, a host that will ban search engine’s bots. Dreamhosts go back to stone age!”. While a calmer one said, “There is too little information for anyone to do a proper analysis. With a dedicated server, I have occasionally had to tell Google and Yahoo! to stay out of my domains for as much as a couple of weeks. They can really hammer a server when they do a deep-crawl, which doesn’t happen often.A site that is getting deep-crawled may be too busy for shared hosting. The last shared hosting provider I had was kind enough to give me six weeks to find a dedicated plan once I’d outgrown their resources. I’ve known people whose hosting providers merely shut down their sites without so much as a heads up email saying, “You should get a new hosting provider immediately”. It sounds very much like growing pains for a successful site. Maybe it’s just time for them to take the next step and move to dedicated hosting.”

This is not the first time DreamHost has manipulated (or tried to) client sites, as is evident in a forum at Digital Point.

Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!