Blocking your Redirect 301 with robots.txt file? Bad Move!

May 30th, 2008 | 1,896 Views RSS Feed

If you're new here, you may want to subscribe to our Full RSS feed to get a daily digest of news around search engine industry.

If you are thinking of blocking your redirect 301 with a robots.txt file, then you would be making a big mistake. According to Search Engine Roundtable, there is a thread at Google Groups that is concentrated on this very concern.

One of the users who probably started the thread took up this topic in a very long post and then finally got a response from Googler JohnMu. According to the user, he moved from domain.com to domain.info, on a domain name sale, but wanted to retain his links, so set up a 301 redirect from .com to .info for a certain period of time. Here is the user's post:

“I recently moved domains from insulators.com (buyer has since turned
it into an ad portal) to insulators.info

All the content has been moved to the new domain. I submitted a request to remove all insulators.com pages several weeks ago, and you can see this is true:

site:insulators.com

Google has indexed a significant part of my site

site:insulators.info

including the first set of key pages that you see for the above.

However, when you do a

link:insulators.info

many of the links show up as links to insulators.com not to
insulators.info!!! The fifthe entry, "berhomepage" .. go there, and
look at the two insulator links at the way bottom… The go to
insulators.com !!!

I think Google is confused. I suspect my queries are suffering, or in
any case, the link: is misleading at least me!

How do I get link: to report the proper links to my new site so I can
monitor my efforts to get other sites links changed over to my new
domain?

If there is any questions, insulators.info returns HTTP/1.1 301 Moved
Permanently to www.insulators.info, and I have set the Google
preference to www.insulators.info.

Sorry for the extra explainations… I'm afraid I'm getting penalized
in a way I don't know, because of the (old) presence of insulators.com

I used to be the top ranked for a number of searches
insulators, glass insulators, telephone insulators, etc.

Now, I don't even see my site in the rankings… well, top 50 like for
glass insulators.”

And now Googler Mu's response:

“Hi Mr. Hemingray

Thanks for telling your interesting story!

One small item which I would recommend doing differently the next time something like this comes up (you never know ), is not using the robots.txt to block crawling while you have a 301 redirect enabled for the domain. By blocking crawling, you're effectively blocking the search engines from recognizing the redirect properly.

Given the things that you have done so far, I think it's probably not that important to check on the details as shown in the link:-query for your site. Over time, things should normalize — you're on the right path!”

In case if you are concerned as to how long the Redirect 301 should be kept active, then you need not worry, because there is a simpler way. You can always collect as much linkage data as you can from the previous domain. This will enable you to go back to those websites and ask them to update your link to the new domain. You can collect linkage data from Yahoo Site Explorer, Google Webmaster Tools, your web analytics, your own database scripts and more.

Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!

Did you like this article?

Debugging blocked URLs
Confused by "blocked by robots.txt" errors? Read this post at Official Google Webmas...
Control The Indexing And Accessing Of Your Sites By Search Engines
A post on Google blog gives important details to the web publishers about how they can control index...
Google Shows 'robots.txt file' in its SERPS! Can it Create Problems? Confusion Prevails!
There is a new concern at the Webmaster World with many Webmasters reporting that Google is showing ...
Microsoft Has Now Introduced Sitemaps Autodiscovery
Here Comes Robots.txt Generator by Google!

Comments

This website uses IntenseDebate comments, but they are not currently loaded because either your browser doesn't support JavaScript, or they didn't load fast enough.