Google Shows 'robots.txt file' in its SERPS! Can it Create Problems? Confusion Prevails!

“From what I can see, Google shows 182,000 robot.txt files in their index – [inurl:robots.txt filetype:txt] is the query I was studying. By posting that specific query, I am making an exception to our usual policy of "no specific searches", but I feel it is generic enough – there are no keywords involved – to allow an exception in this case.

My guess is that somewhere, someone has linked to these robots.txt specific files – including yours, most likely. Certainly the top results for that query are robots.txt files that I know have been linked to – including ours here at WebmasterWorld. Google should simply drop all robots.txt files from their index, IMO – but we're the tail trying to wag the dog on this kind of thing. How is this causing you a problem, Mike? Is your robots.txt actually ranking on a keyword query?

At any rate, this does bring up the crazy question, how can you remove a robots.txt file from Google's index? If you use robots.txt to block it, that would mean that googlebot should not even request robots.txt – an insane loop. And of course, you don't use meta tags in a robots.txt file.

Unless this is some kind of a problem on keyword queries, I'd suggest you just chalk it up to strange and move on.”

“You never know if it is causing a problem with Google. Wondered if it was a side effect of something bad. We were the victim of a server hack (someone uploaded p*rn images into every page) two weeks ago, then last week someone linked to 1000+ pages that did not exist on the site (yes, Google indexed them too as duplicate content even though they had noindex/nofollow tags) so I wondered if this was the latest ranking attack by our competitors.“

“You can try adding this in your .htaccess file:

<filesmatch "robots\.txt">
Header set X-Robots-Tag "noindex, nofollow"

Then try again to remove with the Google Webmaster Tools "Remove URL".