There are many times you may with to exclude certain pages from being indexed by certain engines. One way to do this is by utilizing a robots.txt file and uploading it to the root directory of your Web site.
Basically, you just create a text file with Window’s NotePad or any other editor that can save ASCII .txt files.
You would then upload this robots.txt file to the root directory of your Web site. Although this is a voluntary protocol, most major search engines will honor it.
You can add more lines to exclude pages from other engines by specifying the User-Agent parameter again in the same file, followed by more Disallow lines. Each disallow statement will be applied to the last User-Agent that was specified. If you want to exclude an entire directory, use this syntax:
Other options are to exclude the page from all spiders with:
User-Agent: * Disallow: /mydirectory/
Do NOT use the wildcard (*) character in the Disallow line since that’s not supported.
Make sure you use the proper syntax. If you misspell something, it’s not going to work.
This article is copyrighted and has been reprinted with permission from FirstPlace Software, the makers of WebPosition Gold. FirstPlace Software helped define the SEO industry with the introduction of the first product to track your rankings on the major search engines and to help you improve those rankings. A free trial of WebPosition Gold is available from their Web site.
Explore the award winning services of PageTraffic that puts you in the search engine spotlight.
SEO Services – Search engines have become rigidly stringent and the competition is doing all it can to move up. However, we snatch the top positions for you.
Full Time SEO Consultant – The modern mantras of growth are streamlining and strategizing. Hire an expert full time SEO consultant to manage and take care of both.
SEO Reseller – Expand your client base and service offerings with our private label SEO reseller partnership.
The author is a first generation entrepreneur. He is the founder and CEO of India’s most award winning SEO Agency, PageTraffic. He also works as an advisor and consultant with many large companies. He helps value-conscious businesses build site traffic increase the user base, and boost web sales and bolster customer satisfaction. He can be reached @navneetkaushal on Twitter