#1
Award
Winning
Agency
2016

Selectively Excluding Pages from Being Indexed

There are many times you may with to exclude certain pages from being indexed by certain engines. One way to do this is by utilizing a robots.txt file and uploading it to the root directory of your Web site.

Basically, you just create a text file with Window’s NotePad or any other editor that can save ASCII .txt files.

Use the following syntax:

User-Agent: {SpiderNameHere}
Disallow: {FilenameHere}

For example, to tell Inktomi’s spider, called Slurp, to not index files called orderform.html and junk.html, create a robots.txt file as follows:

User-Agent: ArchitextSpider
Disallow: orderform.html
Disallow: junk.html

You would then upload this robots.txt file to the root directory of your Web site. Although this is a voluntary protocol, most major search engines will honor it.

You can add more lines to exclude pages from other engines by specifying the User-Agent parameter again in the same file, followed by more Disallow lines. Each disallow statement will be applied to the last User-Agent that was specified. If you want to exclude an entire directory, use this syntax:

User-Agent: ArchitextSpider
Disallow: /mydirectory/

Other options are to exclude the page from all spiders with:

User-Agent: *
Disallow: /mydirectory/

Do NOT use the wildcard (*) character in the Disallow line since that’s not supported.

Make sure you use the proper syntax. If you misspell something, it’s not going to work.

About Article

This article is copyrighted and has been reprinted with permission from FirstPlace Software, the makers of WebPosition Gold. FirstPlace Software helped define the SEO industry with the introduction of the first product to track your rankings on the major search engines and to help you improve those rankings. A free trial of WebPosition Gold is available from their Web site.

Related Services

Explore the expert services of Page Traffic that puts you in the search engine spotlight.

  • Search Engine Optimization– Search engines have become rigidly stringent and the competition is doing all it can to move up. However, we snatch the top positions for you.
  • Link Building Services– The definition of quality links is getting narrower. We steer you clear with ‘valuable links.’
  • Hire Full Time Offshore Team– The modern mantras of growth are streamlining and strategizing. Hire our total web management team and take care of both.
Popular Posts
  • Most popular keywords on search engines
  • Happy Diwali from PageTraffic Team
  • SEO Tips- Guide To Cross Linking
  • How many Keywords to use per page for SEO?
  • How To Avoid Search Engine Troubles
Recent Post

Want more? Get updates from PageTraffic, know about latest happenings, SEO news, digital marketing updates.