How to Get Your Pages Indexed and Then Keep Them That Way

As the previous article mentioned, it is an important first step to get your Web site indexed and then to keep it indexed. Here are a number of techniques you should consider to make sure your pages are not among the majority of Web pages that are not found in any search engine:

1. Make sure ALL your pages get indexed.

The more pages you have indexed, the greater chance you have of being found. Does this mean you need to submit every URL of your Web site? No! In fact, many engines don't like you to submit more than one URL per day. One of the best techniques to get all your pages indexed is to create and submit a single "hallway page" as described in last month's article at:

http://www.webposition.com/mp-0799.htm#ONE

This hallway page will link directly to all the other pages on your site that you want indexed, including your doorway pages. Many engines will only spider down one additional level, sometimes two levels looking for pages and will then stop.

To better explain how far a search engine will spider through your site, let's say you have a home page for your Mexican food site called index.htm that you plan to submit. That would be considered LEVEL 1 of the Web site "tree." If the index.htm page linked directly to other pages such as taco.htm and burrito.htm, then those pages would be on LEVEL 2. If taco.htm then linked to cookbook.htm, cookbook.htm would be on LEVEL 3. Finally, a link from cookbook.htm to orderform.htm would put orderform.htm on LEVEL 4. In summary, the layout would look like this:

LEVEL 1 Index.htm

LEVEL 2 tacos.htm and burritos.htm

LEVEL 3 cookbook.htm

LEVEL 4 orderform.htm

The problem is that most search engines refuse to spider more than two or three levels deep. Therefore, all your pages on LEVEL 4, and for some engines your pages on LEVEL 3, would never be found by the search engine unless you submitted them directly. However, submitting them directly is not recommended since it can lower your rankings and is difficult on engines that employ daily limits. That's why you should either design your site to be no more than two or three levels deep, or create a hallway page which links directly to all the pages on your site. This page is basically a table of contents for your Web site for the spider to "feed" on.

2. Order your hallway page links by importance.

It's believed that some search engines limit the number of links they will spider on a single page. The exact limits are unknown (at least to me), so I'd recommend you order your links based on importance. As an additional precaution, you might also consider keeping your links to 50 or less per hallway page.

3. Be patient after submitting

The search engines will vary in the amount of time needed to index you from one day to six weeks. The WebPosition Submitter will give you an estimate of the time required for each engine and keep track of when each page was submitted. However, if you submit your home page, or a hallway page, expect to wait another week or two before the spider returns to index your pages on the second and third levels. Most engines will put second and third level pages in a lower priority queue. Ironically though, once they do spider to the lower level pages, they will often rank them higher than if you submitted them directly.

4. Consider multiple domains if page limits are a problem for you.

Many engines employ limits to the number of pages they will index for each domain. The number varies, but generally you'll be lucky to get more than 400 pages indexed on a single domain. I've heard of people getting 800 or more indexed, but they are more the exception than the rule. If your Web site has a LOT of pages on it, consider creating separate domain names to host different areas of your current Web site. That way you would be more likely to get more total pages indexed.

5. Avoid CGI generated pages

Most major engines will refuse to index pages that point to a CGI script. These type of pages often include a question mark and ampersand symbols to separate the parameters. An example of one of these URLs would be:

http://www.mysite.com/cgi-bin/generate.pl?category=cars&subcat=porsche

Instead of submitting a URL like the one above, you should create separate non-CGI pages for the search engines to index.

If you have dynamically generated pages that are generated without special parameters in the URL and without giving any indication that the server is generating them on the fly, you should be OK.

6. Verify that your pages stay indexed!

Once you've submitted your page, use the WebPosition Gold URL Verification feature in the Reporter which will keep tabs on whether a page is still indexed. Combine this feature with the Scheduler to save time. After running your mission, check the "URL Submission Verification" link on the report menu to see what pages are currently indexed.

Caution: It's important to setup the URL Verification feature properly to get accurate results on each search engine. Before using this feature, read this page on how to avoid the potential pitfalls of verifying whether your page is currently indexed:

http://www.webposition.com/urlnotfoundhelp.htm

7. Keep content fresh

Search engines are known to drop pages that have not changed for a long-time in favor of fresher pages. Therefore, you might consider freshening the content periodically for your main pages. However, for doorway pages that already rank high in the search results, you should think twice about changing anything significant that could jeopardize your top ranking.

8. Re-submit often?

Some Web marketers recommend resubmitting often to improve the odds of staying indexed. I don't see anything wrong with this strategy if done within reason such as once a week, or once a month. However, don't try this tactic with directory engines like Yahoo!.

9. Ultimately your position is what matters most

Getting your pages indexed is generally not difficult. However, most people make the mistake of stopping there. You must take steps to make sure you rank in the top 10 to 20 matches for keywords which are important to your business. Achieving that goal by hand can be a very daunting task which is why we created WebPosition Gold to step you through the process.

This article is copyrighted and has been reprinted with permission from FirstPlace Software, the makers of WebPosition Gold. FirstPlace Software helped define the SEO industry with the introduction of the first product to track your rankings on the major search engines and to help you improve those rankings. A free trial of WebPosition Gold is available from their Web site.

Related Services

Explore the expert services of Page Traffic that puts you in the search engine spotlight.