Why some pages rank high for no apparent reason!

We've all been there: You do a search for a keyword that should be relevant to your Website and a page appears near the top of the results for no logical reason. In many cases, the keyword queried isn't even on that top ranking page! At best, the keyword appears just once or twice while your Web site has been carefully constructed to incorporate that keyword several times - and it still ranks lower than this mystery page! Stop banging your head against the wall - there are reasons. Once you learn them you can combat these pages. Here are some things that can cause a Web page to rank higher than yours even when it doesn't appear to be optimized for a particular keyword or phrase:

1. Out of date pages:

A Webmaster can change a page on their Web site. Unless that changed page is resubmitted, the search engine may not know to re-visit the page, spider it and update its index for quite some time. Even if a Webmaster resubmits their page, some engines take weeks to re-visit and re-index a site.

What can you do about this kind of situation? If this is, in fact, the problem, you can go to the engine's submit URL page and submit the offending page's URL for re-indexing. After being re-indexed, the page should drop in rank. It may take a few weeks until the site is re-indexed, but taking the initiative and re-submitting a site should accelerate the process. Infoseek, AltaVista, and HotBot, will typically re-index a submitted site within 2-3 days or less.

2. Dynamic Page Substitution:

Some Webmasters create scripts on their Web site's server that can literally detect the IP addresses or the "browser name" of a search engine's re-indexing spider visiting their site. When this script detects a search engine's spider it "serves" a different Web page than the one you or I would see after clicking on the link from the search engine's results!

There are a few clues that tell you this technique is being employed:

A. The page you see does not contain the words used to describe it in the search engine's listing. The words used as the site's description in the search engine's index are *always* taken from the actual page itself - from the meta tags or actual copy that makes up the site. If the words found in the search engine's listing for that site are close, but not an exact match for any text on that site or in its meta tags, suspect this technique.

B. The text found in the <TITLE> tag of the Web site is different than the what the search engine's listing uses as the site title (usually the text that is represented as a blue hyperlink to the site). About half of the major search engines use only the text found in the site's title as their title for the site. Or, as above, the text doesn't appear anywhere on the page - in the <TITLE> tag or elsewhere.

C. You submit the URL for re-indexing and, once re-indexed, the site's position does not change, AND, the title and description used by the search engine hasn't changed to reflect what you know is actually on the page you were viewing.

Typically, the page "served" or shown to the search engine's spider is a rather unattractive page. Often these pages are much like any other optimized page that you might create to secure a top ranking - they're primarily text, with high keyword frequency, prominence and weight. The technique they're employing simply hides these pages from the general public - presumably because the optimized page isn't all that attractive, or, so that their "secret techniques" cannot be copied by others seeking to outrank them.

Sometimes this technique can be abused and used to hide pages that employ blatant keyword stuffing, spamming, or other inappropriate techniques. Some engines do not allow the technique and will remove pages from their index that use it. Personally, I feel the technique of swapping pages can degrade the search engines, and should not be used since the results the user sees are not what the search engine actually considered "relevant".

That can of worms aside, your problem is that you need to outrank these hidden pages. To start with, relax, and recognize that this technique gives them no magic bullet advantage. They still have to build a high ranking doorway page to serve to the search engine's spiders. The difficulty in outranking them is that you cannot view their HTML source and check their keyword frequency, weight and so on. You can still tweak your page's keyword content to out score them.

One trick that most people overlook is to review the source code of another page that is ranked higher than one that is hiding the actual doorway page. After all, if a page outranks one using this substitution technique it must have higher concentrations of keywords in the right places. Review this higher ranking page and base your strategy on that page instead - problem solved. You can also alert the search engine by e-mail that a Web site is using this technique and, depending on how they feel about it, they can verify that this is the technique being used and remove the page's listing in their index.

3. The "Ol' Switcheroo" technique:

This technique involves building a page optimized to earn a top ranking and then swapping it out for your "real page" once the site has been indexed. This is a sleazy technique - but easily addressed. Like the page substitution technique described above, you detect that this technique is being employed by looking at the listing in the search engine and then comparing it against what you see on the Web site. If the two don't match, e.g., the title or site description are not found on the actual Web site, chances are this technique was probably used. Unlike the dynamic page substitution technique, when you re-submit the URL to the search engine it will likely drop in rank and the new listing will include copy found on the actual Web page. Resubmitting pages you find using this technique usually causes them to tumble down the search results to a position that doesn't compete with yours.

4. New ranking algorithm:

Search engines change their ranking algorithm from time to time. Techniques that worked well last month, may not be as effective today. Search engines can take some time before they apply a new algorithm to their entire index. Until this happens, some older pages may continue to rank high, even though your submission modeled after their success don't score well.

The solution again: submit the page. This should cause the search engine to apply the "new" rules to the page such that it is measured under the same relevancy system as your page. Once this has occurred, you will find out the "true" rank of the page in question, and you can be certain you are modeling your page after pages that are ranking well under current page scoring rules.

5. Page Popularity:

Another reason pages that don't seem particularly optimized for a given keyword rank well is that hundreds or even thousands of other Web sites have established links to them. Some search engines consider "page popularity," or, how many other Web sites have linked to a particular page in determining how relevant the page is.

To determine if this measure is keeping another page ahead of yours in search results, you'll need to do a "Links to URL" search.

A number of engines support a "Links to URL" search on their "Advanced" search options page. If not, some allow you to type the word "link:" and then your URL into the search field to return a list of sites that the search engine has recorded as linking to yours. If you have WebPosition the Page Critic has an option to automatically check and report your link popularity for the engine you have chosen. We also have a link checker available here: http://www.linkpopularitycheck.com

The popularity measure is another reason to spend part of your marketing effort soliciting links from other sites. I'm not entirely certain if the search engines differentiate between links to your root domain page as opposed to internal pages. I suspect they consider the number of links to a specific page and that also has a trickle down effect through out the domain. If you have an opinion about this, let me know.

6. Search Engine Bugs:

Yes, even the big commercial search engines have bugs. Since they are continually trying to fine tune their system to provide better results, or to beat back the spammers, software glitches or "bugs" can easily make their way into the database. Sometimes it will be corrected quickly but in other cases it may score pages incorrectly or poorly for quite some time.

About all you can do in this situation is to alert the search engine that xyz pages rank high on xyz search, and that they really are not relevant to that particular keyword search. The "smart" search engines will listen and look into why the search results were poor. When people don't find documents they feel are relevant to keywords they queried, they frequently try again on another engine. Search engines don't want that because they make their money on advertising to those visitors.

7. The page is simply well optimized:

Often the reason a page ranks high is it simply fits the criteria that a search engine is looking for.

The search engine's algorithms are fairly sophisticated so sometimes it takes a second look to understand why a page is positioned where it is. A number of factors affect search relevancy including keyword "weight", "prominence", "frequency", and avoiding techniques like repeating keywords too many times - a.k.a "spam."

About the Author

This article is copyrighted and has been reprinted with permission from FirstPlace Software, the makers of WebPosition Gold. FirstPlace Software helped define the SEO industry with the introduction of the first product to track your rankings on the major search engines and to help you improve those rankings. A free trial of WebPosition Gold is available from their Web site.

Related Services

Explore the expert services of Page Traffic that puts you in the search engine spotlight.