Google Finally Speaks About the Ranking Software Blockade!

August 18th, 2008 | 2,039 Views RSS Feed



If you're new here, you may want to subscribe to our Full RSS feed to get a daily digest of news around search engine industry.

At the beginning of this month, we had informed our readers that Google had allegedly blocked one of the most popular search engine ranking softwares, WebPosition.

Now Search Engine Roundtable reports that, over at the Google Groups, Matt Cutts and JohnMu have finally spoken up about why and how Google blocks ranking checking tools.

This is what Matt Cutts had to say:

“Hi Scott. Google does use algorithms and different techniques to block
excessive automated queries and scraping, especially when a someone is
hitting Google quite hard. The reason is that scraping consumes server
resources. We don't want real users to be slowed down or affected just
because a bot is sending bunches of automated queries to Google.

We do turn off a number of tools/bots/IP addresses that scrape us too
heavily. It's a common enough phenomenon that we did a blog post on
Google's Online Security Blog about the subject.
In fact, I know that just a week or so ago our algorithms turned off
an IP belonging to one of the entities that you mentioned in your
post.

In general, I would approach the bizdev folks at Google about how to
send automated queries to Google with permission. Failing that, be
aware that if a tool sends too many queries to Google, we do reserve
the right to disable the IP address(es) of that tool. One thing I
would *not* recommend is that if a tool is blocked for bad behavior,
trying to make the tool more "sneaky" (e.g. trying to make the tool
look closer to a web browser). Attempts to fake out Google and pretend
to be more like a web browser (after you've been blocked once already)
is an example of the sort of thing that is really bad in our opinion. “

JohnMu further stated:

“Hi Scott and a belated welcome to the groups!

This is an interesting topic. I'm not quite sure how all those links
apply to our terms of service, which do not allow these kinds of
automated tools. I wrote a bit about tools like that a short while
ago:

http://groups.google.com/group/Google_Webmaster_Help-Requests/browse_…

A tool accessing other websites should try to obey the rules set forth
by that website. In general, these rules are described in several
ways:

  1. The robots.txt covers which URLs may be accessed and which ones are
    disallowed. You'll notice that in our robots.txt we explicitly
    disallow "/search", which is what most of the ranking tools generally
    try to access.
  2. The server result codes give more information when a URL is
    accessed. When our network recognizes automated queries, it may return
    a result code of 500 or similar.
  3. The HTTP headers returned by the server can provide information
    through the "x-robots-tag".
  4. A HTML page may provide information through a "robots" (or in our
    case, "googlebot") meta tag.

As far as I am aware, there are no "SEO-tools" that have permission to
access our web-search results in an automated way. I am also not aware
of any plans to change that in the near future.â€

Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!


 


Comments

This website uses IntenseDebate comments, but they are not currently loaded because either your browser doesn't support JavaScript, or they didn't load fast enough.

11 Responses to “Google Finally Speaks About the Ranking Software Blockade!”

  1. Search-Optimization Says:
    August 18th, 2008 at 08:33

    Google’s success at organizing the world’s information depends a great deal on search engine optimization experts. These are the people that fill in the Title and Description Meta tags and organize the pages so they have topical community. Search engine optimization experts are the librarians of the Internet making the relevant content easy to identify and find.

    Ranking reports are an important feedback tool to see if we are doing this correctly. Google needs to take a proactive approach and develop their own ranking report tool or work with WebPosition to make this all easy to do without slowing their servers down to much.

  2. 'web designer' Kathaperumal Says:
    August 18th, 2008 at 10:55

    Hello, I am very happy as web designer .i welcome this move of google.com
    common let us face it!

  3. Jim Says:
    August 21st, 2008 at 10:01

    Thanks for your article. i am curious if you have any suggestions about what to do without Webposition. How can I get similar stats for client reports? I would prefer a permanent solution and not continually rotating through programs until Google blocks each one.

  4. George Says:
    August 21st, 2008 at 10:39

    I'm in the same boat as Jim. I would really like to continue offering the same type of ranking reports that WebPosition generated, but I don't know what other tool will be able to do it reliably for me. WebCEO says they can, but they charge for knowledge base updates. I'm not crazy about a subscription type service, so I'm going to keep looking around.

    On a side note, has anyone else picked up on the total lack of respect WebPosition is showing its clients. I've asked their tech support directly what the problem is and there is no acknowledgement that this is a serious problem. They are posting nothing about it on their site or blog, so I'm left to find out what is really going on from blogs like this one. They really haven't handled this well and I think they're kind of scamming people by continuing to promote their rankings report feature. What good is a rankings report without Google results?

  5. George Says:
    August 21st, 2008 at 10:41

    I'm in the same boat as Jim. I would really like to continue offering the same type of ranking reports that WebPosition generated, but I don't know what other tool will be able to do it reliably for me. WebCEO says they can, but they charge for knowledge base updates. I'm not crazy about a subscription type service, so I'm going to keep looking around.

  6. George Says:
    August 21st, 2008 at 10:42

    On a side note, has anyone else picked up on the total lack of respect WebPosition is showing its clients. I've asked their tech support directly what the problem is and there is no acknowledgement that this is a serious problem. They are posting nothing about it on their site or blog, so I'm left to find out what is really going on from blogs like this one. They really haven't handled this well and I think they're kind of scamming people by continuing to promote their rankings report feature. What good is a rankings report without Google results?

  7. Navneet Kaushal Says:
    August 21st, 2008 at 10:47

    I can understand the frustration of webmaster who has to monitor 100s of terms. Its unfair on Google's part to first discontinue the api for search results and now this. They really should give webmaster some options. I won't be surprised if they add a service under Google Webmaster Tools! After all they want whole of the world to signup for it :)

    Till then you can use online tool: http://www.sheerseo.com
    Here is the review of the same: http://www.pagetrafficblog.com/sheerseo-newest-tool-tracking-rankings-index-pages/5034/

  8. Jim Says:
    August 21st, 2008 at 10:47

    I have noticed this issue with custom support as well George. I would continuously get generic email answers that were ellusive and unclear to say the least. Things like, "I have no further information until the update is complete."
    And when I asked when the update would be complete I got the same exact answer!
    Frustrating to say the least.

  9. George Says:
    August 21st, 2008 at 11:05

    Thanks for tip on SheerSEO! It looks promising.

  10. Navneet Kaushal Says:
    August 21st, 2008 at 11:09

    It is good software till the time Google bans it too!

  11. search engine optimisation softwares Says:
    September 10th, 2008 at 04:24

    it is back and working but i haven't looked it with so curosity before it made me think about this software… might increase its sales

Leave a Reply

Back to Top

Life@PageTraffic on Flickr

GalleryBossCafe


More >>

Subscribe To Our SEO Blog


Enter your email address:

Delivered by FeedBurner

Search


PageTraffic on Facebook
SEO Blogs - Blog Catalog Blog Directory
Feedback Form