Google, Microsoft & Yahoo! enable Cross-submissions via robots.txt on Sitemaps.org

In 2006, Google, Microsoft Live Search and Yahoo! came together and extented their joint support for the Sitemaps protocol. So now furthering this team play, Google, Microsoft and Yahoo! have announced the Cross-host submission for Sitemaps.

The Cross-host submission will make it easier for webmasters to manage their Sitemap submissions to the major search engines. With this announcement, webmasters can now submit Sitemaps that correspond to several differently-hosted websites using a single mechanism. For background, a Sitemap file contains the URLs for the pages on a site along with meta-data, such as priority, last crawled date and changed frequency for the content. To ensure validity of this metadata, Sitemaps have previously been required to be on the same host and path as the URLs they contain. This requirement forced the Sitemaps files to be hosted on the same servers as the actual site content.

With today's announcement, a Sitemap can now be hosted on a different host and path than the URLs it contains. For example, say you have a Sitemap (sitemap-www.xml) for the URLs on http://www.example.com but you want to put that Sitemap on http://sitemaps.example.com. That is now possible. To make the Sitemap valid and preserve data security you need to refer to it from the robots.txt file on the site where the URLs it contains are located. For example, add the following link to http://www.example.com/robots.txt:

Sitemap: http://sitemaps.example.com/sitemap-www.xml

The catch is that all the URLs in the sitemap file need to be within the same domain as the robots.txt file (i.e. moneycentral.msn.com/* in this example). This applies equally for sitemap index files and for compressed files. The protocol is offered under an Attribution-ShareAlike Creative Commons License, so it can be used by any search engine, derivative variations using the same license can be created and it can be used for commercial purposes.

With this team play, we believe that their next move together would be to develop meaningful standard support for robots.txt files. But that only time will tell, however for now it is the moment to celebrate, as at least these competitors together are agreeing on something. What do you say?

Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!