View Single Post
  #5 (permalink)  
Old 16-05-2008, 11:41 AM
Seanie's Avatar
Seanie Seanie is offline
Senior Webmaster
 
Join Date: Mar 2008
Location: pd
Posts: 200
Rep Power: 11
Seanie has a spectacular aura about Seanie has a spectacular aura about
Noticed an improvement with sitemap?

I see Google reading the sitemap on one of my sites more than daily, but still almost all the URLs it reads (~400 a day) are extracted from pages. I generate my sitemaps using a script I wrote myself to produce working URLs that have minimal arguments in them (no language=, no currency=, etc), but the robot is still crawling duplicate pages that have different URL args.

Anybody else noticed that?
Reply With Quote