Maybe I should have PM'd Calvyn or limcs with this one, but I was thinking of spidering WMM, but I have some questions about WMM's robots exclusion policy, and policy regarding sitemaps.

If anybody can shed some light on whether my spider would be welcome or not, or someone authoritative could give me a green light, I'd be very grateful!

My spider still doesn't follow off-site links, so WMM's members won't get any direct benefit from this, but when I do permit the spider to follow those links, WMM's members' sites should be at the head of the crawl queue.

__________________
Sean says:
Read my blog | Ask My Spider

those whom do not welcome your spider will do something on robots.txt or their server setting.... so fireup your spider... no need to ask...

anyway... why wouldn't a website allow a crawl???

__________________
Freelance Jobs Post
Get Motivated Daily

You've got me there... I was hoping for a reply from WMM because if you try to load their robots.txt or sitemap.xml, you get a 301 to their web root.

I'm really asking because my spider hasn't had very much testing. Every site I try it on, I discover a new ... feature.

OK, I'll add WMM, see what happens!

__________________
Sean says:
Read my blog | Ask My Spider

Stopped. Time to go to bed! 6MB in 120 pages - should be enough for testing! Some 200KB pages in there, and plenty of new keywords for the index!

__________________
Sean says:
Read my blog | Ask My Spider