Quote:
Originally Posted by Seanie
If anybody can shed some light on whether my spider would be welcome or not, or someone authoritative could give me a green light, I'd be very grateful!
|
those whom do not welcome your spider will do something on robots.txt or their server setting.... so fireup your spider... no need to ask...
anyway... why wouldn't a website allow a crawl???