Go Back   Webmaster Malaysia Forum » Website Design & Development » Website Design

Reply
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 06-08-2008, 06:09 PM
Seanie's Avatar
Senior Webmaster
 
Join Date: Mar 2008
Location: pd
Posts: 224
Rep Power: 11
Seanie has a spectacular aura about Seanie has a spectacular aura about
Can I spider WMM?

Maybe I should have PM'd Calvyn or limcs with this one, but I was thinking of spidering WMM, but I have some questions about WMM's robots exclusion policy, and policy regarding sitemaps.

If anybody can shed some light on whether my spider would be welcome or not, or someone authoritative could give me a green light, I'd be very grateful!

My spider still doesn't follow off-site links, so WMM's members won't get any direct benefit from this, but when I do permit the spider to follow those links, WMM's members' sites should be at the head of the crawl queue.
__________________
Sean says:
Read my blog | Ask My Spider
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2 (permalink)  
Old 07-08-2008, 02:36 AM
iamfreelancer's Avatar
Vibrate your Brain Please
 
Join Date: Sep 2005
Location: in my body lar...
Posts: 1,217
Rep Power: 62
iamfreelancer will become famous soon enough iamfreelancer will become famous soon enough
Quote:
Originally Posted by Seanie View Post

If anybody can shed some light on whether my spider would be welcome or not, or someone authoritative could give me a green light, I'd be very grateful!
those whom do not welcome your spider will do something on robots.txt or their server setting.... so fireup your spider... no need to ask...

anyway... why wouldn't a website allow a crawl???
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3 (permalink)  
Old 07-08-2008, 03:06 AM
Seanie's Avatar
Senior Webmaster
 
Join Date: Mar 2008
Location: pd
Posts: 224
Rep Power: 11
Seanie has a spectacular aura about Seanie has a spectacular aura about
Quote:
why wouldn't a website allow a crawl???
You've got me there... I was hoping for a reply from WMM because if you try to load their robots.txt or sitemap.xml, you get a 301 to their web root.

I'm really asking because my spider hasn't had very much testing. Every site I try it on, I discover a new ... feature.

OK, I'll add WMM, see what happens!
__________________
Sean says:
Read my blog | Ask My Spider
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4 (permalink)  
Old 07-08-2008, 04:16 AM
Seanie's Avatar
Senior Webmaster
 
Join Date: Mar 2008
Location: pd
Posts: 224
Rep Power: 11
Seanie has a spectacular aura about Seanie has a spectacular aura about
Stopped. Time to go to bed! 6MB in 120 pages - should be enough for testing! Some 200KB pages in there, and plenty of new keywords for the index!
__________________
Sean says:
Read my blog | Ask My Spider
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
Bot/Spider List/Spam Bot List genzy Website Programming 0 01-07-2007 10:29 AM
Movie - Spider-man 3 genzy Mamak Stall 28 03-06-2007 05:06 PM
Spider Man 3 at Toy Fair 07 pablovagne Member Introductions 0 14-02-2007 05:43 PM
Spider loves "Sitemap" keyword better? lcf Other Internet Marketing Methods 10 28-04-2005 10:15 PM
Spider-Man 2 Official Site MENJ Websites Review and Suggestion 1 26-01-2004 01:41 AM



All times are GMT +8. The time now is 06:52 PM. Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
LinkBacks Enabled by vBSEO 3.1.0 vBulletin skin by ForumMonkeys.com.


WebmasterMalaysia.com is Proudly Hosted by Exabytes Semi Dedicated Server.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59