04-09-2007, 10:11 AM
|
|
Novice Webmaster
|
|
Join Date: Sep 2007
Location: Wo|2l|] 0f my 0\/\/|\|
Posts: 40
Rep Power: 0
|
|
without robot.txt googlebot still can crawl to your site but with robot.txt u can restrict the crawler to crawl or not to crawl a certain pages of your choice
|