View Single Post
  #5 (permalink)  
Old 10-06-2008, 01:17 AM
Site Booster Site Booster is offline
Novice Webmaster
 
Join Date: Jan 2008
Location: Kuala Lumpur
Posts: 58
Rep Power: 10
Site Booster is on a distinguished road
Arrow

Quote:
Originally Posted by tedsewely View Post
What is robots.txt
How does it help

Thanks
Ted,
While agreeing with all previous posts, I wanna add that robots.txt is a single text file that you add to the root directory of your website. So, even for several search engine agents, crawlers, spiders, ants or whatever they are called, we aren't allowed to use more than one robots.txt.

Besides, robots.txt is one of the two ways of telling some search engines which pages to index and follow and which ones not to. The other way is using "meta robots" at the header section of each webpage's source code.

Happy searching!
Reply With Quote