Quote:
Originally Posted by tedsewely
What is robots.txt
How does it help
Thanks
Ted,
|
While agreeing with all previous posts, I wanna add that robots.txt is a single text file that you add to the root directory of your website. So, even for several search engine agents, crawlers, spiders, ants or whatever they are called, we aren't allowed to use more than one robots.txt.
Besides, robots.txt is one of the two ways of telling some search engines which pages to index and follow and which ones not to. The other way is using "meta robots" at the header section of each webpage's source code.
Happy searching!
