Quote:
Originally Posted by yonghs
Since the subdomain is essentially a folder.....how about putting a robots.txt file in the domain root with the following content:
User-agent: *
Disallow: /subdomainfolder/
OR use htaccess to password protect the folder? Comprehensive guide to .htaccess- password protection |
Oh! I forgot about robots.txt!! Hah. I have it to control the spider from crawling to my feeds... htaccess is definitely not a good idea because I am lazy to login every time I access the subdomain.
Quote:
Originally Posted by mysticmind
User-agent: *
Disallow: all
|
And I am not suppose to put that in robots.txt in domain root, right? That would be disallowing the search engine to crawl in my domain right?
Just wondering how effective this robots.txt is. Is it a Google-only tool, or all crawlers and spiders read robots.txt as well? If I am not mistaken, the link rel=nofollow applies to google only, so I am wondering if robots.txt is the same.