I plan to create a subdomain where I have the exact copy of my live and working site, because I plan to play with Wordpress 2.3 before doing upgrade to my live site, and probably will start playing with Wordpress SVN. Now, if I do an exact copy of content, that means that the search engine will start penalizing my site. So, what can I do to prevent any bots, crawlers or spider from crawling to this copy of my site?

Thanks in advance. I do plan to play around in a local environment, but I would prefer to test it in a Linux environment e.g. my webhost.

__________________
Don't Ask Me Why I Joined This Forum
GeminiGeek's Online Journal

Since the subdomain is essentially a folder.....how about putting a robots.txt file in the domain root with the following content:

User-agent: *
Disallow: /subdomainfolder/

OR use htaccess to password protect the folder? http://www.javascriptkit.com/howto/htaccess3.shtml

__________________
Unique job site
Penang Restaurants, Food and Entertainment Directory

User-agent: *
Disallow: all

__________________
Personal's Blog! - Malaysian Artist!

Oh! I forgot about robots.txt!! Hah. I have it to control the spider from crawling to my feeds... htaccess is definitely not a good idea because I am lazy to login every time I access the subdomain.

And I am not suppose to put that in robots.txt in domain root, right? That would be disallowing the search engine to crawl in my domain right?

Just wondering how effective this robots.txt is. Is it a Google-only tool, or all crawlers and spiders read robots.txt as well? If I am not mistaken, the link rel=nofollow applies to google only, so I am wondering if robots.txt is the same.

__________________
Don't Ask Me Why I Joined This Forum
GeminiGeek's Online Journal

The robots.txt is a standard that all major(proper) search engines should comply .... unless of course there are smaller rogue search engines that may choose to ignore the file. The most effective is still htaccess password protect

It should be placed at the ROOT. You just state the folder to disallow in the robots.txt file.

http://www.maindomain.com/robots.txt

User-agent: *
Disallow: /subfolder/

*****************************************
The above works nicely if you want to block indexing of a folder... but since you are using subdomain http://subdomain.maindomain.com .............. you might put another robots.txt inside the folder http://subdomain.maindomain.com/robots.txt ............... but this time i'm not sure what should be the content???? Is it as below:

User-agent: *
Disallow: all

So, if you use subdomain. You may want to use 2 robots.txt files to be sure. Pease confirm, as I've not used the combination of robots.txt AND subdomain before.

__________________
Unique job site
Penang Restaurants, Food and Entertainment Directory

I googled and found this, and it should answered all the questions:
How do I use a robots.txt file?

That means I only have to put rules in robots.txt for both domain root and subdomain root. Thanks for the answers peoples

__________________
Don't Ask Me Why I Joined This Forum
GeminiGeek's Online Journal

alternatively... if you are using some sort of template ... you can direct insert the meta code below to your html header

__________________
Freelance Jobs Post
Get Motivated Daily

google bot respect the robot.txt
but yahoo slurp etc is SO SO..

__________________
Personal's Blog! - Malaysian Artist!

You can block spiders with a line in robots.txt or use the function in WP which disallows spiders to index the blog.

__________________
Bid Your Web | Top Bidders List | Mishre Directory | VobV Directory

I use all the methods here listed, and so far I think that subdomain isn't indexed. How do I check if my subdomain is indexed?

__________________
Don't Ask Me Why I Joined This Forum
GeminiGeek's Online Journal

Copy about one sentence of text from your subdomain's content .. or the title's text and search in Google/Yahoo. See if the search results shows the url of your subdomain or not.

Or type this in Google search >>> inurl:subdomain.yourdomain.com ... and see if any of the sudomain pages are shown.

__________________
Unique job site
Penang Restaurants, Food and Entertainment Directory

If you don't link to your subdomain it won't be spidered anyway.

Not all spiders respect robots.txt, but put a "disallow:all" in the root of the subdomain because a subdomain is seen as a separate website, and also in the main domain as disallow: /subdomaindirectory/