It is used to control what parts or pages on your website which you don't want search engines to visit and index. Google robots.txt for more details.