Use of robots.txt file in Organic SEO
With the robot.txt file you can instruct the search engines wich folders and pages search engines are allowed to index. As with google sitemaps and the use of the nofollow attribute you can control distribution of page rank. Do notice that the robots.txt file is a public document that anyone who desires to can view. The robots.txt file should not be used for website security purposes as it is insufficient for this and may actually reveal location of information you desire to keep private.
The robots.txt file standard only specifies what to "disallow" the search engines from indexing. Although some search engines also follows use of "allow" it is not recommended as not all search engines follow this and it is not part of the robots.txt standard.
In organic search optimisation the robots.txt file can be used to exclude files that you must have on your website, while they do not bring revenue for your business, such files can be privacy polices, terms of use etc. Thus you might not want to distribute PR (page rank) to these pages and they are neither likely to bring direct traffic to your website. Relevant 1 search engine optimization consultants is a organic seo firm based in London UK. For more information on the use of the robots.txt file contact Relevant 1 SEO today.

|