Robots.txt File

You can control Search Engine Crawlers (Search Bots) with a file called 'robots.txt' which sits in your websites root folder (the root folder of a website is the same as the index page e.g. www.yourdomain.com/robots.txt).

The Robots.txt file is a file in which you list the pages which you do and do not want Search Engines to visit and index. By default, Search Engines will crawl and index your web pages so if you have pages you do not wish to appear in SERPs then you will need to specify which folders/pages you wish to hide. The Robots.txt file is able to control Search Engines individually or in one go.

Security:

If you are going to use the Robots.txt file then you need to be aware of a potential security issue. The Robots.txt is open to anybody to view and is always in the same location so potential hackers can load your Robots.txt to view which pages you do not want the world to see, giving them the location to your private files.

If you have sensitive files on your website which you do not want people to view or download then you should not rely on the Robots.txt file as it does not block access to these files, it simply tells search engines not to index these pages. To secure your files you should use Server-Side Scripting such as ASP or PHP. Alternatively you could use the server to put a password on the folder to protect them from unauthorised access. Another option would be to put the files below the root and load the file via an ASP or PHP script.

Example:

Below is an example of what a Robots.txt file might look like:

User-Agent: *
Disallow: /private/
Disallow: /design/
Disallow: /private/
Disallow: /images/
Disallow: /contact-us.asp

User-Agent: google-bot
Disallow: /forum/register.asp

User-Agent: msnbot
Disallow: /

User-Agent: slurp
Disallow: /

To Disallow all Search Bot you would do the following:

User-Agent: *
Disallow: /

To Disallow a specific Search Engines Bot you would do the following:

User-Agent: google-bot
Disallow: /

Written By: S2H, 24th January 2008

Share This Page

Advertisement