A file placed in the root folder of a website (for example www.example.com/robots.txt) to give instructions to bots. A common usage is to instruct search engines to ignore certain files or directories. The robots.txt file is advisory and relies upon the creator of the bot to follow the instructions.
Some sample robots.txt files:
Allows all bots to visit all files within the site. The * is a wildcard character so all bots are allowed.
User-agent: * Disallow:
Instructs all bots not to access the entire site.
User-agent: * Disallow: /
Instructs bots to ignore the cgi-bin and images directory of the site and also file.html within sample directory.
User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /sample/file.html
Instructs Googlebot not to visit the private directory. Other bots have their own user agent and will disregard this instruction.
User-agent: googlebot Disallow: /private/
See also: Bot