What Is A Robots.txt File?
Friday, July 25th, 2008I have to consider that there are many people who don’t know what a robots.txt file is. In this article I am going to explain what a robots.txt file is, how to use it and why it is important. I am sure that there are many have heard about a robots.txt file is but might not know exactly what does it do and how will it benefit you? So let’s see if we can answer a few questions for you regarding the in’s and out’s of a robots.txt file.
The Simple Explanation
A robots.txt is a very small file that resides on your server that gives specific instructions to webcrawlers such as the famous Googlebot on which directories and files are allowed to be crawled and indexed. It is important to use a robots.txt file because it puts you (the publisher/webmaster) in total control of where webcrawlers are allow to visit on your website. A more detailed explanation can be found here on Wikipedia, at www.robotstxt.org, and here at Matt Cutt’s blog. (more…)
