Robots.txt is a text file you put on your site to tell search engine robots which page you want them not to visit. Using robots.txt file is not the way of preventing your site from attackers or creating a firewall, it is just like putting note i.e. DO NOT ENTER on a unlocked door, it doesn’t provide any protection from thieves, but good guys will not open the door. So if your site really have some sensitive data, its advisable to use robots file in-order to prevent them from getting indexed.
The location of robots.txt is very very important, it must be placed in the root directory of your website, if it is placed somewhere else then search engine will not be able to find that and hence result in indexing of your complete site.
How do search engine actually trace your robots file in the root directory…???
When search engine robots enter any site, say for example www.parasbabbar.in, what they will look for is www.parasbabbar.in/index.php and then before indexing the site they will search for robots.txt file, as www.parasbabbar.in/robots.txt, they will replace /index.php with /robots.txt. If they find robots.txt file, they will follow the instruction written is that else if they don’t find any such file then they’ll simply shift back to www.parasbabbar.in/index.php and index your complete site.
Structure of a robots.txt file
The structure of a robots.txt is pretty simple and barely flexible – it is an endless list of user agents and disallowed files and directories. Basically, the syntax is as follows:
User-agent:
Disallow:
“User-agent” are search engines’ crawlers and disallow: lists the files and directories to be excluded from indexing. In addition to “user-agent:” and “disallow:” entries, you can include comment lines – just put the # sign at the beginning of the line:
# All user agents are disallowed to see the /temp directory.
User-agent: *
Disallow: /temp/
NOTE: Here * stands for all user agent and the directory you want to disallow must always be within “/directory to be disallowed/” (symbol slash, don’t put double inverted commas, see the example above).
For beginners if you don’t have an idea of creating a robots.txt file, kindly use online tools to create robots.txt file, follow the link below….
