Monthly Archives: January 2013

Purpose Robots.txtIt contains instructions to search engine robots and helps us to allow or disallow indexing of certain pages or directories. It has other useful functions, which are applied using special directives. If the settings are correct this file is of great importance for indexing of the resource by search engines.

It is a text file located in the root directory of the site and containing instructions intended for the search engines. This file allows you to restrict indexing to any of the sections and pages of Your site, and specify the file path and the primary mirror. Continue reading