Robots File
What is inside of a robots file...
How to write a robots file
Tips about robots file from Google -
Block or remove pages using a robots.txt file
robots.txt checker
From a handful online robots.txt checker tools tested no one was able for real life situations.
To test a robots.txt file go Google Webmasters Tools > Site configuration > Crawler access
robots.txt examples
Standard robots.txt file. Even if there is nothing to tell robots, the good practice is to have a robots file with some minimum robots disallow content
User-agent: * Disallow:
Do not pay attention to category pages. On example.com/this/that/product?item=1700 pages in folders /this/ and /that/ will not be indexed.
User-agent: * Disallow: /*/
if not *_* and is ? then noindex.
User-agent: * Allow: /*_* Disallow: /*?*
Robot, calm down and wait two seconds before you come again.
Crawl-delay: 2
robots.txt sitemaps
Sitemap: http://www.example.com/sitemap.xml.gz Sitemap: http://www.example.com/sitemap.xml Sitemap: http://www.example.com/sitemap-2.xml Sitemap: http://www.example.com/search-engine-optimisation/sitemap.xml
2011
Examples of robots file, format of robots.txt file in search engine optimisation.