A guide to online robots/spiders/web crawlers/web ants

Robots are programs which go off and can trawl through websites and all documents that are referenced within. Most people know them for their indexing service like the Googlebot, but they also have a few other functions including:
  • Indexing
  • HTML/Link Validation
  • "What's new" monitoring
  • Mirroring
To know more about these programs and how to control their visit through your site check out http://www.robotstxt.org/wc/robots.html
comments powered by Disqus