robots.txt

[Gobala Krishnan]

Hi everyone,

Could anyone advice me whether it's necessary to have a robots.txt file in your site directory?

If it is, then what should it contain?

Thanks,

Bala

[Vishal P. Rao]

Hi Bala,

It's not at all necessary to have a robots.txt file in your site directory. Robots.txt is simply used to specify which areas of your site should be spidered by search engines and which should not be. You can also explicitly exclude certain spiders from spidering your site.

To learn more about it, visit the following url:
http://www.searchengineworld.com/robots/robots_tutorial.htm

Best wishes

Vishal