Author |
Message |
Gobala Krishnan Pro
Joined: 17 Jun 2004 Posts: 57 Location: Malaysia
|
Posted: Wed Jun 30, 2004 4:56 pm Post subject: robots.txt |
|
|
Hi everyone,
Could anyone advice me whether it's necessary to have a robots.txt file in your site directory?
If it is, then what should it contain?
Thanks,
Bala
|
|
Back to top |
|
|
Vishal P. Rao Forum Admin
Joined: 02 May 2004 Posts: 104
|
Posted: Thu Jul 01, 2004 4:05 am Post subject: |
|
|
Hi Bala,
It's not at all necessary to have a robots.txt file in your site directory. Robots.txt is simply used to specify which areas of your site should be spidered by search engines and which should not be. You can also explicitly exclude certain spiders from spidering your site.
To learn more about it, visit the following url:
http://www.searchengineworld.com/robots/robots_tutorial.htm
Best wishes
Vishal
|
|
Back to top |
|
|
|