| Author |
Message |
georgie
Guest
Joined:
Posts:
|
#1 · Posted: 18 Aug 2009 20:02
What's the importance of robots.txt?
How does it work?
|
Dr_Boo
Forums Member
Joined: 12 May 2009
Posts: 52
|
#2 · Posted: 19 Aug 2009 00:15
The robots.txt file is used to tell search engines how to handle your site when their bots/spiders crawl your website. You can specify pages or directories on your site that should be ignored by the bots and not indexed. This comes in handy when you have subfolders for pictures or scripts that you don't want showing up on search results.
You can also specify the path to your sitemap file (if you have one) in the robots.txt file. This will help the bot find all the pages in your site.
I'm pretty new at this too, so if my info is incorrect, hopefully one of the more experienced members *cough*Newbie Shield*cough* will come along and make corrections. 
Good luck to you!
__________________
|
Vishal P. Rao
Joined: 23 Jun 2005
Posts: 1305
|
#3 · Posted: 19 Aug 2009 01:49
Right on Dr_Boo!
Here's the content of a sample robots.txt file:
User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
Disallow: privacy.html
User-agent: * means all search engine bots.
Disallow: /images/
Disallow: /cgi-bin/
Disallow: privacy.html
means the search engine bots should not index sub-directories images and cgi-bin and file privacy.html
__________________
|
mynetworktoday
Forums Member
Joined: 19 Aug 2009
Posts: 1
|
#4 · Posted: 19 Aug 2009 06:31
Thanks for the information. I'm working with DotNetNuke. I know nothing about robots. Could you please direct me to a good location to learn other than google?
__________________
|
Vishal P. Rao
Joined: 23 Jun 2005
Posts: 1305
|
#5 · Posted: 20 Aug 2009 00:41
Here's where you can learn more about robots.txt:
http://www.robotstxt.org/robotstxt.html
__________________
|
Maurice
Guest
Joined:
Posts:
|
#6 · Posted: 26 Aug 2009 02:22
Use robots.txt to hide landing pages that you don;t want competitors to see.....
|
josaphlewis
Guest
Joined:
Posts:
|
#7 · Posted: 15 Jul 2010 23:33
Robots.txt is just a regular text file saved on a website. On request, the specified robots will ignore specified files or directories in their search. It is an algorithm. If it is not there on your site, Google will not crawl it.
|
newbiz02
Forums Member
Joined: 24 Jul 2008
Posts: 167
|
#8 · Posted: 1 Oct 2010 10:25 · Edited by: newbiz02
I use robots.txt to hide the plugins folder on some of my sites so
that google doesn't know that they are *auto*--=blogs. lol
I mean the sites look great so there is no reason to suspect
anything 
__________________
The FAST TRACK CASH is everything you need to know about Clickbank and Affiliate marketing. Learn From the Beginning. |
shannonpaul2
Forums Member
Joined: 30 Sep 2010
Posts: 5
|
#9 · Posted: 4 Oct 2010 02:24
Robots.txt is exactly a normal text file saved on a website. For the asking, the conditioned robots will ignore defined files or directories in their search.
__________________
|
Ron S
Guest
Joined:
Posts:
|
#10 · Posted: 24 Oct 2010 07:20 · Edited by: Ron S
refer to http://www.robotstxt.org/robotstxt.html
|
Justine
Forums Member
Joined: 7 Nov 2010
Posts: 41
|
#11 · Posted: 16 Nov 2010 13:56
Unfortunately the link doesn't work "Internal Server Error"...
__________________
Justyna Bizdra, author of Traffic Ten - step-by-step Traffic Generation Blueprint |
