x27;t done!
- Original Message -
From: "Randell B Adkins" <[EMAIL PROTECTED]>
To: "CF-Community" <[EMAIL PROTECTED]>
Sent: Thursday, July 17, 2003 11:06 AM
Subject: Re: Search Engine / robot.txt
| Then what is the best way to prohibit directories from
| being
]>
Sent: Thursday, July 17, 2003 8:11 AM
Subject: Re: Search Engine / robot.txt
| oi Randell!!
|
| yeah, I don't even know if they will index the url but they go
no further
than that(if they
| use the robots.txt file)
|
|
|
|
|
| Thursday, J
://www.forta.com/cf/isp/isp.cfm?isp_id=772
==
If you are not satisfied with my service, my job isn't done!
- Original Message -
From: "Critz" <[EMAIL PROTECTED]>
To: "CF-Community" <[EMAIL PROTECTED]>
Sent: Thursday, July
oi Randell!!
yeah, I don't even know if they will index the url but they go no further than
that(if they
use the robots.txt file)
Thursday, July 17, 2003, 8:57:55 AM, you wrote:
RBA> Thanks also by using:
RBA> User-agent: *
RBA> Disallow: /
The file must be plural as in "robots.txt" not "robot.txt"
Sample configuration:
==
User-agent: *
Disallow: /secure/
Disallow: /images/
Disallow: /db/
Disallow: /data/
Disallow: /database/
Disallow: /related/
Disallow: /reviews/
Disallow: /cfide/
Disallow: /flashcom
# don'
Thanks also by using:
User-agent: *
Disallow: /
That would allow all BOTs to hit your site but not index any pages.
Is that right? so in theory it would index just the URL and the
META tag data?
>>> [EMAIL PROTECTED] 07/17/03 08:49AM >>>
oi Randell!!
http://www.google.com/search?hl=en&ie=UT
Here is one that I have used in the past:
http://www.payperclicksearchengines.com/creating-robots-txt-file.shtml
> -Original Message-
> From: Randell B Adkins [mailto:[EMAIL PROTECTED]
> Sent: Thursday, July 17, 2003 8:44 AM
> To: CF-Community
> Subject: Search Engine / robot.txt
>
>
oi Randell!!
http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=how+to+build+a+robots.txt&btnG=Google+Search
first return looks like it might splain it...
mine all just say:
User-agent: *
Disallow: /
:)
Thursday, July 17, 2003, 8:43:30 AM, you wrot
