RE: OT: Keep search engine spyders from following links
if i remember correctly you can put all the pages you dont want into a folder, then tell the spider not to index this folder using robots.txt. This would prevent all robots from indexing a folder called private: User-agent: * Disallow: /private/ You can also specify pages: User-agent: * Disallow: /private/secret.html HTH will -Original Message- From: Craig Dudley [mailto:[EMAIL PROTECTED]] Sent: 11 December 2001 11:51 To: CF-Talk Subject: RE: OT: Keep search engine spyders from following links Doesn't robots.txt do the exact job you are after? I don't know the systnax, but any visting spider will most likely request that file 1st, you can tell the spider what to index, and what not to index. Craig. -Original Message- From: Jim McAtee [mailto:[EMAIL PROTECTED]] Sent: 11 December 2001 00:20 To: CF-Talk Subject: Re: OT: Keep search engine spyders from following links Joe, I'll try it out. Since the links all appear on CF-generated pages (pulled from database tables) I think I should be able to just Unicode the links within the CF templates themselves. Is there a CF Unicode tag or function somewhere, or do Unicode'd string simply use the ASCII codes for each character? Jim - Original Message - From: "Joseph DeVore" <[EMAIL PROTECTED]> To: "CF-Talk" <[EMAIL PROTECTED]> Sent: Monday, December 10, 2001 5:05 PM Subject: RE: OT: Keep search engine spyders from following links > Jim, > > It is possible to UNICODE HREFs and a lot of bots ignore these types of > links. I use unicoded HREFs for all of my email addresses so spam bots don't > grab them. I haven't tested unicoded HREFs in all browsers, just Netscape > 4.08 and IE 4+ on Windows > > If you want to take a look at a unicoded href you can use this form to > generate them: > > http://cftags.veloxweb.com/unicode/ > > > HTH, > > Joseph DeVore > VeloxWeb Technologies > > > > -Original Message----- > From: Jim McAtee [mailto:[EMAIL PROTECTED]] > Sent: Monday, December 10, 2001 3:35 PM > To: CF-Talk > Subject: Re: OT: Keep search engine spyders from following links > > > IIS. What exactly would the ISAPI application do? > > I suppose within my redirection template I _could_ parse the cgi.useragent > string and neither record nor redirect known spyders. Keeping an up-to-date > list of agents might be a pain, though. I'd rather just tell the syders > (even if only the well-behaved ones) not to follow the link. > > Jim > > > - Original Message - > From: "Alex" <[EMAIL PROTECTED]> > To: "CF-Talk" <[EMAIL PROTECTED]> > Sent: Monday, December 10, 2001 3:39 PM > Subject: Re: OT: Keep search engine spyders from following links > > > > what's your web server? You can write an apache module to do this and I > > would assume you could do the same with ISAPI > > > > On Mon, 10 Dec 2001, Jim McAtee wrote: > > > > > On a couple of web pages, I run links to outside web sites through a CF > > > template that records the click-through along with some browser and > referer > > > stats in a table, then redirects the browser to the target URL. Is > there > > > any way to prevent search engine spyders from following these links? If > I'm > > > not mistaken, there's no way to use a robots.txt to prevent this > behavior, > > > or is there? > > > > > > Thanks, > > > Jim > > ~~ Get the mailserver that powers this list at http://www.coolfusion.com FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: OT: Keep search engine spyders from following links
Doesn't robots.txt do the exact job you are after? I don't know the systnax, but any visting spider will most likely request that file 1st, you can tell the spider what to index, and what not to index. Craig. -Original Message- From: Jim McAtee [mailto:[EMAIL PROTECTED]] Sent: 11 December 2001 00:20 To: CF-Talk Subject: Re: OT: Keep search engine spyders from following links Joe, I'll try it out. Since the links all appear on CF-generated pages (pulled from database tables) I think I should be able to just Unicode the links within the CF templates themselves. Is there a CF Unicode tag or function somewhere, or do Unicode'd string simply use the ASCII codes for each character? Jim - Original Message - From: "Joseph DeVore" <[EMAIL PROTECTED]> To: "CF-Talk" <[EMAIL PROTECTED]> Sent: Monday, December 10, 2001 5:05 PM Subject: RE: OT: Keep search engine spyders from following links > Jim, > > It is possible to UNICODE HREFs and a lot of bots ignore these types of > links. I use unicoded HREFs for all of my email addresses so spam bots don't > grab them. I haven't tested unicoded HREFs in all browsers, just Netscape > 4.08 and IE 4+ on Windows > > If you want to take a look at a unicoded href you can use this form to > generate them: > > http://cftags.veloxweb.com/unicode/ > > > HTH, > > Joseph DeVore > VeloxWeb Technologies > > > > -Original Message----- > From: Jim McAtee [mailto:[EMAIL PROTECTED]] > Sent: Monday, December 10, 2001 3:35 PM > To: CF-Talk > Subject: Re: OT: Keep search engine spyders from following links > > > IIS. What exactly would the ISAPI application do? > > I suppose within my redirection template I _could_ parse the cgi.useragent > string and neither record nor redirect known spyders. Keeping an up-to-date > list of agents might be a pain, though. I'd rather just tell the syders > (even if only the well-behaved ones) not to follow the link. > > Jim > > > - Original Message - > From: "Alex" <[EMAIL PROTECTED]> > To: "CF-Talk" <[EMAIL PROTECTED]> > Sent: Monday, December 10, 2001 3:39 PM > Subject: Re: OT: Keep search engine spyders from following links > > > > what's your web server? You can write an apache module to do this and I > > would assume you could do the same with ISAPI > > > > On Mon, 10 Dec 2001, Jim McAtee wrote: > > > > > On a couple of web pages, I run links to outside web sites through a CF > > > template that records the click-through along with some browser and > referer > > > stats in a table, then redirects the browser to the target URL. Is > there > > > any way to prevent search engine spyders from following these links? If > I'm > > > not mistaken, there's no way to use a robots.txt to prevent this > behavior, > > > or is there? > > > > > > Thanks, > > > Jim > > ~~ Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
Re: OT: Keep search engine spyders from following links
Joe, I'll try it out. Since the links all appear on CF-generated pages (pulled from database tables) I think I should be able to just Unicode the links within the CF templates themselves. Is there a CF Unicode tag or function somewhere, or do Unicode'd string simply use the ASCII codes for each character? Jim - Original Message - From: "Joseph DeVore" <[EMAIL PROTECTED]> To: "CF-Talk" <[EMAIL PROTECTED]> Sent: Monday, December 10, 2001 5:05 PM Subject: RE: OT: Keep search engine spyders from following links > Jim, > > It is possible to UNICODE HREFs and a lot of bots ignore these types of > links. I use unicoded HREFs for all of my email addresses so spam bots don't > grab them. I haven't tested unicoded HREFs in all browsers, just Netscape > 4.08 and IE 4+ on Windows > > If you want to take a look at a unicoded href you can use this form to > generate them: > > http://cftags.veloxweb.com/unicode/ > > > HTH, > > Joseph DeVore > VeloxWeb Technologies > > > > -Original Message- > From: Jim McAtee [mailto:[EMAIL PROTECTED]] > Sent: Monday, December 10, 2001 3:35 PM > To: CF-Talk > Subject: Re: OT: Keep search engine spyders from following links > > > IIS. What exactly would the ISAPI application do? > > I suppose within my redirection template I _could_ parse the cgi.useragent > string and neither record nor redirect known spyders. Keeping an up-to-date > list of agents might be a pain, though. I'd rather just tell the syders > (even if only the well-behaved ones) not to follow the link. > > Jim > > > ----- Original Message - > From: "Alex" <[EMAIL PROTECTED]> > To: "CF-Talk" <[EMAIL PROTECTED]> > Sent: Monday, December 10, 2001 3:39 PM > Subject: Re: OT: Keep search engine spyders from following links > > > > what's your web server? You can write an apache module to do this and I > > would assume you could do the same with ISAPI > > > > On Mon, 10 Dec 2001, Jim McAtee wrote: > > > > > On a couple of web pages, I run links to outside web sites through a CF > > > template that records the click-through along with some browser and > referer > > > stats in a table, then redirects the browser to the target URL. Is > there > > > any way to prevent search engine spyders from following these links? If > I'm > > > not mistaken, there's no way to use a robots.txt to prevent this > behavior, > > > or is there? > > > > > > Thanks, > > > Jim > > ~~ Get the mailserver that powers this list at http://www.coolfusion.com FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: OT: Keep search engine spyders from following links
Jim, It is possible to UNICODE HREFs and a lot of bots ignore these types of links. I use unicoded HREFs for all of my email addresses so spam bots don't grab them. I haven't tested unicoded HREFs in all browsers, just Netscape 4.08 and IE 4+ on Windows If you want to take a look at a unicoded href you can use this form to generate them: http://cftags.veloxweb.com/unicode/ HTH, Joseph DeVore VeloxWeb Technologies -Original Message- From: Jim McAtee [mailto:[EMAIL PROTECTED]] Sent: Monday, December 10, 2001 3:35 PM To: CF-Talk Subject: Re: OT: Keep search engine spyders from following links IIS. What exactly would the ISAPI application do? I suppose within my redirection template I _could_ parse the cgi.useragent string and neither record nor redirect known spyders. Keeping an up-to-date list of agents might be a pain, though. I'd rather just tell the syders (even if only the well-behaved ones) not to follow the link. Jim - Original Message - From: "Alex" <[EMAIL PROTECTED]> To: "CF-Talk" <[EMAIL PROTECTED]> Sent: Monday, December 10, 2001 3:39 PM Subject: Re: OT: Keep search engine spyders from following links > what's your web server? You can write an apache module to do this and I > would assume you could do the same with ISAPI > > On Mon, 10 Dec 2001, Jim McAtee wrote: > > > On a couple of web pages, I run links to outside web sites through a CF > > template that records the click-through along with some browser and referer > > stats in a table, then redirects the browser to the target URL. Is there > > any way to prevent search engine spyders from following these links? If I'm > > not mistaken, there's no way to use a robots.txt to prevent this behavior, > > or is there? > > > > Thanks, > > Jim ~~ Structure your ColdFusion code with Fusebox. Get the official book at http://www.fusionauthority.com/bkinfo.cfm FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
Re: OT: Keep search engine spyders from following links
IIS. What exactly would the ISAPI application do? I suppose within my redirection template I _could_ parse the cgi.useragent string and neither record nor redirect known spyders. Keeping an up-to-date list of agents might be a pain, though. I'd rather just tell the syders (even if only the well-behaved ones) not to follow the link. Jim - Original Message - From: "Alex" <[EMAIL PROTECTED]> To: "CF-Talk" <[EMAIL PROTECTED]> Sent: Monday, December 10, 2001 3:39 PM Subject: Re: OT: Keep search engine spyders from following links > what's your web server? You can write an apache module to do this and I > would assume you could do the same with ISAPI > > On Mon, 10 Dec 2001, Jim McAtee wrote: > > > On a couple of web pages, I run links to outside web sites through a CF > > template that records the click-through along with some browser and referer > > stats in a table, then redirects the browser to the target URL. Is there > > any way to prevent search engine spyders from following these links? If I'm > > not mistaken, there's no way to use a robots.txt to prevent this behavior, > > or is there? > > > > Thanks, > > Jim ~~ Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
Re: OT: Keep search engine spyders from following links
what's your web server? You can write an apache module to do this and I would assume you could do the same with ISAPI On Mon, 10 Dec 2001, Jim McAtee wrote: > On a couple of web pages, I run links to outside web sites through a CF > template that records the click-through along with some browser and referer > stats in a table, then redirects the browser to the target URL. Is there > any way to prevent search engine spyders from following these links? If I'm > not mistaken, there's no way to use a robots.txt to prevent this behavior, > or is there? > > Thanks, > Jim > ~~ Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
