RE: NANOG Digest, Vol 32, Issue 25

[Scott Weeks]

--- m.ho...@hotze.com wrote:
From: Martin Hotze 

I have a private website; I don't want the site to be listed or content found 
via a search engine. I want to be able to give the URL out to friends etc. but 
I don't want all of the world hotlink or whatever[...]
--


Don't put links on the main page.  Put up example.com with only   
or nothing even.  Then your friends have to know to go to 
example.com/mypage.html but the web crawlers never know about mypage.html 
because there's no link on the top page.

scott

RE: NANOG Digest, Vol 32, Issue 25

[Martin Hotze]

> -Original Message-
> Message: 3
> Date: Wed, 8 Sep 2010 10:24:38 +
> From: Nathan Eisenberg 
> Subject: RE: yahoo crawlers hammering us
> To: "nanog@nanog.org" 
> Message-ID:
>   <8c26a4fdae599041a13eb499117d3c28164ac...@ex-mb-
> 1.corp.atlasnetworks.us>
> 

> Customers don't want to deliver their content to search engines?  That
> seems silly.

I have a private website; I don't want the site to be listed or content found 
via a search engine. I want to be able to give the URL out to friends etc. but 
I don't want all of the world hotlink or whatever - sure, I can password 
protect the site, but for now I have a ton of rewrite conditions in place.

#m