> > Namely, the [EMAIL PROTECTED] file, configured to exclude everything?
> It's coded into FProxy. I put it there because people were having
> problems with spiders overloading their nodes.
Understood and respected, as per IRC discussion.
It's a mystery to me how roving web spiders found people's fproxy ports to
begin with. Perhaps from the 'Referrer' fields in web server logs?
I've worked around this restriction by hardwiring FwProxy to 404 on any hit
to robots.txt.
BTW, I use a personal spider to crawl some of the freesites, to enable me to
surf them faster.
Cheers
David
----- Original Message -----
From: "Mark J. Roberts" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Friday, June 15, 2001 16:52
Subject: Re: [freenet-chat] OK - fess up
> On Fri, Jun 15, 2001 at 05:40:56PM +1200, David McNab wrote:
> > Alright alright,
> >
> > Who planted the &*$%&@ spider-repellant in Freenet?
> >
> > Namely, the [EMAIL PROTECTED] file, configured to exclude everything?
>
> It's coded into FProxy. I put it there because people were having
> problems with spiders overloading their nodes.
>
>
> --
> "Is our system--was it invented by scientists?"
> "No. If scientists had invented it, they'd have tried it out first on
hamsters."
> Mark Roberts | [EMAIL PROTECTED]
>
> _______________________________________________
> Chat mailing list
> [EMAIL PROTECTED]
> http://lists.freenetproject.org/mailman/listinfo/chat
>
_______________________________________________
Chat mailing list
[EMAIL PROTECTED]
http://lists.freenetproject.org/mailman/listinfo/chat
