I've got a robots.txt, and a script that loops to infinity.
Actually, it's a useful page on the server, there's a list that can be
ordered two ways, and switching from one to the other increments a parameter
at the end of the invocation.
A robot has no business reading that specific page in the
On 7/25/06, Mike Erdely [EMAIL PROTECTED] wrote:
prad wrote:
what is the best way to stop those robots and spiders from getting in?
Someone on this list (who can reveal themselves if they want) has a
pretty good setup to block disrespectful robots.
They have a robots.txt file that specifies a
On 7/25/06, prad [EMAIL PROTECTED] wrote:
what is the best way to stop those robots and spiders from getting in?
The sure way to stop robots and spiders is to shut down your web
server. I don't suppose that's the answer you're looking for.
Treat malicious robots as malicious/unwelcome users.
prad wrote:
what is the best way to stop those robots and spiders from getting in?
Someone on this list (who can reveal themselves if they want) has a
pretty good setup to block disrespectful robots.
They have a robots.txt file that specifies a Disallow: /somedir/.
Anyone that actually
From: [EMAIL PROTECTED]
what is the best way to stop those robots and spiders from getting in?
.htaccess?
robot.txt and apache directives?
find them on the access_log and block with pf?
i should also ask whether it is a good idea to block robots
in the first place
since some do help
5 matches
Mail list logo
