prad wrote:
what is the best way to stop those robots and spiders from getting in?
Someone on this list (who can reveal themselves if they want) has a pretty good setup to block "disrespectful" robots.
They have a robots.txt file that specifies a "Disallow: /somedir/". Anyone that actually GOES into that directory gets blocked by PF.
It'd be pretty easy to parse your /var/www/logs/access_log for accesses of "/somedir/" and have them added to a table.
-ME
