Re: block bots and spiders

2011-07-12 Thread Phang Mulianto
yes Bruno, that's what i mean, so this is doable.. and also can prevent DDOS attack to your core web server... On Tue, Jul 12, 2011 at 6:28 PM, bruno desthuilliers < bruno.desthuilli...@gmail.com> wrote: > On Jul 12, 11:57 am, Phang Mulianto wrote: > > hi.. > > > > why not block the spiders in t

Re: block bots and spiders

2011-07-12 Thread bruno desthuilliers
On Jul 12, 11:57 am, Phang Mulianto wrote: > hi.. > > why not block the spiders in the web server environtment . > i think it is better than has to put the code in the apps. > > is it possible ? because i had read that some web server can block > unnecesary request before reach the real web server

Re: block bots and spiders

2011-07-12 Thread Phang Mulianto
hi.. why not block the spiders in the web server environtment . i think it is better than has to put the code in the apps. is it possible ? because i had read that some web server can block unnecesary request before reach the real web server..like nginx, lighttpd, etc.. On Tue, Jul 12, 2011 at

Re: block bots and spiders

2011-07-12 Thread Jonas Geiregat
> Hi all, > > has someone an effective way to block away bots and spiders? > There are so many ignoring robots.txt - besides facebook and tons of > Java/xyz clients there are many more illegal spiders around. > > So does anyone have a performance effective way to block them out? > Or do you thi

block bots and spiders

2011-07-12 Thread Henrik Genssen
Hi all, has someone an effective way to block away bots and spiders? There are so many ignoring robots.txt - besides facebook and tons of Java/xyz clients there are many more illegal spiders around. So does anyone have a performance effective way to block them out? Or do you think - if performanc