Re: IP based instant throttle?
[EMAIL PROTECTED] (Randal L. Schwartz) wrote: I'm told that my CPU throttler was used at etoys.com for a similar purpose, and permitted them to keep from losing millions of dollars of revenue due to people spidering their catalog. That's correct, although it was actually a bunch of DoS attacks that we were using it against. We modified it to just count hits, and skip the CPU stuff. It worked well across a cluster, using NFS to share the files with the hit data in them. Since it's an access handler, it was easy to just turn it on for specific URLs where repeated access hurts. This avoids any issues with parallel fetches for images. We also used cookies (verified similarly to the ticket scheme in the Eagle Book) as the primary identifier and only fell back to IP if there was no valid cookie. This can help with the proxy (i.e. many users with one IP) problem, but you still have to make exceptions for things like AOL proxies that can blast you with legitimate traffic. If thousands of AOL users all click on an ad banner in the same 10 seconds, you don't want to ban them. - Perrin
Re: IP based instant throttle?
Ken == Ken Williams [EMAIL PROTECTED] writes: Ken [EMAIL PROTECTED] (Randal L. Schwartz) wrote: It would be pretty simple, basing it on my CPU-limiting throttle that I've published in Linux Magazine http://www.stonehenge.com/merlyn/LinuxMag/col17.html. Just grab a flock on the CPU-logging file in the post-read-request phase instead of writing to it. If you can't get the flock, reject the request. Release the flock by closing the file in the log phase. But this'd sure mess up my ordinary visit to you, since my browser makes 4 connections in parallel to fetch images, and I believe most browsers do that these days. Ken I was thinking about that too, and concluded that you'd only want to Ken throttle the back-end server in a 2-server setup. That would usually Ken (save for subrequests) only be 1 request throttled per page-load. I Ken tend not to care about the front-end, because overload is rarely a Ken problem there. Well, if the reason you're throttling is to block excessive usage of the machine, the full monty of CPU limiting will do that just fine, since images are delivered quickly, but anything that eats CPU starts pushing the counter up to the max. That's why I have my CPU throttler, and it worked fine to prevent me from being slashdotted that one day I was mentioned there. I'm told that my CPU throttler was used at etoys.com for a similar purpose, and permitted them to keep from losing millions of dollars of revenue due to people spidering their catalog. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: IP based instant throttle?
[EMAIL PROTECTED] said: Well, if the reason you're throttling is to block excessive usage of the machine, the full monty of CPU limiting will do that just fine, one kind of DOS would not be caught by looking at CPU usage, it is one that I have experienced a number of times, namely the use of some misconfigured offline browsing tool that would just open as many concurrent connections as it can until it has read all pages on your server. I don't know whether some of these tools are misconfigured out of the box, or users changed the settings. Some idiots do that even behind a modem, so the limit is not CPU but bandwidth, as all of these connections go through the same slow wire. Your CPU will then be mostly idle, with full IP output queues and all Apache processes in the W sate. As soon one of the requests times out, the tool opens a new one. It should be easy to hack Apache::SpeedLimit to count concurrent accesses instead of number of accesses over a certain time and lock out the client if it reaches some max number. Is this the best way to do this or are there better ideas? Roman Maeder
RE: IP based instant throttle?
You'd want to look at the scoreboard. mod_throttle_access (http://www.fremen.org/apache/) does this function based upon URI. The only changes would be to base it upon client IP and change the scope to allow it to be outside a Directory block. It would be a minor change. Regards, Christian -Original Message- From: Justin [mailto:[EMAIL PROTECTED]] Sent: Thursday, June 07, 2001 7:51 PM To: [EMAIL PROTECTED] Subject: IP based instant throttle? Does anyone see the value in a Throttle module that looked at the apache parent status block and rejected any request where another child was already busy servicing *that same IP* ? (note: the real IP is in the header in a backend setup so it is not possible to dig it out across children without creating another bit of shared memory or using the filesystem?). I'm still finding existing throttle modules do not pickup and block parallel or fast request streams fast enough .. ok there are no massive outages but 10 seconds of delay for everyone because all demons are busy servicing the same guy before we can conclude we're being flooded is not really great.. modperl driven forums (or PHP ones even) can be killed this way since there are so many links on one page, all active.. thanks for any thoughts on this. -Justin
Re: IP based instant throttle?
good ideas, thanks. as someone said its cloggage on the backend due to either SQL server contention or more likely largish pages draining to the user even with all the buffers en-route helping to mitigate this. you can't win : if they are on a modem they can tie up 8 modperl demons, and if they are on a cable modem they can disrupt your SQL server creating select/insert locks and yet more stalled stuff. A cable modem, user could request 1mbit/s of dynamic content.. thats a big ask.. Since the clogging is not images (that is hopefully handled by an image server like mathopd), its modperl pages, I'm going to try a timed ban triggered by parallel requests from a single IP. And yes it does happen often enough to annoy.. (often might be two or three times a day even though as a percentage of uniques its very tiny) since many of the culprits don't even know what they've got installed on their PCs and are on dhcp addresses and probably never return anyway IP bans after the event are never any good and may hit the next user who picks up the IP. -Justin On Thu, Jun 07, 2001 at 07:34:45PM -0700, Randal L. Schwartz wrote: Justin == Justin [EMAIL PROTECTED] writes: Justin Does anyone see the value in a Throttle module that looked at Justin the apache parent status block and rejected any request where Justin another child was already busy servicing *that same IP* ? Justin (note: the real IP is in the header in a backend setup so it Justin is not possible to dig it out across children without Justin creating another bit of shared memory or using the filesystem?). Justin I'm still finding existing throttle modules do not pickup and Justin block parallel or fast request streams fast enough .. ok there are Justin no massive outages but 10 seconds of delay for everyone because Justin all demons are busy servicing the same guy before we can conclude Justin we're being flooded is not really great.. modperl driven forums Justin (or PHP ones even) can be killed this way since there are so Justin many links on one page, all active.. It would be pretty simple, basing it on my CPU-limiting throttle that I've published in Linux Magazine http://www.stonehenge.com/merlyn/LinuxMag/col17.html. Just grab a flock on the CPU-logging file in the post-read-request phase instead of writing to it. If you can't get the flock, reject the request. Release the flock by closing the file in the log phase. But this'd sure mess up my ordinary visit to you, since my browser makes 4 connections in parallel to fetch images, and I believe most browsers do that these days. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: IP based instant throttle?
I'm glad I haven't got your user.. I think most any site on the net can be brought to its knees by, for example, stuffing its site search form with random but very common words and pressing the post button and issuing these requests as frequently as possible from a long list of open proxies.. or how about repeatedly fetching random pages of very old postings such that the SQL server index/table memory cache becomes useless... nightmare ;) All one can do is respond with appropriate measures at the time of the attack, which is why working in modperl is cool because of the ease with which one can patch in defenses and modify them while live. Writing a short script that takes the last 20 minutes of access_log and automatically identifies abuse based on frequency of request, IPs and URL patterns, and drops the route to those IPs is a good start.. to have this auto-triggered from a site monitoring script is even better. -Justin On Thu, Jun 07, 2001 at 08:37:04PM -0700, Jeremy Rusnak wrote: Hi all, Just thought I would add my two cents...I run an online gaming site and the end users often decide to mess with our systems. We service a pretty juvenile crowd in some regards. So there definately is a need for better protection from floods. I've had one user in particular who has been attacking our site regularly for the past year and a half. He'll setup a couple machines with scripts to call forum posting scripts with random information passed into them. He'll call a generic CGI script ten times a second because he can tell it slows down the server. He'll bombard the servers with huge UDP packets. He bulks E-mails viruses and zombies to our usersIt's insane. In short, this is a big issue for sites that get a decent amount of traffic. Better flood protection is always a good thing. We've got a great Cisco firewall that stops a lot of these kinds of things, but this fellow discovered open proxies and has been a pain ever since. He has a script that bombards us using a different proxy every five seconds. (There are lists out there updated in real-time with hundreds of open proxies thanks to the privacy advocates on the Net.) By the way, the guy is in Spain so the government can't/won't do anything. WE've blocked have the providers in Spain as a result of him getting a new IP when he has been stupid enough to use a real IP. So I would suggest that rate limiting based on IP address is a start, but it isn't the end all. You've got to have a big bag of tricks. Don't just look for one solution. Jeremy -Original Message- From: Martin Redington [mailto:[EMAIL PROTECTED]] Sent: Thursday, June 07, 2001 6:08 PM To: [EMAIL PROTECTED] Cc: Justin Subject: Re: IP based instant throttle? Do you get flooded that frequently that this is an issue? I've seen DOS, and various buffer overflows etc. in the real world, but I've never seen this. Unless its happening very often, I would have thought that some monitoring and a 2am Deny from ip in your httpd.conf would be enough ... On Friday, June 8, 2001, at 01:50 am, Justin wrote: Does anyone see the value in a Throttle module that looked at the apache parent status block and rejected any request where another child was already busy servicing *that same IP* ? (note: the real IP is in the header in a backend setup so it is not possible to dig it out across children without creating another bit of shared memory or using the filesystem?). I'm still finding existing throttle modules do not pickup and block parallel or fast request streams fast enough .. ok there are no massive outages but 10 seconds of delay for everyone because all demons are busy servicing the same guy before we can conclude we're being flooded is not really great.. modperl driven forums (or PHP ones even) can be killed this way since there are so many links on one page, all active.. thanks for any thoughts on this. -Justin
Re: IP based instant throttle?
Justin == Justin [EMAIL PROTECTED] writes: Justin Does anyone see the value in a Throttle module that looked at Justin the apache parent status block and rejected any request where Justin another child was already busy servicing *that same IP* ? Justin (note: the real IP is in the header in a backend setup so it Justin is not possible to dig it out across children without Justin creating another bit of shared memory or using the filesystem?). Justin I'm still finding existing throttle modules do not pickup and Justin block parallel or fast request streams fast enough .. ok there are Justin no massive outages but 10 seconds of delay for everyone because Justin all demons are busy servicing the same guy before we can conclude Justin we're being flooded is not really great.. modperl driven forums Justin (or PHP ones even) can be killed this way since there are so Justin many links on one page, all active.. It would be pretty simple, basing it on my CPU-limiting throttle that I've published in Linux Magazine http://www.stonehenge.com/merlyn/LinuxMag/col17.html. Just grab a flock on the CPU-logging file in the post-read-request phase instead of writing to it. If you can't get the flock, reject the request. Release the flock by closing the file in the log phase. But this'd sure mess up my ordinary visit to you, since my browser makes 4 connections in parallel to fetch images, and I believe most browsers do that these days. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
RE: IP based instant throttle?
Hi all, Just thought I would add my two cents...I run an online gaming site and the end users often decide to mess with our systems. We service a pretty juvenile crowd in some regards. So there definately is a need for better protection from floods. I've had one user in particular who has been attacking our site regularly for the past year and a half. He'll setup a couple machines with scripts to call forum posting scripts with random information passed into them. He'll call a generic CGI script ten times a second because he can tell it slows down the server. He'll bombard the servers with huge UDP packets. He bulks E-mails viruses and zombies to our usersIt's insane. In short, this is a big issue for sites that get a decent amount of traffic. Better flood protection is always a good thing. We've got a great Cisco firewall that stops a lot of these kinds of things, but this fellow discovered open proxies and has been a pain ever since. He has a script that bombards us using a different proxy every five seconds. (There are lists out there updated in real-time with hundreds of open proxies thanks to the privacy advocates on the Net.) By the way, the guy is in Spain so the government can't/won't do anything. WE've blocked have the providers in Spain as a result of him getting a new IP when he has been stupid enough to use a real IP. So I would suggest that rate limiting based on IP address is a start, but it isn't the end all. You've got to have a big bag of tricks. Don't just look for one solution. Jeremy -Original Message- From: Martin Redington [mailto:[EMAIL PROTECTED]] Sent: Thursday, June 07, 2001 6:08 PM To: [EMAIL PROTECTED] Cc: Justin Subject: Re: IP based instant throttle? Do you get flooded that frequently that this is an issue? I've seen DOS, and various buffer overflows etc. in the real world, but I've never seen this. Unless its happening very often, I would have thought that some monitoring and a 2am Deny from ip in your httpd.conf would be enough ... On Friday, June 8, 2001, at 01:50 am, Justin wrote: Does anyone see the value in a Throttle module that looked at the apache parent status block and rejected any request where another child was already busy servicing *that same IP* ? (note: the real IP is in the header in a backend setup so it is not possible to dig it out across children without creating another bit of shared memory or using the filesystem?). I'm still finding existing throttle modules do not pickup and block parallel or fast request streams fast enough .. ok there are no massive outages but 10 seconds of delay for everyone because all demons are busy servicing the same guy before we can conclude we're being flooded is not really great.. modperl driven forums (or PHP ones even) can be killed this way since there are so many links on one page, all active.. thanks for any thoughts on this. -Justin
Re: IP based instant throttle?
Do you get flooded that frequently that this is an issue? I've seen DOS, and various buffer overflows etc. in the real world, but I've never seen this. Unless its happening very often, I would have thought that some monitoring and a 2am Deny from ip in your httpd.conf would be enough ... On Friday, June 8, 2001, at 01:50 am, Justin wrote: Does anyone see the value in a Throttle module that looked at the apache parent status block and rejected any request where another child was already busy servicing *that same IP* ? (note: the real IP is in the header in a backend setup so it is not possible to dig it out across children without creating another bit of shared memory or using the filesystem?). I'm still finding existing throttle modules do not pickup and block parallel or fast request streams fast enough .. ok there are no massive outages but 10 seconds of delay for everyone because all demons are busy servicing the same guy before we can conclude we're being flooded is not really great.. modperl driven forums (or PHP ones even) can be killed this way since there are so many links on one page, all active.. thanks for any thoughts on this. -Justin
Re: IP based instant throttle?
Hi Justin, What about the case where you would have several people accessing the site from behind a firewall at the same time? They would appear to be coming from the same IP Address. On Friday, June 8, 2001, at 01:50 am, Justin wrote: Does anyone see the value in a Throttle module that looked at the apache parent status block and rejected any request where another child was already busy servicing *that same IP* ?
