Re: dynamic referer protecion
Hi, On Mon, Mar 15, 2010 at 07:54:19PM +0100, Miko?aj Radzewicz wrote: > Dear Sir, > I have been using haproxy for a couple of weeks in some basic > configuration. Since 2 weeks we have been suffering from some DoS > attacks to our web servers which made them causes 500 due to extreamly > high number of connections. All of them are caused through the > referers - links(urls) to our web servers are put on very load pages > causing run out of pool of connection on our web servers. Is it some > way to protect our infrastructur using haproxy? Are you planning to > add sth like that in the future? The first thing you must do is to set a "maxconn" value on each "server" line in haproxy's config to limit the number of concurrent connections per server to a level the server can sustain. This will ensure that your servers are not saturated anymore. The second thing is to correctly configure your various timeouts so that you don't needlessly keep a lot of connections on haproxy or your servers when you suppose that the client might already have gone. For instance, a client will not wait more than 30-40 seconds for something that does not seem to come, so let's have your server timeout and queue timeout at these levels. You must also set "option abortonclose" so that each connection aborted while the request is still in the queue will not be sent to a server. Then if you know what to identify in the requests, you can eliminate them or "tarpit" them, which consists in keeping the connection open for some predefined time to slow down the attacker. But since what you describe looks more like normal browsers, maybe a drop will be enough. If you can identify a set of referrers, you can block on that. You can even redirect the request to the site holding the referrer. Probably that they'll check their page and fix it quickly after detecting the higher load ! For instance : acl from_site1 hdr_beg(referer) http://site1.dom/ redirect location http://site1.dom/ if from_site1 ... Regards, Willy
dynamic referer protecion
Dear Sir, I have been using haproxy for a couple of weeks in some basic configuration. Since 2 weeks we have been suffering from some DoS attacks to our web servers which made them causes 500 due to extreamly high number of connections. All of them are caused through the referers - links(urls) to our web servers are put on very load pages causing run out of pool of connection on our web servers. Is it some way to protect our infrastructur using haproxy? Are you planning to add sth like that in the future? thx in advance! -- Regards, MR
Re: Throughput degradation after upgrading haproxy from 1.3.22 to 1.4.1
Hi Erik, On Mon, Mar 15, 2010 at 10:27:38AM +0100, Erik Gulliksson wrote: > Hi Willy, > > Thanks for your elaborative answer. > > > Did you observe anything special about the CPU usage ? Was it lower > > than with 1.3 ? If so, it would indicate some additional delay somewhere. > > If it was higher, it could indicate that the Transfer-encoding parser > > takes too many cycles but my preliminary tests proved it to be quite > > efficient. > > I did not notice anything special about CPU usage. It seems to be > around 2-4% with both versions. When checking munin-graphs, this > morning I did however notice that the counter "connection resets > received" from "netstat -s" was increasing a lot more with 1.4. > > This led me to look at the log more closely, and there seems to be a > lot new errors that looks something like this: > w.x.y.z:4004 [15/Mar/2010:09:50:51.190] fe_xxx be_yyy/upload-srvX > 0/0/0/-1/62 502 391 - PR-- 9/6/6/3/0 0/0 "PUT /dav/filename.ext > HTTP/1.1" Interesting ! It looks like haproxy has aborted because the server returned an invalid response. You can check that using socat on the stats socket. For instance : echo "show errors" | socat stdio unix-connect:/var/run/haproxy.stat If you don't get anything, then it's something else :-/ > This is only for a few of the PUT requests, most requests seem to get > proxied successfully. I will try to reproduce this in a more > controlled lab setup where I can sniff HTTP-headers to see what is > actually sent in the request. That would obviously help too :-) > > No, I've run POST requests (very similar to PUT), except that there > > was no Transfer-Encoding in the requests. It's interesting that you're > > doing that in the request, because Apache removed support for TE:chunked > > a few years ago because there was no user. Also, most of my POST tests > > were not performance related. > > Interesting. We do use Apache for parts of this application on the > backend side, although PUT requests are handled by an in-house > developed Erlang application. OK. > > A big part has changed, in previous version, haproxy did not care > > at all about the payload. It only saw headers. Now with keepalive > > support, it has to find requests/responses bounds and as such must > > parse the transfer-encoding and content-lengths. However, transfer > > encoding is nice to components such as haproxy because it's very > > cheap. Haproxy reads a chunk size (one line), then forwards that > > many bytes, then reads a new chunk size, etc... So this is really > > a cheap operation. My tests have shown no issue at gigabit/s speeds > > with just a few bytes per chunk. > > > > I suspect that the application tries to use the chunked encoding > > to simulate a bidirectionnal access. In this case, it might be > > waiting for data pending in the kernel buffers which were sent by > > haproxy with the MSG_MORE flag, indicating that more data are > > following (and so you should observe a low CPU usage). > > > > Could you please do a small test : in src/stream_sock.c, please > > comment out line 616 : > > > > 615 /* this flag has precedence over the rest */ > > 616 // if (b->flags & BF_SEND_DONTWAIT) > > 617 send_flag &= ~MSG_MORE; > > > > It will unconditionally disable use of MSG_MORE. If this fixes the > > issue for you, I'll probably have to add an option to disable this > > packet merging for very specific applications. > > I tried to comment out the line above as instructed, but it made no > noticable change. As stated above, I will try to reproduce the problem > in a lab setup. This may be an issue with our application rather than > haproxy. OK, thanks for testing ! Best regards, Willy
Re: Throughput degradation after upgrading haproxy from 1.3.22 to 1.4.1
Hi Willy, Thanks for your elaborative answer. > Did you observe anything special about the CPU usage ? Was it lower > than with 1.3 ? If so, it would indicate some additional delay somewhere. > If it was higher, it could indicate that the Transfer-encoding parser > takes too many cycles but my preliminary tests proved it to be quite > efficient. I did not notice anything special about CPU usage. It seems to be around 2-4% with both versions. When checking munin-graphs, this morning I did however notice that the counter "connection resets received" from "netstat -s" was increasing a lot more with 1.4. This led me to look at the log more closely, and there seems to be a lot new errors that looks something like this: w.x.y.z:4004 [15/Mar/2010:09:50:51.190] fe_xxx be_yyy/upload-srvX 0/0/0/-1/62 502 391 - PR-- 9/6/6/3/0 0/0 "PUT /dav/filename.ext HTTP/1.1" This is only for a few of the PUT requests, most requests seem to get proxied successfully. I will try to reproduce this in a more controlled lab setup where I can sniff HTTP-headers to see what is actually sent in the request. > No, I've run POST requests (very similar to PUT), except that there > was no Transfer-Encoding in the requests. It's interesting that you're > doing that in the request, because Apache removed support for TE:chunked > a few years ago because there was no user. Also, most of my POST tests > were not performance related. Interesting. We do use Apache for parts of this application on the backend side, although PUT requests are handled by an in-house developed Erlang application. > A big part has changed, in previous version, haproxy did not care > at all about the payload. It only saw headers. Now with keepalive > support, it has to find requests/responses bounds and as such must > parse the transfer-encoding and content-lengths. However, transfer > encoding is nice to components such as haproxy because it's very > cheap. Haproxy reads a chunk size (one line), then forwards that > many bytes, then reads a new chunk size, etc... So this is really > a cheap operation. My tests have shown no issue at gigabit/s speeds > with just a few bytes per chunk. > > I suspect that the application tries to use the chunked encoding > to simulate a bidirectionnal access. In this case, it might be > waiting for data pending in the kernel buffers which were sent by > haproxy with the MSG_MORE flag, indicating that more data are > following (and so you should observe a low CPU usage). > > Could you please do a small test : in src/stream_sock.c, please > comment out line 616 : > > 615 /* this flag has precedence over the rest */ > 616 // if (b->flags & BF_SEND_DONTWAIT) > 617 send_flag &= ~MSG_MORE; > > It will unconditionally disable use of MSG_MORE. If this fixes the > issue for you, I'll probably have to add an option to disable this > packet merging for very specific applications. I tried to comment out the line above as instructed, but it made no noticable change. As stated above, I will try to reproduce the problem in a lab setup. This may be an issue with our application rather than haproxy. Best regards Erik -- Erik Gulliksson, erik.gulliks...@diino.net System Administrator, Diino AB http://www.diino.com
