Re: haproxy 1.3.16 getting really really closer
Hi Willy, What a great new feature! Being able to limit connections/second is something we've probably all needed for a long time. Does this system use the failover "down" message if the client has to wait too long before having their request served? As I was thinking about the advantages of this, it occurred to me that there could be another useful scenario. Let's say that we have a powerful cluster of servers which can pretty much handle whatever traffic (within normal peaks and so on) is thrown at it. In this case, you really wouldn't want to put a request limiter on it. What I *would* want to do is still have some protection against attacks, primarily by restricting requests/second on an IP/URL basis. We can't just restrict based on IP address as people behind NATs could quickly get wrongfully limited. But if we took the combination of IP and URL and said "each backend can only receive x requests per seconds per IP and URL combination" we could also handle this other case. Just a thought. Keep up the good work! Best, JLS John L. Singleton jsing...@gmail.com On Mar 7, 2009, at 3:50 AM, Jeffrey 'jf' Lim wrote: On Sat, Mar 7, 2009 at 7:32 PM, Willy Tarreau wrote: Hi Jeff, On Sat, Mar 07, 2009 at 07:03:15PM +0800, Jeffrey 'jf' Lim wrote: Woohoo!! :) thanks, Willy, for the work. Seems like a really great list of stuff there. Especially love the "HTTP invalid request and response captures per frontend/backend" feature - I would definitely love to be able to see what we're getting over here where we use haproxy It's really useful. Many people have wasted a lot of time with traffic captures trying to catch an anomaly, while haproxy knows when there is an error, so simplifies the troubleshooting a lot to only capture errors. Also, now I see what people send in their attacks ;-) heh, yeah, I want to know too! ;p Probably that the feature will be improved so that we can decide by configuration what type of errors should cause the request/response to be captured. or where it should get logged to? it sounds like right now it's only getting logged to memory. (I mean, how about on disk? although, of course conversely, we want to prevent attacks from overwhelming our disk as well...) One question if u dont mind - session rate limiting on frontends - what's the use case for this? There are several use cases : - you can limit the request rate on a fragile server which has a dedicated frontend (eg: a local search engine) - if you're hosting several customer sites on your own infrastructure, you may want to limit each of them to a reasonable load so that none of them consumes all the CPU resources - you can limit incoming mail rate to protect your SMTP relays, especially when you have anti-virus and anti-spam forking like mad under load (I've done that here at home to protect my poor old vax). - it can help protecting your servers against DoS attacks. I'm sure other people will find smarter ideas and usages ;-) :) thanks for that. I was initially thinking that putting a maxconn on your frontend and/or backends would do it - but yeah, I see now how a having request rate might be useful as well... thanks, -jf -- In the meantime, here is your PSA: "It's so hard to write a graphics driver that open-sourcing it would not help." -- Andrew Fear, Software Product Manager, NVIDIA Corporation http://kerneltrap.org/node/7228
Re: haproxy 1.3.16 getting really really closer
On Sat, Mar 07, 2009 at 07:50:26PM +0800, Jeffrey 'jf' Lim wrote: > > Probably that the feature will be improved so that we can decide by > > configuration what type of errors should cause the request/response > > to be captured. > > > > or where it should get logged to? it sounds like right now it's only > getting logged to memory. only the last request/response per frontend/backend gets logged in memory. You can consult them by issuing "show errors" on the unix stats socket, for instance : # echo "show errors" | socat stdio /var/run/haproxy.sock > (I mean, how about on disk? although, of > course conversely, we want to prevent attacks from overwhelming our > disk as well...) no, haproxy will never write anything to disk for 2 reasons : - it's supposed to be chrooted in an empty, read-only directory for obvious security reasons. Eventhough no break-in was ever reported, the risk of a bug increases with the number of new features. - async I/O are not present on all systems and are rarely fully async, so any FS access will cause small pauses to the whole process. Regards, Willy
Re: haproxy 1.3.16 getting really really closer
On Sat, Mar 7, 2009 at 7:32 PM, Willy Tarreau wrote: > Hi Jeff, > > On Sat, Mar 07, 2009 at 07:03:15PM +0800, Jeffrey 'jf' Lim wrote: >> Woohoo!! :) thanks, Willy, for the work. Seems like a really great >> list of stuff there. >> >> Especially love the "HTTP invalid request and response captures per >> frontend/backend" feature - I would definitely love to be able to see >> what we're getting over here where we use haproxy > > It's really useful. Many people have wasted a lot of time with traffic > captures trying to catch an anomaly, while haproxy knows when there is > an error, so simplifies the troubleshooting a lot to only capture errors. > Also, now I see what people send in their attacks ;-) > heh, yeah, I want to know too! ;p > Probably that the feature will be improved so that we can decide by > configuration what type of errors should cause the request/response > to be captured. > or where it should get logged to? it sounds like right now it's only getting logged to memory. (I mean, how about on disk? although, of course conversely, we want to prevent attacks from overwhelming our disk as well...) >> One question if u dont mind - session rate limiting on frontends - >> what's the use case for this? > > There are several use cases : > - you can limit the request rate on a fragile server which has a > dedicated frontend (eg: a local search engine) > > - if you're hosting several customer sites on your own infrastructure, > you may want to limit each of them to a reasonable load so that none > of them consumes all the CPU resources > > - you can limit incoming mail rate to protect your SMTP relays, > especially when you have anti-virus and anti-spam forking like mad > under load (I've done that here at home to protect my poor old vax). > > - it can help protecting your servers against DoS attacks. > > I'm sure other people will find smarter ideas and usages ;-) > :) thanks for that. I was initially thinking that putting a maxconn on your frontend and/or backends would do it - but yeah, I see now how a having request rate might be useful as well... thanks, -jf -- In the meantime, here is your PSA: "It's so hard to write a graphics driver that open-sourcing it would not help." -- Andrew Fear, Software Product Manager, NVIDIA Corporation http://kerneltrap.org/node/7228
Re: haproxy 1.3.16 getting really really closer
Hi Jeff, On Sat, Mar 07, 2009 at 07:03:15PM +0800, Jeffrey 'jf' Lim wrote: > Woohoo!! :) thanks, Willy, for the work. Seems like a really great > list of stuff there. > > Especially love the "HTTP invalid request and response captures per > frontend/backend" feature - I would definitely love to be able to see > what we're getting over here where we use haproxy It's really useful. Many people have wasted a lot of time with traffic captures trying to catch an anomaly, while haproxy knows when there is an error, so simplifies the troubleshooting a lot to only capture errors. Also, now I see what people send in their attacks ;-) Probably that the feature will be improved so that we can decide by configuration what type of errors should cause the request/response to be captured. > One question if u dont mind - session rate limiting on frontends - > what's the use case for this? There are several use cases : - you can limit the request rate on a fragile server which has a dedicated frontend (eg: a local search engine) - if you're hosting several customer sites on your own infrastructure, you may want to limit each of them to a reasonable load so that none of them consumes all the CPU resources - you can limit incoming mail rate to protect your SMTP relays, especially when you have anti-virus and anti-spam forking like mad under load (I've done that here at home to protect my poor old vax). - it can help protecting your servers against DoS attacks. I'm sure other people will find smarter ideas and usages ;-) Regards, Willy
Re: "option httpchk" is reporting servers as down when they're not
On Sat, Mar 7, 2009 at 2:38 AM, Willy Tarreau wrote: > Hi Thomas, > > On Thu, Mar 05, 2009 at 08:45:20AM -0500, Allen, Thomas wrote: >> Hi Jeff, >> >> The thing is that if I don't include the health check, the load balancer >> works fine and each server receives equal distribution. I have no idea why >> the servers would be reported as "down" but still work when unchecked. > > It is possible that your servers expect the "Host:" header to > be set during the checks. There's a trick to do it right now > (don't forget to escape spaces) : > > option httpchk GET /index.php HTTP/1.0\r\nHost:\ www.mydomain.com > you know Thomas, Willy may be very right here. And I just realized as well - u say u're using 'option httpchk /index.php'? - without specifying the 'GET' verb? -jf > Also, you should check the server's logs to see why it is reporting > the service as down. And as a last resort, a tcpdump of the traffic > between haproxy and a failed server will show you both the request > and the complete error from the server. > > Regards, > Willy > >
Re: haproxy 1.3.16 getting really really closer
- session rate limiting on frontends : using "rate-limit sessions XXX" > it is now possible to limit the rate at which a frontend will accept > connections. This is very accurate too. Right now it's only limited > to sessions per second, but it will evolve to different periods. I've > successfully tried values as low as 1 and as high as 55000 sessions/s, > all of them gave me the expected performance within +/- 0.1% due to > the measuring resolution. This means you can simply set a limit per > application. Massive collocations will surely love this :-) > > > Some other features are pending : the recent http-check content inspection, > refined request/response ACLs, etc... those will more likely be merged after > 1.3.16. > > After that, I plan to work on smarter memory allocation (for buffers), and > a general QoS framework supporting shared and maybe hierarchical classes > of traffic, as well as per-IP and per-group classes. The idea is simply to > be able to limit the per-client data rate, request rate, and do that globally > or per frontend, etc... > > I'm not even discussing keep-alive, SSL or SMTP because I'm fairly certain > that those are the immediate big questions that will be asked after the > release. > > The current version has been running in production with tcp-splicing on a > very big site for more than one month, and on my small site for about as > long. I know that some of the people on the list also use it regularly. > For that reason, I think I'll finish a little bit of polishing and release > it as 1.3.16 very soon. The last GIT version will appear as a new snapshot > in a few hours in the usual "snapshot" directory as 20090307. > > As usual, you will have to use it with care, as there certainly are a lot > of remaining bugs. But there are nice features too :-) I think that bugs > will then be easier to fix once identified. Time will tell if they are > also easier to spot :-) > > Regards, > Willy > > >
