Re: CPU usage increase / bad performance after upgrading haproxy from 1.4.25 to 1.5.11
I've just re-upgraded my haproxy instances to 1.5.11 and added the option http-tunnel as you suggested. I'm waiting for things to stabilize before jumping to conclusions, but it looks a bit better at the moment. I'll return with verdict about performance in a while after monitoring further! So far it looks a lot better with the option http-tunnel option set after upgrading to 1.5.11 at ~12:00 today. I also manage to dig out a combined graph in custom date format to get a better view of tcp_connections, something that i completely overlooked yesterday, and the default option http-keep-alive seems to have caused some havoc to my amount of established connections which explain the increased cpu-usage: http://imgur.com/EToNrhS I also noted that we seem to have rather unevenly distributed clients with our rrdns as the 03 server recieves more reqeusts overall, but that's another story and have noting to do with 1.4 to 1.5 upgrade. Thanks for your help so far Lukas, I really appreciate it. I should also have listen better to PiBa-NL in #haproxy@FreeNode who suggested the same change to look at yesterday evening. :) /Björn
RE: CPU usage increase / bad performance after upgrading haproxy from 1.4.25 to 1.5.11
I've just re-upgraded my haproxy instances to 1.5.11 and added the option http-tunnel as you suggested. I'm waiting for things to stabilize before jumping to conclusions, but it looks a bit better at the moment. I'll return with verdict about performance in a while after monitoring further! So far it looks a lot better with the option http-tunnel option set after upgrading to 1.5.11 at ~12:00 today. I also manage to dig out a combined graph in custom date format to get a better view of tcp_connections, something that i completely overlooked yesterday, and the default option http-keep-alive seems to have caused some havoc to my amount of established connections which explain the increased cpu-usage: http://imgur.com/EToNrhS I also noted that we seem to have rather unevenly distributed clients with our rrdns as the 03 server recieves more reqeusts overall, but that's another story and have noting to do with 1.4 to 1.5 upgrade. Thanks for your help so far Lukas, I really appreciate it. I should also have listen better to PiBa-NL in haproxy@FreeNode who suggested the same change to look at yesterday evening. :) Thanks for confirming. Now, I think if you would remove option forceclose and use: option http-keep-alive option prefer-last-server everywhere, you could achieve similar performance. Using keep-alive instead of tunneling has a lot of benefits. option forwardfor for example works on all requests, not just the first in the session, etc. Regards, Lukas
Re: CPU usage increase / bad performance after upgrading haproxy from 1.4.25 to 1.5.11
Setting: option http-tunnel Can you provide the output of haproxy -vv please? Hi Lukas, I've just re-upgraded my haproxy instances to 1.5.11 and added the option http-tunnel as you suggested. I'm waiting for things to stabilize before jumping to conclusions, but it looks a bit better at the moment. Here's the output of haproxy -vv on 1.4.25: http://paste.ubuntu.com/10550216/ and here's the output of the same for 1.5.11: http://paste.ubuntu.com/10550222/ I note there's different polling, sepoll vs epoll, but I don't know what that means (yet) :) I'll return with verdict about performance in a while after monitoring further! /Björn
RE: CPU usage increase / bad performance after upgrading haproxy from 1.4.25 to 1.5.11
Hi, I have not been able to reproduce the behavior in my lab-environment, and I would very much appreciate pointers as how to troubleshoot this further, and where to look. I have a feeling that haproxy 1.5 have default values which I've not taken into consideration that might cause this. Please advice! :-) Setting: option http-tunnel in the defaults will restore 1.4 behavior regarding http mode. Not sure why the default keep-alive mode in 1.5 would cause twice as much CPU though ... Lukas
RE: CPU usage increase / bad performance after upgrading haproxy from 1.4.25 to 1.5.11
I have not been able to reproduce the behavior in my lab-environment, and I would very much appreciate pointers as how to troubleshoot this further, and where to look. I have a feeling that haproxy 1.5 have default values which I've not taken into consideration that might cause this. Please advice! :-) Setting: option http-tunnel in the defaults will restore 1.4 behavior regarding http mode. Not sure why the default keep-alive mode in 1.5 would cause twice as much CPU though ... Can you provide the output of haproxy -vv please? Lukas
Re: CPU usage increase / bad performance after upgrading haproxy from 1.4.25 to 1.5.11
Hi all, Le 06/03/2015 16:28, Lukas Tribus a écrit : I've just re-upgraded my haproxy instances to 1.5.11 and added the option http-tunnel as you suggested. I'm waiting for things to stabilize before jumping to conclusions, but it looks a bit better at the moment. I'll return with verdict about performance in a while after monitoring further! So far it looks a lot better with the option http-tunnel option set after upgrading to 1.5.11 at ~12:00 today. I also manage to dig out a combined graph in custom date format to get a better view of tcp_connections, something that i completely overlooked yesterday, and the default option http-keep-alive seems to have caused some havoc to my amount of established connections which explain the increased cpu-usage: http://imgur.com/EToNrhS I also noted that we seem to have rather unevenly distributed clients with our rrdns as the 03 server recieves more reqeusts overall, but that's another story and have noting to do with 1.4 to 1.5 upgrade. Thanks for your help so far Lukas, I really appreciate it. I should also have listen better to PiBa-NL in haproxy@FreeNode who suggested the same change to look at yesterday evening. :) Thanks for confirming. Now, I think if you would remove option forceclose and use: option http-keep-alive option prefer-last-server Yes, those options should reduce the CPU usage. This also requires to remove option http-tunnel of course ;-) everywhere, you could achieve similar performance. Using keep-alive instead of tunneling has a lot of benefits. option forwardfor for example works on all requests, not just the first in the session, etc. Björn, what it means is that your configuration (using tunnel mode) has some side effects both on 1.4 and 1.5 that you may not be aware of. There is the X-Forwarded-For header as Lukas said, but also the backend choice. For example : - Client opens the connection - Client requests / on port 80 = haproxy send the request to default_backend - Client requests /server with the same connection = haproxy won't do any processing and will send the request to default_backend, and will not match the rule use_backend bidding_backend if is_bid Another issue you should know about your configuration : timeout http-keep-alive and timeout http-request don't work in a backend section (this is a special cas that only work when the frontend is in tcp). So, you'll never have a keep-alive timeout of 60 seconds and a htttp-requess timeout of 90 seconds, as declared in bidding_backend. Hope this helps, -- Cyril Bonté
