Re: [2.2.5] High cpu usage after switch to threads
sob., 21 lis 2020 o 07:13 Willy Tarreau napisał(a): > So I guess we'll use you as a beta tester once we're starting to see > promising solutions ;-) > I'll test it happily :)
Re: [2.2.5] High cpu usage after switch to threads
Hi Maciej, On Wed, Nov 18, 2020 at 12:33:23PM +0100, Maciej Zdeb wrote: > Hi again, > > So "# some headers manipulation, nothing different then on other clusters" > was the important factor in config. Under this comment I've hidden from you > one of our LUA scripts that is doing header manipulation like deleting all > headers from request when its name begins with "abc*". We're doing it on > all HAProxy servers, but only here it has such a big impact on the CPU, > because of huge RPS. > > If I understand correctly: > with nbproc = 20, lua interpreter worked on every process > with nbproc=1, nbthread=20, lua interpreter works on single process/thread > > I suspect that running lua on multiple threads is not a trivial task... You've indeed nailed it down to the exact *last* showstopper against killing nbproc (which is why I didn't kill it in 2.3). Lua is inherenty single-threaded. We've started to work with Thierry on porting it to threads. Based on limited support of threading in Lua (requires a recompilation of the library to pass lock callbacks), and the very low number of people who require high Lua performance, we currently suspect that we'll adopt the following approach: - the default will not change - lock callback registration will be supported when a multi-threaded Lua library will be detected so that, by default, you just have to rebuild liblua (possibly after applying a patch) to get full threading support. I initially thought it would be a problem, but I was wrong, we used to explain everyone how to build lua 5.3 when 5.2 was the only one shipped in distros, and it never was a problem, so here it should be the same. This would allow to instantly boost the Lua performance without touching the Lua code. - add a specific global option to use thread-local, lockless, Lua stacks that are all independent on each other (just like you currently have with processes). However there will likely be some constraints at init time. This would be the fastest and cleanest, essentially for those who have control over their Lua code (the vast majority of users). And it would not require to rebuild the library. I *absolutely* want to see threaded Lua support in 2.4, otherwise we'll be stuck with this nasty nbproc that is causing more and more trouble to the internal architecture. So I guess we'll use you as a beta tester once we're starting to see promising solutions ;-) Cheers, Willy
Re: [2.2.5] High cpu usage after switch to threads
Tim. Cool big thank to clarify that for me. Regards Aleks On 19.11.20 17:03, Tim Düsterhus wrote: Aleks, Am 19.11.20 um 16:53 schrieb Aleksandar Lazic: When a H2 client send the header in lowercase then and h1 in mixed-case could the "del-header" line not match when it's only written in lowercase or mixed-case . HTTP headers are defined to be case-insensitive. You quoted it yourself: Just as in HTTP/1.x, header field names are strings of ASCII characters that are compared in a case-insensitive fashion. "as in HTTP/1.x" Being able to request "case insensitive matching" with -i is redundant and confusing for the administrator. Personally I define all my headers within ACLs in a lowercase fashion and that works for whatever casing the client wants to use today. I think that was also one of the reason why the h1-case-adjust* feature exists ;-) The "feature" exists to support broken software that pretends to speak HTTP when it in fact does not. Best regards Tim Düsterhus
Re: [2.2.5] High cpu usage after switch to threads
Aleks, Am 19.11.20 um 16:53 schrieb Aleksandar Lazic: > When a H2 client send the header in lowercase then and h1 in mixed-case > could the "del-header" > line not match when it's only written in lowercase or mixed-case . HTTP headers are defined to be case-insensitive. You quoted it yourself: >Just as in HTTP/1.x, header field names are strings of ASCII >characters that are compared in a case-insensitive fashion. "as in HTTP/1.x" Being able to request "case insensitive matching" with -i is redundant and confusing for the administrator. Personally I define all my headers within ACLs in a lowercase fashion and that works for whatever casing the client wants to use today. > I think that was also one of the reason why the h1-case-adjust* feature > exists ;-) The "feature" exists to support broken software that pretends to speak HTTP when it in fact does not. Best regards Tim Düsterhus
Re: [2.2.5] High cpu usage after switch to threads
Hi. On 19.11.20 16:16, Maciej Zdeb wrote: Hi, Alaksandar I've looked into code and... :) Great ;-) śr., 18 lis 2020 o 15:30 Aleksandar Lazic mailto:al-hapr...@none.at>> napisał(a): Can you think to respectthe '-i'. http://git.haproxy.org/?p=haproxy.git&a=search&h=HEAD&st=grep&s=PAT_MF_IGNORE_CASE I'm not sure if I understand you correctly, but in case of http-request del-header the "case insensitivity" must be always enabled, because header names should be case insensitive according to RFC. So we should not implement "-i" flag in this scenario. Well in H2 are the headers in lowercase as far as I understand this part of H2 RFC in a proper way. But I'm not the expert in H2 so please correct me if I'm wrong. Hypertext Transfer Protocol Version 2 (HTTP/2) https://tools.ietf.org/html/rfc7540#section-8.1.2 ``` 8.1.2. HTTP Header Fields ... Just as in HTTP/1.x, header field names are strings of ASCII characters that are compared in a case-insensitive fashion. However, header field names MUST be converted to lowercase prior to their encoding in HTTP/2. ... ``` When a H2 client send the header in lowercase then and h1 in mixed-case could the "del-header" line not match when it's only written in lowercase or mixed-case . I think that was also one of the reason why the h1-case-adjust* feature exists ;-) Additional Info. What I have see in the the checking of '-i' (PAT_MF_IGNORE_CASE), the '-m reg' functions have not the PAT_MF_IGNORE_CASE check. I think you're looking at the regex execution, but flags are considered during regex compile. If you look at regex_comp function http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/regex.c;h=45a7e9004e8e4f6ad9604ed9a858aba0060b6204;hb=0217b7b24bb33d746d2bf625f5e894007517d1b0#l312 you'll notice cs param. Function is called in pattern.c and couple other places. In my opinion -i with -m reg is perfectly valid and should work. Cool great, I thought I missed something. Thanks for the clarification. Regards Aleks
Re: [2.2.5] High cpu usage after switch to threads
Hi, Alaksandar I've looked into code and... :) śr., 18 lis 2020 o 15:30 Aleksandar Lazic napisał(a): > Can you think to respectthe '-i'. > > http://git.haproxy.org/?p=haproxy.git&a=search&h=HEAD&st=grep&s=PAT_MF_IGNORE_CASE > I'm not sure if I understand you correctly, but in case of http-request del-header the "case insensitivity" must be always enabled, because header names should be case insensitive according to RFC. So we should not implement "-i" flag in this scenario. > Additional Info. > > What I have see in the the checking of '-i' (PAT_MF_IGNORE_CASE), the '-m > reg' functions > have not the PAT_MF_IGNORE_CASE check. > I think you're looking at the regex execution, but flags are considered during regex compile. If you look at regex_comp function http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/regex.c;h=45a7e9004e8e4f6ad9604ed9a858aba0060b6204;hb=0217b7b24bb33d746d2bf625f5e894007517d1b0#l312 you'll notice cs param. Function is called in pattern.c and couple other places. In my opinion -i with -m reg is perfectly valid and should work.
Re: [2.2.5] High cpu usage after switch to threads
Maciej, Am 18.11.20 um 18:48 schrieb Maciej Zdeb: > Tim thanks for the hint! You're welcome. > Aleksandar I’ll do my best, however I’m still learning HAProxy internals > and refreshing my C skills after very long break. ;) First, I’ll try to > deliver something simple like „-m beg” and after review from the team we’ll > see. As a community contributor that regularly sends patches for stuff and that also sometimes reviews patches from other contributors: I must say that you made a very good start with your first patch. Christopher made some adjustments, but it's clear that you read the CONTRIBUTING guide, because all the formal requirements were met. From my experience many already fail at that stage. Just add reg-tests for your next development and I'd say it's perfect :-) I'm looking forward to see your patches, even if I don't need them personally. > If someone is in hurry with this issue and wants to implement it asap, then > just let me know. > Best regards Tim Düsterhus
Re: [2.2.5] High cpu usage after switch to threads
Tim thanks for the hint! Aleksandar I’ll do my best, however I’m still learning HAProxy internals and refreshing my C skills after very long break. ;) First, I’ll try to deliver something simple like „-m beg” and after review from the team we’ll see. If someone is in hurry with this issue and wants to implement it asap, then just let me know. W dniu śr., 18.11.2020 o 16:49 Tim Düsterhus napisał(a): > Maciej, > > Am 18.11.20 um 14:22 schrieb Maciej Zdeb: > > I've found an earlier discussion about replacing reqidel (and others) in > > 2.x: https://www.mail-archive.com/haproxy@formilux.org/msg36321.html > > > > So basically we're lacking: > > http-request del-header x-private- -m beg > > http-request del-header x-.*company -m reg > > http-request del-header -tracea -m end > > > > I'll try to implement it in the free time. > > Please refer to this issue: https://github.com/haproxy/haproxy/issues/909 > > Best regards > Tim Düsterhus > > >
Re: [2.2.5] High cpu usage after switch to threads
Maciej, Am 18.11.20 um 14:22 schrieb Maciej Zdeb: > I've found an earlier discussion about replacing reqidel (and others) in > 2.x: https://www.mail-archive.com/haproxy@formilux.org/msg36321.html > > So basically we're lacking: > http-request del-header x-private- -m beg > http-request del-header x-.*company -m reg > http-request del-header -tracea -m end > > I'll try to implement it in the free time. Please refer to this issue: https://github.com/haproxy/haproxy/issues/909 Best regards Tim Düsterhus
Re: [2.2.5] High cpu usage after switch to threads
Hi Maciej.
On 18.11.20 14:22, Maciej Zdeb wrote:
I've found an earlier discussion about replacing reqidel (and others) in 2.x: https://www.mail-archive.com/haproxy@formilux.org/msg36321.html
So basically we're lacking:
http-request del-header x-private- -m beg
http-request del-header x-.*company -m reg
http-request del-header -tracea -m end
I'll try to implement it in the free time.
If I'm allowed to raise a wish, even I know and respect your time and your
passion.
Can you think to respectthe '-i'.
http://git.haproxy.org/?p=haproxy.git&a=search&h=HEAD&st=grep&s=PAT_MF_IGNORE_CASE
Additional Info.
What I have see in the the checking of '-i' (PAT_MF_IGNORE_CASE), the '-m reg'
functions
have not the PAT_MF_IGNORE_CASE check.
Maybe I'm wrong but is the '-i' respected by '-m reg' pattern, because I don't
see the
'icase' variable in this functions or any other check for PAT_MF_IGNORE_CASE
flag.
http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/pattern.c;hb=0217b7b24bb33d746d2bf625f5e894007517d1b0#l569
struct pattern *pat_match_regm
http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/pattern.c;hb=0217b7b24bb33d746d2bf625f5e894007517d1b0#l596
struct pattern *pat_match_reg
This both functions uses 'regex_exec_match2()' where I also don't see the
PAT_MF_IGNORE_CASE check
http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/regex.c;hb=0217b7b24bb33d746d2bf625f5e894007517d1b0#l217
I have never used '-i' with regex so maybe it's a magic in the code which I
don't recognize.
Regards
Aleks
śr., 18 lis 2020 o 13:20 Maciej Zdeb mailto:mac...@zdeb.pl>>
napisał(a):
Sure, the biggest problem is to delete header by matching prefix:
load_blacklist = function(service)
local prefix = '/etc/haproxy/configs/maps/header_blacklist'
local blacklist = {}
blacklist.req = {}
blacklist.res = {}
blacklist.req.str = Map.new(string.format('%s_%s_req.map', prefix,
service), Map._str)
blacklist.req.beg = Map.new(string.format('%s_%s_req_beg.map', prefix,
service), Map._beg)
return blacklist
end
blacklist = {}
blacklist.testsite = load_blacklist('testsite')
is_denied = function(bl, name)
return bl ~= nil and (bl.str:lookup(name) ~= nil or
bl.beg:lookup(name) ~= nil)
end
req_header_filter = function(txn, service)
local req_headers = txn.http:req_get_headers()
for name, _ in pairs(req_headers) do
if is_denied(blacklist[service].req, name) then
txn.http:req_del_header(name)
end
end
end
core.register_action('req_header_filter', { 'http-req' },
req_header_filter, 1)
śr., 18 lis 2020 o 12:46 Julien Pivotto mailto:roidelapl...@inuits.eu>> napisał(a):
On 18 Nov 12:33, Maciej Zdeb wrote:
> Hi again,
>
> So "# some headers manipulation, nothing different then on other
clusters"
> was the important factor in config. Under this comment I've hidden
from you
> one of our LUA scripts that is doing header manipulation like
deleting all
> headers from request when its name begins with "abc*". We're doing
it on
> all HAProxy servers, but only here it has such a big impact on the
CPU,
> because of huge RPS.
>
> If I understand correctly:
> with nbproc = 20, lua interpreter worked on every process
> with nbproc=1, nbthread=20, lua interpreter works on single
process/thread
>
> I suspect that running lua on multiple threads is not a trivial
task...
If you can share your lua script maybe we can see if this is doable
more natively in haproxy
>
>
>
>
> wt., 17 lis 2020 o 15:50 Maciej Zdeb mailto:mac...@zdeb.pl>> napisał(a):
>
> > Hi,
> >
> > We're in a process of migration from HAProxy[2.2.5] working on
multiple
> > processes to multiple threads. Additional motivation came from the
> > announcement that the "nbproc" directive was marked as deprecated
and will
> > be killed in 2.5.
> >
> > Mostly the migration went smoothly but on one of our clusters the
CPU
> > usage went so high that we were forced to rollback to nbproc.
There is
> > nothing unusual in the config, but the traffic on this particular
cluster
> > is quite unusual.
> >
> > With nbproc set to 20 CPU idle drops at most to 70%, with nbthread
= 20
> > after a couple of minutes at idle 50% it drops to 0%. HAProxy
> > processes/threads are working on dedicated/isolated CPU cores.
> >
> > [image: image.png]
> >
> > I mentioned that traffic is quite unusual, because most of it are
http
> > requests with some payload in headers and very very small
response
Re: [2.2.5] High cpu usage after switch to threads
I've found an earlier discussion about replacing reqidel (and others) in
2.x: https://www.mail-archive.com/haproxy@formilux.org/msg36321.html
So basically we're lacking:
http-request del-header x-private- -m beg
http-request del-header x-.*company -m reg
http-request del-header -tracea -m end
I'll try to implement it in the free time.
śr., 18 lis 2020 o 13:20 Maciej Zdeb napisał(a):
> Sure, the biggest problem is to delete header by matching prefix:
>
> load_blacklist = function(service)
> local prefix = '/etc/haproxy/configs/maps/header_blacklist'
> local blacklist = {}
>
> blacklist.req = {}
> blacklist.res = {}
> blacklist.req.str = Map.new(string.format('%s_%s_req.map', prefix,
> service), Map._str)
> blacklist.req.beg = Map.new(string.format('%s_%s_req_beg.map', prefix,
> service), Map._beg)
>
> return blacklist
> end
>
> blacklist = {}
> blacklist.testsite = load_blacklist('testsite')
>
> is_denied = function(bl, name)
> return bl ~= nil and (bl.str:lookup(name) ~= nil or
> bl.beg:lookup(name) ~= nil)
> end
>
> req_header_filter = function(txn, service)
> local req_headers = txn.http:req_get_headers()
> for name, _ in pairs(req_headers) do
> if is_denied(blacklist[service].req, name) then
> txn.http:req_del_header(name)
> end
> end
> end
>
> core.register_action('req_header_filter', { 'http-req' },
> req_header_filter, 1)
>
> śr., 18 lis 2020 o 12:46 Julien Pivotto
> napisał(a):
>
>> On 18 Nov 12:33, Maciej Zdeb wrote:
>> > Hi again,
>> >
>> > So "# some headers manipulation, nothing different then on other
>> clusters"
>> > was the important factor in config. Under this comment I've hidden from
>> you
>> > one of our LUA scripts that is doing header manipulation like deleting
>> all
>> > headers from request when its name begins with "abc*". We're doing it on
>> > all HAProxy servers, but only here it has such a big impact on the CPU,
>> > because of huge RPS.
>> >
>> > If I understand correctly:
>> > with nbproc = 20, lua interpreter worked on every process
>> > with nbproc=1, nbthread=20, lua interpreter works on single
>> process/thread
>> >
>> > I suspect that running lua on multiple threads is not a trivial task...
>>
>> If you can share your lua script maybe we can see if this is doable
>> more natively in haproxy
>>
>> >
>> >
>> >
>> >
>> > wt., 17 lis 2020 o 15:50 Maciej Zdeb napisał(a):
>> >
>> > > Hi,
>> > >
>> > > We're in a process of migration from HAProxy[2.2.5] working on
>> multiple
>> > > processes to multiple threads. Additional motivation came from the
>> > > announcement that the "nbproc" directive was marked as deprecated and
>> will
>> > > be killed in 2.5.
>> > >
>> > > Mostly the migration went smoothly but on one of our clusters the CPU
>> > > usage went so high that we were forced to rollback to nbproc. There is
>> > > nothing unusual in the config, but the traffic on this particular
>> cluster
>> > > is quite unusual.
>> > >
>> > > With nbproc set to 20 CPU idle drops at most to 70%, with nbthread =
>> 20
>> > > after a couple of minutes at idle 50% it drops to 0%. HAProxy
>> > > processes/threads are working on dedicated/isolated CPU cores.
>> > >
>> > > [image: image.png]
>> > >
>> > > I mentioned that traffic is quite unusual, because most of it are http
>> > > requests with some payload in headers and very very small responses
>> (like
>> > > 200 OK). On multi-proc setup HAProxy handles about 20 to 30k of
>> connections
>> > > (on frontend and backend) and about 10-20k of http requests. Incoming
>> > > traffic is just about 100-200Mbit/s and outgoing 40-100Mbit/s from
>> frontend
>> > > perspective.
>> > >
>> > > Did someone experience similar behavior of HAProxy? I'll try to
>> collect
>> > > more data and generate similar traffic with sample config to show a
>> > > difference in performance between nbproc and nbthread.
>> > >
>> > > I'll greatly appreciate any hints on what I should focus. :)
>> > >
>> > > Current config is close to:
>> > > frontend front
>> > > mode http
>> > > option http-keep-alive
>> > > http-request add-header X-Forwarded-For %[src]
>> > >
>> > > # some headers manipulation, nothing different then on other
>> clusters
>> > >
>> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
>> process 1
>> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
>> process 2
>> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
>> process 3
>> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
>> process 4
>> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
>> process 5
>> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
>> process 6
>> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
>> process 7
>> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
>> process 8
>> > >
Re: [2.2.5] High cpu usage after switch to threads
Sure, the biggest problem is to delete header by matching prefix:
load_blacklist = function(service)
local prefix = '/etc/haproxy/configs/maps/header_blacklist'
local blacklist = {}
blacklist.req = {}
blacklist.res = {}
blacklist.req.str = Map.new(string.format('%s_%s_req.map', prefix,
service), Map._str)
blacklist.req.beg = Map.new(string.format('%s_%s_req_beg.map', prefix,
service), Map._beg)
return blacklist
end
blacklist = {}
blacklist.testsite = load_blacklist('testsite')
is_denied = function(bl, name)
return bl ~= nil and (bl.str:lookup(name) ~= nil or bl.beg:lookup(name)
~= nil)
end
req_header_filter = function(txn, service)
local req_headers = txn.http:req_get_headers()
for name, _ in pairs(req_headers) do
if is_denied(blacklist[service].req, name) then
txn.http:req_del_header(name)
end
end
end
core.register_action('req_header_filter', { 'http-req' },
req_header_filter, 1)
śr., 18 lis 2020 o 12:46 Julien Pivotto napisał(a):
> On 18 Nov 12:33, Maciej Zdeb wrote:
> > Hi again,
> >
> > So "# some headers manipulation, nothing different then on other
> clusters"
> > was the important factor in config. Under this comment I've hidden from
> you
> > one of our LUA scripts that is doing header manipulation like deleting
> all
> > headers from request when its name begins with "abc*". We're doing it on
> > all HAProxy servers, but only here it has such a big impact on the CPU,
> > because of huge RPS.
> >
> > If I understand correctly:
> > with nbproc = 20, lua interpreter worked on every process
> > with nbproc=1, nbthread=20, lua interpreter works on single
> process/thread
> >
> > I suspect that running lua on multiple threads is not a trivial task...
>
> If you can share your lua script maybe we can see if this is doable
> more natively in haproxy
>
> >
> >
> >
> >
> > wt., 17 lis 2020 o 15:50 Maciej Zdeb napisał(a):
> >
> > > Hi,
> > >
> > > We're in a process of migration from HAProxy[2.2.5] working on multiple
> > > processes to multiple threads. Additional motivation came from the
> > > announcement that the "nbproc" directive was marked as deprecated and
> will
> > > be killed in 2.5.
> > >
> > > Mostly the migration went smoothly but on one of our clusters the CPU
> > > usage went so high that we were forced to rollback to nbproc. There is
> > > nothing unusual in the config, but the traffic on this particular
> cluster
> > > is quite unusual.
> > >
> > > With nbproc set to 20 CPU idle drops at most to 70%, with nbthread = 20
> > > after a couple of minutes at idle 50% it drops to 0%. HAProxy
> > > processes/threads are working on dedicated/isolated CPU cores.
> > >
> > > [image: image.png]
> > >
> > > I mentioned that traffic is quite unusual, because most of it are http
> > > requests with some payload in headers and very very small responses
> (like
> > > 200 OK). On multi-proc setup HAProxy handles about 20 to 30k of
> connections
> > > (on frontend and backend) and about 10-20k of http requests. Incoming
> > > traffic is just about 100-200Mbit/s and outgoing 40-100Mbit/s from
> frontend
> > > perspective.
> > >
> > > Did someone experience similar behavior of HAProxy? I'll try to collect
> > > more data and generate similar traffic with sample config to show a
> > > difference in performance between nbproc and nbthread.
> > >
> > > I'll greatly appreciate any hints on what I should focus. :)
> > >
> > > Current config is close to:
> > > frontend front
> > > mode http
> > > option http-keep-alive
> > > http-request add-header X-Forwarded-For %[src]
> > >
> > > # some headers manipulation, nothing different then on other
> clusters
> > >
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process 1
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process 2
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process 3
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process 4
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process 5
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process 6
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process 7
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process 8
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process 9
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process
> > > 10
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process
> > > 11
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process
> > > 12
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process
> > > 13
> > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1
> process
> > > 14
> > > bind x.x.x.x:443 ssl cr
Re: [2.2.5] High cpu usage after switch to threads
On 18 Nov 12:33, Maciej Zdeb wrote: > Hi again, > > So "# some headers manipulation, nothing different then on other clusters" > was the important factor in config. Under this comment I've hidden from you > one of our LUA scripts that is doing header manipulation like deleting all > headers from request when its name begins with "abc*". We're doing it on > all HAProxy servers, but only here it has such a big impact on the CPU, > because of huge RPS. > > If I understand correctly: > with nbproc = 20, lua interpreter worked on every process > with nbproc=1, nbthread=20, lua interpreter works on single process/thread > > I suspect that running lua on multiple threads is not a trivial task... If you can share your lua script maybe we can see if this is doable more natively in haproxy > > > > > wt., 17 lis 2020 o 15:50 Maciej Zdeb napisał(a): > > > Hi, > > > > We're in a process of migration from HAProxy[2.2.5] working on multiple > > processes to multiple threads. Additional motivation came from the > > announcement that the "nbproc" directive was marked as deprecated and will > > be killed in 2.5. > > > > Mostly the migration went smoothly but on one of our clusters the CPU > > usage went so high that we were forced to rollback to nbproc. There is > > nothing unusual in the config, but the traffic on this particular cluster > > is quite unusual. > > > > With nbproc set to 20 CPU idle drops at most to 70%, with nbthread = 20 > > after a couple of minutes at idle 50% it drops to 0%. HAProxy > > processes/threads are working on dedicated/isolated CPU cores. > > > > [image: image.png] > > > > I mentioned that traffic is quite unusual, because most of it are http > > requests with some payload in headers and very very small responses (like > > 200 OK). On multi-proc setup HAProxy handles about 20 to 30k of connections > > (on frontend and backend) and about 10-20k of http requests. Incoming > > traffic is just about 100-200Mbit/s and outgoing 40-100Mbit/s from frontend > > perspective. > > > > Did someone experience similar behavior of HAProxy? I'll try to collect > > more data and generate similar traffic with sample config to show a > > difference in performance between nbproc and nbthread. > > > > I'll greatly appreciate any hints on what I should focus. :) > > > > Current config is close to: > > frontend front > > mode http > > option http-keep-alive > > http-request add-header X-Forwarded-For %[src] > > > > # some headers manipulation, nothing different then on other clusters > > > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 1 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 2 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 3 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 4 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 5 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 6 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 7 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 8 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 9 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 10 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 11 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 12 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 13 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 14 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 15 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 16 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 17 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 18 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 19 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 20 > > default_backend back > > > > backend back > > option http-keep-alive > > mode http > > http-reuse always > > option httpchk GET /health HTTP/1.0\r\nHost:\ example.com > > http-check expect string OK > > > > server slot_0_checker 10.x.x.x:31180 check weight 54 > > server slot_1_checker 10.x.x.x:31146 check weight 33 > > server slot_2_checker 10.x.x.x:31313 check weight 55 > > server slot_3_checker 10.x.x.x:31281 check weight 33 disabled > > server slot_4_checker 10.x.x.x:31717 check weight 55 > > server slot_5_checker 10.x.x.x:31031 check weight 76 > > server slot_6_checker 10.x.x.x:31124 check weight 50 > > server slot_7_checker 10.x.x.x:31353 check weight 48 > > server slot_8_checker 10.x.x.x:31839 check weight 33 > > se
