Improve config with lots of distinct matching types
Hello list, tl;dr is: How can I avoid configuring dozens of http-request with
the same acl?
My use case is a haproxy cluster receiving requests for hundreds of distinct
hostnames, several of them with a dozen or so distinct paths, and a few more
than 5k distinct backends that sends these requests to a cluster of about 15k
containers. In the very beginning we were adding one http-request per rule but
this wasn't scaling well, so changed the approach to a map() converter with
something like `# ` which improved performance a
lot.
Now we're starting to add other dimensions to that host+path matching, which is
at least query params and header matching, so starting again with one
http-request keyword per rule, which we already know that will not scale.
My starting idea was to evolve the map() converter strategy to add these new
matching rules but couldn't figure out a way to do it. I thought about making
two levels of map() but it doesn't allow dynamic input, so I wouldn't have a
way to configure the second one.
Another idea was to mimic how other proxies do their configuration: an outer
block defines the hostname level, everything inside that block already have an
implicit `{ hdr(host) host1 }`. A second block within the first one defines the
path, so everything inside it has already an implicit `{ path /bar }`.
Maybe haproxy has already a keyword or configuration for that use case, if so
I'd love to hear some advices. If not this is a draft of a proposal that I'd
love to hear from you if it's something doable:
case (acl-name|anonymous-acl) {
http-request bar # outer acl implicit
http-request baz if { hdr(x-bar) baz } # outer acl implicitly ANDed here
case (acl-name|anonymous-acl) {
http-request bazz # both outer acls ANDing here
}
}
case keyword starts a block, a closing brace alone in the line closes the inner
block. Indentation doesn't matter at all. haproxy now knows that if the outer
acl doesn't match, all the inner keywords should be ignored. Does it make sense?
~jm
map_dir() fail when using path as input
Hi list, giving the configuration below, I was expecting "ok1" response instead
of "ok2". What am I doing wrong?
/tmp/p1
/ ok
h.cfg
defaults
timeout client 1m
timeout server 1m
timeout connect 5s
mode http
listen l1
bind :8000
http-request set-var(req.path) path
http-request set-var(req.res1) path,map_dir(/tmp/p1)
http-request set-var(req.res2) var(req.path),map_dir(/tmp/p1)
http-request return string ok1 content-type text/plain if { var(req.res1)
-m str ok }
http-request return string ok2 content-type text/plain if { var(req.res2)
-m str ok }
http-request return string fail content-type text/plain
$ haproxy -v
/ $ haproxy -v
HAProxy version 2.5.5-384c5c5 2022/03/14 - https://haproxy.org/
...
Running on: Linux 5.10.77-flatcar #1 SMP Fri Nov 5 17:49:48 -00 2021 x86_64
$ curl localhost:8000
ok2
~jm
changes in 2.5
Hello list, I have a consumer of the master socket’s `show proc` output and I observed that 2.5 changed its lay out, and this change lead me to two doubts: - Is there a release notes or something with all the backward compatibility changes between minor versions? I observed that 2.5 now requires that var() has a match type, but maybe there are other changes that I’m missing and I should take care; - Do you suggest me a good way to identify the proxy version so I can use a distinct parser for the show proc output? I’m currently using the last field of the second line, but might be a better option out there that I’m not aware of. ~jm
Re: Limit requests with peers on 2 independent HAProxies to one backend
> Em 8 de nov. de 2021, à(s) 08:26, Aleksandar Lazic
> escreveu:
>
>
> Hi.
>
> I have 2 LB's which should limit the connection to one backend.
>
> I would try to use "conn_cur" in a stick table and share it via peers.
> Have anyone such a solution already in place?
Hi Alex, I’ve already posted another question with a similar config which
worked like a charm in my tests:
https://www.mail-archive.com/haproxy@formilux.org/msg39753.html
~jm
> That's my assuption for the config.
>
> ```
> peers be_pixel_peers
> bind 9123
> log global
> localpeer {{ ansible_nodename }}
> server lb1 lb1.domain.com:1024
> server lb2 lb2.domain.com:1024
>
>
> backend be_pixel_persons
> log global
>
> acl port_pixel dst_port {{ dst_ports["pixel"] }}
> tcp-request content silent-drop if port_pixel !{ src -f
> /etc/haproxy/whitelist.acl }
>
> option httpchk GET /alive
> http-check connect ssl
> timeout check 20s
> timeout server 300s
>
> # limit connection to backend
>
> stick-table type ip size 1m expire 10m store conn_cur peers be_pixel_peers
> http-request deny if { src,table_table_conn_cur(sc_conn_cur) gt 100 }
>
>
>
> http-request capture req.fhdr(Referer) id 0
> http-request capture req.fhdr(User-Agent) id 1
> http-request capture req.hdr(host) id 2
> http-request capture var(txn.cap_alg_keysize) id 3
> http-request capture var(txn.cap_cipher) id 4
> http-request capture var(txn.cap_protocol) id 5
>
> http-response set-header X-Server %s
>
> balance roundrobin
>
> server pixel_persons1 {{ hosts["pixel_persons1"] }}:8184 resolvers mydns ssl
> check check-ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem maxconn 2
> weight 20
> server pixel_persons2 {{ hosts["pixel_persons2"] }}:8184 resolvers mydns ssl
> check check-ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem maxconn 2
> weight 20
> server pixel_persons3 {{ hosts["pixel_persons3"] }}:8184 resolvers mydns ssl
> check check-ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem maxconn 8
> weight 80
>
> ```
>
> Regards
> Alex
>
Add vary header with a new value
Hello list, I need to add a Vary header with Origin value in order to properly
configure a Cors Allow Origin with dynamic content. This works well provided
that my backend doesn’t need to configure another value to the Vary header. I
observed that if I configure set-header, this will overwrite the header
provided by the app; if I configure add-header, this will create another header
instead of concatenate the values into a single one, as suggested here[1] and
here[2].
At this moment these are the best configurations that I managed to build:
A)
http-response set-header vary %[res.hdr(vary)],origin if { res.hdr(vary) -m
found }
http-response set-header vary origin if ! { res.hdr(vary) -m found }
B)
http-response replace-header vary (.*) \1,origin if { res.hdr(vary) -m found }
http-response set-header vary origin if ! { res.hdr(vary) -m found }
Option A would remove some previous duplicated vary header, preserving just one
of them; option B would add origin on all duplicated headers. And all of them
seems to be more verbose than needed.
It seems that I’m missing some haproxy config here. Is there a better or
suggested way to add a new value to an existing header without adding a new
whole header line?
~jm
[1] https://datatracker.ietf.org/doc/html/rfc7230#section-3.2.2
[2] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Vary
Re: no-stop keyword proposal
> Em 27 de jul. de 2021, à(s) 10:03, William Lallemand
> escreveu:
>
> On Tue, Jul 20, 2021 at 12:18:18PM -0300, Joao Morais wrote:
>>
>> Regarding the use case: I need the ability to reach a stopping, but
>> still running haproxy instance to, at least:
>> [...]
>>
>
> This is already possible using the master CLI, it was done to access all
> processes, old and new. It was designed to do that kind of things.
Yes! I discovered that a few days ago!
>> diff --git a/src/cli.c b/src/cli.c
>> index b3132191d..4285e5a72 100644
>> --- a/src/cli.c
>> +++ b/src/cli.c
>> @@ -1841,6 +1841,19 @@ static int bind_parse_level(char **args, int cur_arg,
>> struct proxy *px, struct b
>> return 0;
>> }
>>
>> +/* parse the "no-stop" bind keyword */
>> +static int bind_parse_no_stop(char **args, int cur_arg, struct proxy *px,
>> struct bind_conf *conf, char **err)
>> +{
>> +struct listener *l;
>> +
>> +list_for_each_entry(l, &conf->listeners, by_bind) {
>> +l->options |= LI_O_NOSTOP;
>> +HA_ATOMIC_INC(&unstoppable_jobs);
>> +}
>> +
>> +return 0;
>> +}
>> +
>> static int bind_parse_severity_output(char **args, int cur_arg, struct proxy
>> *px, struct bind_conf *conf, char **err)
>> {
>> if (!*args[cur_arg + 1]) {
>> @@ -2984,6 +2997,7 @@ INITCALL1(STG_REGISTER, cfg_register_keywords,
>> &cfg_kws);
>> static struct bind_kw_list bind_kws = { "STAT", { }, {
>> { "level", bind_parse_level,1 }, /* set the unix socket admin
>> level */
>> { "expose-fd", bind_parse_expose_fd, 1 }, /* set the unix socket expose
>> fd rights */
>> +{ "no-stop", bind_parse_no_stop, 0 }, /* flag LI_O_NOSTOP in the
>> listener options */
>> { "severity-output", bind_parse_severity_output, 1 }, /* set the
>> severity output format */
>> { NULL, NULL, 0 },
>> }};
>>
>>
>
> That's not a good idea in my opinion, it's an internal state that should
> be used only in the case of communication between the master and the
> workers, if we expose this to users we will probably have a lot of
> corner cases to handle.
> This keyword is only meant to say to a worker that it must keep the
> communication with the master even if it's trying to exit, so we could
> do some maintenance or debugging over the master CLI.
I completely understand and thanks for the detailed explanation. Things would
be easier if all of our deployments use master/worker mode, but some of them
uses standalone servers. Anyway Willy’s suggestion helped here, I configured
the stats timeout in line with hard-stop-after and I’m connecting to it before
issue the reload, this give me a established connection to use while sessions
are alive.
~jm
Re: no-stop keyword proposal
> Em 20 de jul. de 2021, à(s) 14:17, Willy Tarreau escreveu: > > Hi Joao, > > On Tue, Jul 20, 2021 at 12:18:18PM -0300, Joao Morais wrote: >> >> Hello list, the diff below is a proposal to add a bind keyword used to flag >> LI_O_NOSTOP option in the bind's listener. >> >> Regarding the use case: I need the ability to reach a stopping, but still >> running haproxy instance to, at least: 1) fairly distribute shutdown sessions >> of long running connections (usually websockets) before hard-stop-after >> timeouts and kicks all the remaining connections at the same time[1]; 2) >> collect some relevant metrics from a stopping instance, e.g. current sessions >> and rps, which would be otherwise lost when these metrics are collected only >> from the current instance. > > It's a bit confusing for me because it mentions two opposite needs, one > applies to the listeners (and will thus either prevent the new process > from binding, or randomly distribute connections between the old and the > new one), and the other one implies killing random and active connections, > something we don't do at all. Hi Willy, I think I wasn’t clear about the needs and the proposal itself. I’ve actually two needs regarding instances being stopped (SIGUSR) but holding some active connections: 1) collect metrics in order to create real data about these connections and new data crossing them - some tcp connections and tunnels create the most distortions specially in bytes in/out and concurrent connections; 2) allow to distribute some shutdown sessions before hard-stop expires. So here is the proposal, which would allow me to connect to an old instance in order to change it’s state or collect it’s data, not limited to shutdown sessions or metrics: a way to connect to these old instances, so a stats socket alive comes to mind. You’re correct that reuse the same socket would balance requests between older and newer instances, depending on the SO_REUSESOCKET state, but as stated below the socket would be changed (unix socket, new path on every reload). This wouldn’t be an option to use on a static config of course. >> Regarding the patch: it's just the changes I needed to make and confirm that >> it works like I was expecting, provided that the listening socket is changed >> before reloading haproxy into a new instance. Please let me know if such >> improvement can be made and also if I'm in the right path. > > That's quite of a concern to me because this means that you'll accumulate > plenty of old processes, even if they do not have any connection at all > anymore. The instance wouldn’t run forever - well, I did that test and inc unstoppable_jobs seems to be enough to see the instance shutting down after the last connection being closed. > I think that your various needs would have to be addressed differently > (the killing of active connections and keeping the old process active). > For example, if you connect to the old process' CLI it will not quit, as > this socket counts for one. So maybe as long as you can connect there it > is enough to keep it alive and monitorable ? This seems promising, I’ll give this a try. It’s the opposite compared with what I’m currently doing, I’ll need to take care with the timeout, but at least I have something to start right now. Any advice here is very welcome. > For the connection shutdown, maybe we could extend "shutdown sessions" to > take a percentage, and it could apply some randomness over all connections. > This way you could periodically emit some shutdowns to kill 1% of the > connections every few seconds until you reach 100%. It's possible that > for high numbers of long connections, this significantly improves the > reload. I don't know if we could even easily automate this, but I do > see some value in it. It could sometimes kill some stats connections > as well, but with a bit of cheating that could be avoided. Maybe configure something that works with hard-stop-after? Adding another duration or a percentage of the whole hard-stop-after config where the shutdown starts - so a hard-stop-after of 30m and the configuration asking HAProxy to start the shutdown in the last 10% of the whole hard-stop-after time would start to shutdown 1000 session in the last 3 minutes, closing about 5-6 sessions per second. -Joao Morais
no-stop keyword proposal
Hello list, the diff below is a proposal to add a bind keyword used to flag
LI_O_NOSTOP option in the bind’s listener.
Regarding the use case: I need the ability to reach a stopping, but still
running haproxy instance to, at least: 1) fairly distribute shutdown sessions
of long running connections (usually websockets) before hard-stop-after
timeouts and kicks all the remaining connections at the same time[1]; 2)
collect some relevant metrics from a stopping instance, e.g. current sessions
and rps, which would be otherwise lost when these metrics are collected only
from the current instance.
Regarding the patch: it’s just the changes I needed to make and confirm that it
works like I was expecting, provided that the listening socket is changed
before reloading haproxy into a new instance. Please let me know if such
improvement can be made and also if I’m in the right path.
~jm
[1] https://www.mail-archive.com/haproxy@formilux.org/msg40916.html
diff --git a/src/cli.c b/src/cli.c
index b3132191d..4285e5a72 100644
--- a/src/cli.c
+++ b/src/cli.c
@@ -1841,6 +1841,19 @@ static int bind_parse_level(char **args, int cur_arg,
struct proxy *px, struct b
return 0;
}
+/* parse the "no-stop" bind keyword */
+static int bind_parse_no_stop(char **args, int cur_arg, struct proxy *px,
struct bind_conf *conf, char **err)
+{
+ struct listener *l;
+
+ list_for_each_entry(l, &conf->listeners, by_bind) {
+ l->options |= LI_O_NOSTOP;
+ HA_ATOMIC_INC(&unstoppable_jobs);
+ }
+
+ return 0;
+}
+
static int bind_parse_severity_output(char **args, int cur_arg, struct proxy
*px, struct bind_conf *conf, char **err)
{
if (!*args[cur_arg + 1]) {
@@ -2984,6 +2997,7 @@ INITCALL1(STG_REGISTER, cfg_register_keywords, &cfg_kws);
static struct bind_kw_list bind_kws = { "STAT", { }, {
{ "level", bind_parse_level,1 }, /* set the unix socket admin
level */
{ "expose-fd", bind_parse_expose_fd, 1 }, /* set the unix socket expose
fd rights */
+ { "no-stop", bind_parse_no_stop, 0 }, /* flag LI_O_NOSTOP in the
listener options */
{ "severity-output", bind_parse_severity_output, 1 }, /* set the
severity output format */
{ NULL, NULL, 0 },
}};
Re: missing sessions from show sess
> Em 16 de jul. de 2021, à(s) 22:16, Joao Morais
> escreveu:
>
> ...
>
> # show sess
>
>$ awk '{print $4}' sess |sort |uniq -c
> 1
> 1 fe=GLOBAL
> 1902 fe=_front__tls
> 2 fe=_front_http
> 38 fe=_front_https
>
> ...
>
> The third output shows the number of current sessions per frontend,
> _front__tls seems to be correct, however http and https are apparently
> missing lots of sessions. Shouldn’t the other frontends have a similar number
> of sessions compared with their show stat counterpart? Is there a way to find
> and list the remaining http and https connections?
Now I think I can answer myself, after a few tests on a local environment,
maybe this can be useful for someone in the future having the same doubts
(including me).
_front__tls is a mode tcp proxy, so the number of active sessions, listed by
show sess, will be the same the number of connections to that frontend.
_front_http and _front_https are mode http proxies, so the number of active
sessions, listed by show sess, will be the number of running http requests. As
soon as the http request finishes, the session disappear from the list, event
though the client is still connected to the frontend - global ConnCurrs counts
this.
The difference between global ConnCurrs (show info) or scur (show stat) and the
list above (show sess) is apparently the active connections, counting in show
info and show stat, without a running http request or a http tunnel (eg
websocket), so without counting in show sess. If my assumption is correct, "
4. scur [LFBS]: current sessions" from the management guide could be clarified
on how it works on mode http frontends.
~jm
missing sessions from show sess
Hi there, I read frontend stats, global info and current sessions from a
running haproxy, almost at the same time. Here are the outputs:
# show stat -1 1 -1 typed -- all frontends
$ sed -En '/(pxname|scur)/s/.*://p' fronts
_front__tls
1906
_front_http
40
_front_https
1862
stats
0
prometheus
3
healthz
0
# show info
$ grep CurrConns info
CurrConns: 3809
# show sess
$ awk '{print $4}' sess |sort |uniq -c
1
1 fe=GLOBAL
1902 fe=_front__tls
2 fe=_front_http
38 fe=_front_https
The first two outputs seem correct: show info says 3809 and all the frontends
sum 3811.
The third output shows the number of current sessions per frontend, _front__tls
seems to be correct, however http and https are apparently missing lots of
sessions. Shouldn’t the other frontends have a similar number of sessions
compared with their show stat counterpart? Is there a way to find and list the
remaining http and https connections?
Here is part of the configuration file:
listen _front__tls
mode tcp
bind :443,:::443
... some ssl-passthrough related keywords
server _default_server_https_socket
unix@/var/run/haproxy/_https_socket.sock send-proxy-v2
frontend _front_http
mode http
bind :80,:::80
...
frontend _front_https
mode http
bind unix@/var/run/haproxy/_https_socket.sock accept-proxy ssl ...
…
~jm
fairly distribute shutdown session a few minutes before hard-stop-after expires
Hello list, we have a HAProxy cluster in front of some chat-like applications. This HAProxy cluster is dynamically updated and now and then the instances need to be reloaded. Some of the applications behind this cluster have a few thousand of active users and, every time that the old instance’s hard-stop-after expires, closing the remaining connections, the applications receive an avalanche of reconnections from the clients (usually browsers) almost at the same time. What I’m planning to do is to monitor every old instance and start a `shutdown session` via cli, a few connections at a time, fairly distributing them in the last 25% or so of the remaining time. However maybe someone has a better idea or even pointing to a configuration that does more or less what I’m trying to implement. ~jm
Re: SNI spoofing in HAproxy?
> Em 5 de jul. de 2021, à(s) 09:30, Froehlich, Dominik
> escreveu:
>
> Here is my iteration of your solution:
>
> http-request set-var(txn.host) hdr(host),field(1,:)
> acl ssl_sni_http_host_match ssl_fc_sni,strcmp(txn.host) eq 0
> http-request deny deny_status 421 if !ssl_sni_http_host_match {
> ssl_fc_has_sni }
The last line doesn’t make sense to me if you use mTLS. Using this
configuration you’d let the client choose to not inform the SNI extension and
use whatever header he want to use. You should either block the request if SNI
and Host header differs despite the use of the SNI extension (which will lead
to block due to the match failing) or ignore the SNI at all. The former should
be used on multi site deployments, the later should be used when you have only
one single domain per ip+port. You cannot mix both, you cannot optionally use
the SNI extension and also use mTLS on multi site deployments for security
reason.
~jm
Re: Inconsistent reading of txn vars from Lua script
> Em 12 de mai. de 2021, à(s) 02:47, Willy Tarreau escreveu: > > On Tue, May 11, 2021 at 05:41:28PM -0300, Joao Morais wrote: > >> Just to confirm how it works, I created the snippet below: >> >>http-request lua.auth ## assigning txn.core >>http-request return lf-string %[var(txn.code)] content-type text/plain >> >> It worked since the first run and this is the only place I declared txn.code. >> Does this mean that a var is created in the following conditions? >> Any change in the sentences below? >> >> - after the first read from a Lua script >> - after the first write from a Lua script provided that ifexists parameter >> is set to false >> - always exists, if used anywhere in the configuration file > > It's not a matter of first or second access. It's that the function > you used initially resulted in always allocating an entry for the > variable's name, causing some huge memory usage for those who were > using them like maps and performing random lookups there. In order > to avoid this, Tim added an extra argument saying that we're just > performing an opportunistic lookup and that the variable must not > be created if it does not exist. Afaics this is only an option for set_var()? Here is the doc: TXN.set_var(TXN, var, value[, ifexist]) TXN.get_var(TXN, var) http://www.arpalert.org/src/haproxy-lua-api/2.3/index.html (link is broken from haproxy site, is the source versioned somewhere so I can send a patch?) So lookups would still create variables, hence the “in the second run it works” - my second script (which reads) create the entry and my first script (which writes with ifexists true) can find it and update in the second run and beyond. > Other parts of the code (the native config parts I mean) which use > variables always result in a creation because these names are static. > So my understanding is that it can be simplified to this: > - a variable declared in the config always exists > - a variable accessed from Lua with ifexists set to true will not >be created but will be found if it exists > - a vraiable accessed from Lua with ifexists set to false or not >present will always be created during the lookup. Taking the sentence above I’d update this to: - s/variable accessed/variable updated, set_var(),/ - variable accessed from Lua, get_var(), will always be created Does this make any sense?
Re: Inconsistent reading of txn vars from Lua script
> Em 10 de mai. de 2021, à(s) 18:04, Willy Tarreau escreveu:
>
> On Mon, May 10, 2021 at 10:41:36PM +0200, Willy Tarreau wrote:
>>> core.register_action("auth", { "http-req" }, function(txn)
>>> txn:set_var("txn.code", 401, true)
>
> So the problem is exactly here and it works as designed. This
> argument "ifexist" was added a year ago to avoid Lua allocating
> random variable names:
>
> 4e172c93f ("MEDIUM: lua: Add `ifexist` parameter to `set_var`")
>
> What the "true" argument does here is to refrain from creating
> the variable if it does not exist. After you look it up from the
> service, the variable gets created and it exists, hence why it
> then works next times.
>
> If you want it to always be created (which I assume you want
> to), just drop this argument or explicitly set it to false.
Thanks Willy for the explanation and sorry about the false alarm, I didn’t see
the whole picture here.
Just to confirm how it works, I created the snippet below:
http-request lua.auth ## assigning txn.core
http-request return lf-string %[var(txn.code)] content-type text/plain
It worked since the first run and this is the only place I declared txn.code.
Does this mean that a var is created in the following conditions? Any change in
the sentences below?
- after the first read from a Lua script
- after the first write from a Lua script provided that ifexists parameter is
set to false
- always exists, if used anywhere in the configuration file
Thanks,
Inconsistent reading of txn vars from Lua script
Hello again! Here are the snippets running with 2.4-dev18 - docker image
haproxy:2.4-dev18-alpine:
$ cat h.cfg
global
log stdout format raw local0
lua-load /tmp/h/svc1.lua
lua-load /tmp/h/svc2.lua
defaults
timeout server 1m
timeout client 1m
timeout connect 5s
log global
listen l
mode http
bind :8000
option httplog
http-request lua.auth
http-request use-service lua.send-failure
$ cat svc1.lua
core.register_action("auth", { "http-req" }, function(txn)
txn:set_var("txn.code", 401, true)
end, 0)
$ cat svc2.lua
core.register_service("send-failure", "http", function(applet)
response = applet:get_var("txn.code")
if response ~= nil then
applet:set_status(response)
else
applet:set_status(403)
end
applet:add_header("Content-Length", 0)
applet:add_header("Content-Type", "text/plain")
applet:start_response()
end)
Now curl’ing the config above:
$ curl -i localhost:8000
HTTP/1.1 403 Forbidden
content-type: text/plain
content-length: 0
$ curl -i localhost:8000
HTTP/1.1 401 Unauthorized
content-type: text/plain
content-length: 0
The first run is always a 403 which means that the reading of the txn.code
retuned nil, all the next attempts correctly returns 401. Maybe I’m missing
some kind of initialization here? Otherwise I’m happy to provide this as a
GitHub issue.
~jm
Read req scoped var from a Lua service
Hello list, following a few questions about Lua and HAProxy vars: Is there a way to read req scoped vars from a Lua script registered with core.register_service()? My attempts so far didn’t succeed, I need to copy the value to a txn scoped var before call the service. Another question, one of these vars has a comma separated list of strings that I declare with str(itemOne,itemTwo), but HAProxy complains with: error detected in proxy 'l' while parsing 'http-request set-var(txn.myvar)' rule : fetch method 'str' : expected ')' before ‘,itemTwo)’. Tested with 2.2 and 2.4. No matter I escape the comma or place inside quotes, the error is the same. Is there a way to circumvent this behavior? Last but not least haproxy.org points to a non existent Lua reference manual, maybe /2.0dev/ should be changed to a stable version. ~jm
How to read the state of a mTLS connection
Hi, I started logging mTLS connection failures in order to understand how frequently and why they fail for our users. From the collected data I observed that about 1% of the client certificates has some kind of issue that neither ssl_c_ca_err nor ssl_c_err reports, but it’s reported by ssl_c_verify; about 0.5% or so has distinct errors between ssl_c_ca_err+ssl_c_err and ssl_c_verify; and almost 99% has ssl_c_verify reporting exactly what ssl_c_ca_err and/or ssl_c_err reports. Added `ca-ignore-err all` and `crt-ignore-err all` in the bind line in order to have a chance to collect these datas. My doubt is how these samples intersect - if this happens at all? Maybe there are other samples out there that should be used and I’m missing them? ~jm
Configure peers on clusters with 20+ instances
Hello list. I'm implementing peers in order to share rps and other metrics between all instances of a haproxy cluster, so I have a global view of these data. Here is a snippet of my poc which simply does a request count: global localpeer h1 ... listen l1 ... http-request track-sc0 int(1) table p/t1 http-request set-var(req.gpc0) sc_inc_gpc0(0) http-request set-var(req.gpc0) sc_get_gpc0(0,p/t2),add(req.gpc0) http-request set-var(req.gpc0) sc_get_gpc0(0,p/t3),add(req.gpc0) http-request return hdr x-out %[var(req.gpc0)] peers p bind :9001 log stdout format raw local0 server h1 server h2 127.0.0.1:9002 server h3 127.0.0.1:9003 table t1 type integer size 1 store gpc0 table t2 type integer size 1 store gpc0 table t3 type integer size 1 store gpc0 Our biggest cluster has actually 25 haproxy instances, meaning 25 tables per instance, and 25 set-var + add() per request per tracking data. On top of that all the 25 instances will share their 25 tables to all of the other 24 instances. Build and maintain such configuration isn't a problem at all because it's automated, but how does it scale? Starting from how much instances should I change the approach and try, eg, to elect a controller that receives everything from everybody and delivers grouped data? Any advice or best practice will be very much appreciated, thanks! ~jm
Re: parsing reload cmd output from master cli
> Em 5 de fev. de 2021, à(s) 04:33, Willy Tarreau escreveu:
>
> On Wed, Feb 03, 2021 at 06:32:33PM -0300, Joao Morais wrote:
>>
>> Hello William, here[1] is some context. I implemented a parsing of the
>> reload command sent to the master cli, leave it running for a while and got
>> an index out of bounds (in my code) this week.
>>
>> I'm using this lay out to parse the reload output:
>>
>> // 1 3 4 6
>> 8 8
>> //
>> 0...|...6...|...2...|...8...|...4...|...0...|...8
>> //
>> // #
>>
>> // 1 master 0 2
>> 0d00h01m28s 2.2.3-0e58a34
>> // # workers
>> // 3 worker 1 0
>> 0d00h00m00s 2.2.3-0e58a34
>> // # old workers
>> // 2 worker [was: 1]1
>> 0d00h00m28s 2.2.3-0e58a34
>> // # programs
>> //
>>
>> Apparently I found a line that: starting char isn't '#', have 32 chars or
>> more, have less than 48 chars. Is that even possible?
>
> I had a quick look at the code producing this output and it writes
> full lines at once, but the second one is a string which is mostly a
> constant ("worker", "master"), or taken from a structure for old
> programs. So I'm seeing two possibilities:
> - either your parser received an incomplete line due to network buffering
>and decided to parse it before getting the trailing "\n"; this is a
>pretty common issue in line-oriented parsers ported to consume data
>from other processes ;
>
> - or the process memory was corrupted so that an old program's "child->id"
>contained some crap and a "\n", which would have resulted in printing
>the PID, this crappy string, and the rest of the line on a second separate
>line.
>
> The last point seems pretty possible given that your outdated version is
> affected by 123 known bugs, 2 of which could cause memory corruption.
So we probably ended with the first one - currently using 2.2.8, this output
came from my drafts when I was designing the remote restart. =)
>> How would you expand the lay out above, so I can improve my parser and my
>> tests? Thanks!
>
> Please read the chunk_appendf() statements in cli_io_handler_show_proc(),
> they're pretty explicit and you'll get the exact output format for various
> types of dumps. But that's basically what's documented above, it's just
> that it may help you to figure some constants there (or that some fields
> are unsigned for example).
Thank you so much for the insight! This will help us to prepare the code for
such corner cases.
~jm
Re: A faster way to lookup 5k hosts+paths
> Em 5 de fev. de 2021, à(s) 04:17, Willy Tarreau escreveu:
>
> Hi Joao,
>
> On Tue, Feb 02, 2021 at 09:03:06PM -0300, Joao Morais wrote:
>>
>> Hello list, I've about 5000 hostnames + path that should be mapped to 3000
>> backends or so. I'm using map converters and the lay out is something like
>> this:
>>
>> /dir/file.map
>>d1.tld/path/sub back1
>>d1.tld/path back2
>>d2.tld/path/sub/other back3
>>d2.tld/path/sub back4
>>d2.tld/path back5
>>d3.tld/path/sub back6
>>d3.tld/path back7
>>
>> And frontend looks like this:
>>
>>http-request set-var(req.backend) base,map_beg(/dir/file.map)
>>use_backend %[var(req.backend)] if { var(req.backend) -m found }
>>
>> The problem here is that the map has about 5k lines and the search isn't
>> indexed. I could eg split this map into smaller ones and create an index of
>> maps stored in another map, but cannot use the result because the file name
>> param in the converter cannot be dynamic. Any other idea to help haproxy find
>> the backend in a faster way?
>
> Not sure why you're saying it's not indexed. It uses map_beg so it will
> use a prefix tree for the lookup, it should be pretty fast.
Hi Willy, thanks for clarifying this. My question and my concern were based on
the doc which says "Other keys are stored in lists, so the first matching
occurrence will be used.”. My understanding were that it wasn’t indexed, but
instead comparing one item at a time from the top, specially when map_reg
belongs to other keys as well. So it’s good to know that map_beg (and also
map_dir?) already does a pretty good job.
> By the way, you could simplify your rules this way:
>
> use_backend base,map_beg(/dir/file.map)
>
> Indeed, a use_backend rule which doesn't find the backend will do nothing
> and continue to next rules.
Ah sure, I’m aware of this construction - except the fact that I can safely
remove the condition, tks - but I’ve actually some other configs between the
converter and the use_backend. Should’ve used a “…” between both in the
example. =)
~jm
parsing reload cmd output from master cli
Hello William, here[1] is some context. I implemented a parsing of the reload command sent to the master cli, leave it running for a while and got an index out of bounds (in my code) this week. I'm using this lay out to parse the reload output: // 1 3 4 6 8 8 // 0...|...6...|...2...|...8...|...4...|...0...|...8 // // # // 1 master 0 2 0d00h01m28s 2.2.3-0e58a34 // # workers // 3 worker 1 0 0d00h00m00s 2.2.3-0e58a34 // # old workers // 2 worker [was: 1]1 0d00h00m28s 2.2.3-0e58a34 // # programs // Apparently I found a line that: starting char isn't '#', have 32 chars or more, have less than 48 chars. Is that even possible? How would you expand the lay out above, so I can improve my parser and my tests? Thanks! ~jm [1] https://www.mail-archive.com/haproxy@formilux.org/msg38415.html
A faster way to lookup 5k hosts+paths
Hello list, I've about 5000 hostnames + path that should be mapped to 3000
backends or so. I'm using map converters and the lay out is something like this:
/dir/file.map
d1.tld/path/sub back1
d1.tld/path back2
d2.tld/path/sub/other back3
d2.tld/path/sub back4
d2.tld/path back5
d3.tld/path/sub back6
d3.tld/path back7
And frontend looks like this:
http-request set-var(req.backend) base,map_beg(/dir/file.map)
use_backend %[var(req.backend)] if { var(req.backend) -m found }
The problem here is that the map has about 5k lines and the search isn't
indexed. I could eg split this map into smaller ones and create an index of
maps stored in another map, but cannot use the result because the file name
param in the converter cannot be dynamic. Any other idea to help haproxy find
the backend in a faster way?
~jm
Change precedence using maps and distinct match types
Hello list, I'm configuring a couple of maps used to route requests based on
hostname and path. The map lay out is pretty much like this:
sub.tld/path1 name_backend1
...
I have some distinct ways to match a request: str, beg, dir, reg, using their
map converter derivatives.
map_str()
sub.tld/path/exact-match-based name_backend1
...
map_beg()
sub.tld/path/beginning-based name_backend2
...
map_dir()
sub.tld/path/subdir-based name_backend3
...
map_reg()
sub.tld/path/regex-based name_backend4
...
Here is a haproxy frontend snippet:
http-request set-var(req.backend) base,map_str(/dir/to/map_str)
http-request set-var(req.backend) base,map_beg(/dir/to/map_beg) if !{
var(req.backend) -m found }
http-request set-var(req.backend) base,map_dir(/dir/to/map_dir) if !{
var(req.backend) -m found }
http-request set-var(req.backend) base,map_reg(/dir/to/map_reg) if !{
var(req.backend) -m found }
Everything works pretty good provided that paths under the same hostname
doesn't overlap each other, or if they overlap, they belong to the same
matching pattern. However we have scenarios like this:
map_reg() -- sub.tld/path/sub[0-9]/?.* name_backend1
map_beg() -- sub.tld/path name_backend2
map_reg() -- sub.tld/p[a-z].* name_backend3
The order of the lines states the precedence - first match should win, but the
way my config was built doesn't allow to implement this. /path/sub1 will proxy
to backend2 if map_beg() comes first (backend1 should win), and /path/x will
proxy to backend3 if map_reg() comes first (backend2 should win). I'm using
maps due to performance reasons, we've currently about 20k lines of
hostname+path and about 5k backends.
To the best of my knowledge I'll need to create several maps per match type,
and reorder them accordingly - but maybe I'm missing something here. Any advice?
~jm
Re: [PATCH] DOC: clarify how to create a fallback crt
> Em 24 de nov de 2020, à(s) 05:47, William Lallemand > escreveu: > > Hello Joao, > > On Sat, Nov 21, 2020 at 12:33:38PM -0300, Joao Morais wrote: >> >> It’s indeed rather confusing, sorry about the mess. >> >> Here is a new proposal of the last paragraph, how it sounds? - suggestions >> welcome, note that I’m not very familiar with english >> >> >> >> The first declared certificate of a bind line is used as the default >> certificate, either from crt or crt-list option, which haproxy should use in >> the TLS handshake if no other certificate matches. This certificate will >> also >> be used if the provided SNI matches its CN or SAN, even if a matching SNI >> filter is found on any crt-list. The SNI filter !* can be used after the >> first >> declared certificate to not include its CN and SAN in the SNI tree, so it >> will >> never match except if no other certificate matches. This way the first >> declared certificate act as a fallback. > > It looks good in my opinion, can you make a new patch for it? Sure! Attached a new patch on top of current master. 0001-DOC-better-describes-how-to-configure-a-fallback-crt.patch Description: Binary data
Re: [PATCH] DOC: clarify how to create a fallback crt
> Em 21 de nov de 2020, à(s) 12:00, William Lallemand > escreveu: > > On Sat, Nov 21, 2020 at 07:48:48AM -0300, Joao Morais wrote: >> >> The attached patch adds some clarification on how one can declare a >> proper fallback certificate using crt-list. Feel free to ask me to >> tune verbosity to a higher or lower level. >> > > That's actually a bit confusing, because the first line of a crt-list is > not the default certificate. The default certificate is the first > certificate declared on a bind line. > > For example: > > bind :443 ssl crt default.pem crt-list list1.crtlist > bind :443 ssl crt-list list1.crtlist crt-list list2.crtlist > > In the first case, the fallback certificate will be "default.pem", and > in the second case, it will be the fist line of "list1.crtlist”. It’s indeed rather confusing, sorry about the mess. Here is a new proposal of the last paragraph, how it sounds? - suggestions welcome, note that I’m not very familiar with english The first declared certificate of a bind line is used as the default certificate, either from crt or crt-list option, which haproxy should use in the TLS handshake if no other certificate matches. This certificate will also be used if the provided SNI matches its CN or SAN, even if a matching SNI filter is found on any crt-list. The SNI filter !* can be used after the first declared certificate to not include its CN and SAN in the SNI tree, so it will never match except if no other certificate matches. This way the first declared certificate act as a fallback.
[PATCH] DOC: clarify how to create a fallback crt
The attached patch adds some clarification on how one can declare a proper fallback certificate using crt-list. Feel free to ask me to tune verbosity to a higher or lower level. 0001-DOC-clarify-how-to-create-a-fallback-crt.patch Description: Binary data
Re: Use default/first crt only if all snifilter fails
> Em 17 de nov de 2020, à(s) 05:28, William Lallemand > escreveu: > > You could also do > > /tmp/default.pem !* > > That will ignore the creation of the SNI entries. Wow thank you so much Willian, as far as I can tell and based on ~5min tests, this worked like a charm without any backward compatibility issue. I consider this a nice addition to the doc that should be merged as far as the `!*` snifilter works. I can provide a patch if you consider this a feature instead of a hack, let me know if I can help. ~jm
Use default/first crt only if all snifilter fails
Hello list, I have a `crt-list` keyword configuring a list of crt/keys, something like this: /tmp/default.pem /tmp/a.pema.local /tmp/b.pemb.local We consider the first line the fallback certificate - that one that should be used if everything else fails. We've however one situation where default.pem is valid for a.local, and since default.pem is the first crt due to its fallback status, default.pem is used in the handshake instead of a.pem. Is there a way to configure a certificate to just act as a fallback crt? Any other advice here? ~jm
Re: check successful reload using master cli
> Em 15 de set de 2020, à(s) 12:36, William Lallemand > escreveu: > > Oh right... the space in "[was: ]" is troublesome for cutting the string, > we must remove it. It's not a problem at all when using chunks of fixed size, even if columns differ between them, and the lay out ([was: ...]) remains the same. > Regarding the size of the fields, 16 bytes should be enough but there is > the expection of the version field that can be much larger if a dev > version with the commit ID in the version is used. If changing the spec, please provide also a way to distinguish the haproxy version via master cli which is not in the same `show proc` command, something like a `show info` or `version`, or at most in a fixed position at the start of `show proc`, so I can use it to version the output parser. ~jm
Re: check successful reload using master cli
> Em 14 de set de 2020, à(s) 19:14, William Lallemand > escreveu: > > Hello, > > On Mon, Sep 14, 2020 at 12:09:21PM -0300, Joao Morais wrote: >> Hello list, I'm working on an automation around haproxy process >> lifecycle in master-worker mode. It's working nice but I'm not >> confident that all premisses I used are correct. Please provide some >> guidance if I did any wrong assumption, RTFM link is welcome as usual. >> > > The documentation about the master CLI is probably light about how it's > implemented so I'll try to be as clear as possible. > Cristal clear, thank you so much. Just a few more doubts below. >> First of all I figured out that master cli will "connection refused" >> after a reload cmd during the start of the new worker. I'm using a >> unix socket but I assume the behaviour would be the same using >> ip+port. > > Indeed, during a reload all connections to the master CLI are closed > because the master process is executed again. There is currently no > mecanism to recover a previous connection from the previous process. > That's the same with the unix socket or an IP socket. > Is there any chance to be fast enough to connect and receive a response to a `show proc` command before the master started the reload process? Sleep 1ms maybe? Or when master finishes the `reload` response it’ll immediately shutdown its listener? >> I also observed that the `show proc` output isn't that >> machine-friendly, anyway it's not a big deal, some regex does the job >> and I'm assuming the lay out is immutable. Maybe I missed some >> optional json output in the doc? >> > It is supposed to be splitable easily with cut, but if that does not > work this is a clearly a bug. So may I assume fields have 16 bytes and relative pid is represented as "[was: <0-9...>]", otherwise we have a bug? ~jm
check successful reload using master cli
Hello list, I'm working on an automation around haproxy process lifecycle in master-worker mode. It's working nice but I'm not confident that all premisses I used are correct. Please provide some guidance if I did any wrong assumption, RTFM link is welcome as usual. First of all I figured out that master cli will "connection refused" after a reload cmd during the start of the new worker. I'm using a unix socket but I assume the behaviour would be the same using ip+port. I'm also using a successful connection as the meaning of a finished reload process, failed or not. Please let me know if there is a way to be notified, I'm retrying to connect within a for-loop and a growing sleep between attempts. Another assumption I made is that if I don't have any worker in the `# workers` list, it means that the last reload failed. I've observed that as soon as the master cli accepts connections again, the new worker is already there provided that the reload didn't fail. I also observed that the `show proc` output isn't that machine-friendly, anyway it's not a big deal, some regex does the job and I'm assuming the lay out is immutable. Maybe I missed some optional json output in the doc? Thanks in advance, ~jm
Redefine 401 error page
Hello list, the 401 is one of the http status code haproxy generates itself:
https://github.com/haproxy/haproxy/blob/v2.1.0/doc/configuration.txt#L363
This cannot however be overwritten using the errorfile keyword as stated in the
doc:
https://github.com/haproxy/haproxy/blob/v2.1.0/doc/configuration.txt#L3558
and also testing myself:
[WARNING] 142/002731 (1) : parsing [/tmp/haproxy/h.cfg:9] : status code 401
not
handled by 'errorfile', error customization will be ignored.
I'm aware that a Lua script can generate a custom page and an arbitrary http
status code which could work around this:
core.register_service("send-401", "http", function(applet)
send(applet, 401, [[
My custom 401 page
]])
end)
... but is there a way to, instead, customize the output of `http-request auth`?
Config:
$ cat h.cfg
defaults
timeout client 1s
timeout server 1s
timeout connect 1s
mode http
listen l
bind :8000
http-request auth if { always_true }
errorfile 401 /tmp/haproxy/401.http
server l 10.0.0.10:8000
Output:
$ curl -i localhost:8000 --user a:b
HTTP/1.1 401 Unauthorized
content-length: 112
cache-control: no-cache
content-type: text/html
www-authenticate: Basic realm=“l"
connection: close
401 Unauthorized
You need a valid user and password to access this content.
~jm
Improve a metric collector
Hello list. I’m improving a metric collector for a haproxy cluster and want to confirm if my findings and sentenses below are correct. My main goal using these metrics is to know how far from exhaustion my haproxy cluster is. 1. Source of the metric: I’m parsing `show info` from admin socket and collecting Idle_pct. I’m collecting this metric every 500ms because this is the exact time between Idle_pct updates. If I’d collect more often this would just be a spend of haproxy time for nothing. If I’d collect less often I’d loose information leading to an imprecise result. Out of curiosity: I’m converting Idle_pct back to time in order to put the amount of processing time in a time series db. This allows me to put the metric in distinct resolutions and easily convert back to idle/busy pct using `rate()`. 2. Meaning of Idle_pct in a multi-threaded deployment: If I understood the code correctly, whenever haproxy process my `show info` command, a distinct thread might be responsible for that. Every thread has its own Idle_pct calc and they are not shared, I cannot eg query the mean Idle_pct between all running threads. On one side I’m always looking to a fraction of the haproxy process on every collect, on the other side the workload should be equally distributed between all threads so the metric should be almost the same, with some negligible difference between them. 3. Meaning of the latency collecting `show info` I’m also using the latency during the collect of the metric to measure how fast haproxy is doing its job. If the mean time to collect a response for `show info` is growing, the same should be happening with client requests and server responses as well, because neither the TCP sockets used for client/server talk nor the unix socket for admin queries has a shortcut, all of them are processed in the same queue, in the same order the kernel received them. Are my sentenses correct? Is there anything else I could have a look in order to improve my metrics?
Re: ModSecurity testing
> Em 13 de dez de 2019, à(s) 10:09, Christopher Faulet > escreveu: > > Le 10/12/2019 à 05:24, Igor Cicimov a écrit : >> >> Testing with Haproxy 2.0.10 but same result with 1.8.23. The versions of >> ModSecurity is 2.9.2 and the OWASP rules v3.0.2 >> What am I doing wrong? Can anyone provide a request that should confirm if >> the module is working or not from or share the experience from their own >> setup? > > Hi Igor, > > First of all, I don't know how the modsecurity agent really work. But I'm > surprised to see it returns -101. In the code, -1, 0 or an HTTP status code > is expected. And only 0 or the HTTP status code is returned to HAProxy. I > don't know if -101 is a valid return value from modsecurity point of view. > But it is not from the agent one. > > Then, You don't have an error 403 because the variable txn.modsec.code is > negative, so the deny http-request rule is never triggered. So, I guess your > error 400 comes from your webserver. You can enabled HTTP log to have more > information. > > Finally, I notice some requests to the SPOA agent seems to have failed. The > variable is not set (- in the logs). You can try to enable SPOE logs in your > SPOE engine configuration. Take a look at the SPOE documentation > (doc/SPOE.txt) for more information. Hi, perhaps this thread helps: https://www.mail-archive.com/haproxy@formilux.org/msg30061.html And perhaps this building of ModSecurity SPOA will also help: https://github.com/jcmoraisjr/modsecurity-spoa/blob/v0.5/rootfs/Dockerfile ~jm
Re: [PATCH] BUG/MINOR: config: Warn cookie domain only if missing embedded dot
Hi Willy, > Em 30 de out de 2019, à(s) 01:41, Willy Tarreau escreveu: > > Hi Joao, > > On Tue, Oct 29, 2019 at 09:10:11PM -0300, Joao Morais wrote: >> >> What I need to implement is a way to share the sticky session cookie between >> two distinct but related domains, say haproxy.org and haproxy.com, without >> its subdomains. I couldn't do that in a way that worked and without a >> warning. > > Interesting, that's probably what forced browsers to ignore the leading > dot when in the early 2000s we started to see host-less site names. Then > I think we can indeed include your patch but then we also need to change > the comment mentioning RFC2109 and update it to 6265. Yup, attached a new patch. Note however I only know two or three words in english beyond “hello” and “good morning”, so feel free to update subject to a proper one and comment to what is really happening and what the spec is really forbiding or not. No need to be reviewed. =) Thanks! ~jm 0001-BUG-MINOR-config-Update-cookie-domain-warn-to-RFC626.patch Description: Binary data
Re: [PATCH] BUG/MINOR: config: Warn cookie domain only if missing embedded dot
Hi Willy, > Em 29 de out de 2019, à(s) 04:27, Willy Tarreau escreveu: > > No, please look at the RFC again, it's very precise on this : > https://tools.ietf.org/html/rfc2109 Thanks for taking the time to review my patch. In fact I read RFC 6265 which doesn’t take the leading dot as mandatory. 6265 obsoletes 2965, which obsoletes 2109. What I need to implement is a way to share the sticky session cookie between two distinct but related domains, say haproxy.org and haproxy.com, without its subdomains. I couldn’t do that in a way that worked and without a warning. ~jm
[PATCH] BUG/MINOR: config: Warn cookie domain only if missing embedded dot
Hi list, the attached patch fixes a warn message if the domain option, from cookie keyword, configures a domain without starting with a dot. ~jm 0001-BUG-MINOR-config-Warn-cookie-domain-only-if-missing-.patch Description: Binary data
Re: segmentation fault on 1.8.21
> Em 23 de ago de 2019, à(s) 08:16, Willy Tarreau escreveu: > > On Fri, Aug 23, 2019 at 11:47:46AM +0200, Willy Tarreau wrote: >> In the mean time you can apply the patch above. It will reject the >> first hunk but the second one applies and will address the issue. > > I've now backported all the pending patches for 1.8. You can git-pull > it if it's easier for you. Hi Willy, current 1.8 code (commit dcb8c97) fixes the issue we found on spoe, thanks! > Thank you for this report Thank you all for the great job on haproxy. ~jm
segmentation fault on 1.8.21
Hi list, I can reproduce a segmentation fault on HAProxy 1.8.21. No problem with 1.8.20, 1.9.10 or 2.0.5. Is there anything else I can provide or test on my environment? -- haproxy.cfg: ... frontend f mode http ... filter spoe engine modsecurity config /etc/haproxy/spoe-modsecurity.conf ... backend spoe-modsecurity mode tcp timeout connect 5s timeout server 5s server modsec-spoa0 192.168.100.99:12345 ... -- spoe-modsecurity.conf: [modsecurity] spoe-agent modsecurity-agent messages check-request option var-prefix modsec timeout hello 11s timeout idle28s timeout processing 39s use-backend spoe-modsecurity spoe-message check-request args unique-id method path query req.ver req.hdrs_bin req.body_size req.body event on-frontend-http-request -- Backtrace: Program received signal SIGSEGV, Segmentation fault. 0x55f747657c3c in spoe_encode_message (s=s@entry=0x55f7485bf320, ctx=ctx@entry=0x55f7485bf740, msg=msg@entry=0x55f7484ba680, dir=dir@entry=0, buf=buf@entry=0x7ffe4fb78088, end=end@entry=0x55f7485d7910 "") at src/flt_spoe.c:2190 2190src/flt_spoe.c: No such file or directory. (gdb) bt #0 0x55f747657c3c in spoe_encode_message (s=s@entry=0x55f7485bf320, ctx=ctx@entry=0x55f7485bf740, msg=msg@entry=0x55f7484ba680, dir=dir@entry=0, buf=buf@entry=0x7ffe4fb78088, end=end@entry=0x55f7485d7910 "") at src/flt_spoe.c:2190 #1 0x55f7476585f5 in spoe_encode_messages (type=, dir=0, messages=0x55f7484ba4a0, ctx=0x55f7485bf740, s=0x55f7485bf320) at src/flt_spoe.c:2233 #2 spoe_process_messages (s=0x55f7485bf320, ctx=0x55f7485bf740, messages=, dir=dir@entry=0, type=type@entry=1) at src/flt_spoe.c:2616 #3 0x55f74765a145 in spoe_process_event (ev=, ctx=, s=) at src/flt_spoe.c:2703 #4 spoe_chn_pre_analyze (s=, filter=, chn=0x55f7485bf330, an_bit=) at src/flt_spoe.c:3129 #5 0x55f7476c1595 in flt_pre_analyze (s=s@entry=0x55f7485bf320, chn=chn@entry=0x55f7485bf330, an_bit=an_bit@entry=16) at src/filters.c:805 #6 0x55f74764dfd5 in process_stream (t=t@entry=0x55f7485bf680) at src/stream.c:1906 #7 0x55f7476d5844 in process_runnable_tasks () at src/task.c:229 #8 0x55f747683214 in run_poll_loop () at src/haproxy.c:2419 #9 run_thread_poll_loop (data=) at src/haproxy.c:2489 #10 0x55f7475e261c in main (argc=, argv=) at src/haproxy.c:3092
Send http 413 response
Hello list. I'm trying to send a HTTP 413 to the user based on the hdr(Content-Length). What I've tried so far: 1. Create a http413 backend only with `errorfile 400` + `http-request deny_status 400`. In the frontend, configure a `use_backend http413 if `. This is my current approach but it is wasting some time in the frontend for every single request of every single backend - we have about 1000 backends and only about 10% need to check Content Length - btw distinct content lengths. 2. Use the `errorfile 400` approach in the same backend that does the load balance. This doesn't sound good because I'm overwriting some internal response code and its payload. Btw, what if I need another 2, 3, 10 http response codes? 3. Use some creativity eg: errorfile 413 + deny_status 413; use_backend inside another backend; what more? The later doesn't make sense but the former is a pitty that isn't supported. Is there another way to deny a http request with a custom status and html content that I’m missing? Thanks! ~jm
Re: Host header and sni extension differ
Hi Willy, > Em 17 de mai de 2019, à(s) 04:03, Willy Tarreau escreveu: > > Hi Jarno, > > On Thu, May 16, 2019 at 06:49:56PM +0300, Jarno Huuskonen wrote: >> Do the myapp.io and anotherapp.com share same certificate (ie. >> certificate has both myapp.io and anotherapp.com SAN) ? >> >> AFAIK browser can reuse the same tls connection if the certificate >> covers both names. > > Absolutely, I've already read about this though I don't know the > implementations details. Similar concepts have been discussed quite > a bit on the HTTP WG, though I don't undertand the details of each > variation. The main thing is that sometimes the browser will consider > that the connection is safe to be used for another domain name because > the first one is considered authoritative on it. I'm not sure whether > it only learns this from the cert or also from some response headers > though. This is also why I always say that routing on SNI is wrong > and that only the Host header is relevant. Everything was working without a single problem because we in fact route our requests based on Host header. The problem started when we need validate some requests based on client certs and, randomly, the DN and SHA1 headers wasn’t provided. >> When the host/sni differ do you have an earlier >> connection (for example from same ip/port) using matching sni/host in your >> logs ? > > Normally there should indeed be one. Yep, I can now confirm this one. A few minutes before the divergent Host x SNI, the user made some requests to the domain of the “wrong” SNI. ~jm
Re: Host header and sni extension differ
Hey guys, > Em 16 de mai de 2019, à(s) 15:05, Tim Düsterhus escreveu: > > Am 16.05.19 um 17:49 schrieb Jarno Huuskonen: >> Do the myapp.io and anotherapp.com share same certificate (ie. >> certificate has both myapp.io and anotherapp.com SAN) ? >> >> AFAIK browser can reuse the same tls connection if the certificate >> covers both names. When the host/sni differ do you have an earlier >> connection (for example from same ip/port) using matching sni/host in your >> logs ? I can now confirm that 100% of the distinct SNIs has certificates whose SAN also matches the Host hostname. Thank you so much for bringing some light to this problem! Time to test some patches in the config file. ~jm
Host header and sni extension differ
Hi list! The symptom is as follow: when logging Host: header I receive `myapp.io` while in the same request the sni extension says `anotherapp.com`. This happens in a very few requests (about 0.5%) but this is enough to make some noise - regarding server certificate used in the handshake, and also the ca-file used in handshakes with client certs. When they differ, the header is right and the sni is wrong. I can confirm that every "myapp.io" or "anotherapp.com" resolves to the same haproxy cluster. I can also confirm that all agents are browsers (Chrome and Firefox) running in Linux and, based on the "myapp.io" and "anotherapp.com" samples I saw together in the logs, the user is using both applications at the same time, probably from the same instance of the browser. Is there something the browser and/or HAProxy is or isn't doing here in order to mess host header and sni? Some config snippets, any change suggestion? (besides upgrade HAProxy) HAProxy 1.8.19 global daemon nbthread 3 cpu-map auto:1/1-3 0-2 stats socket /var/run/haproxy-stats.sock level admin expose-fd listeners maxconn 12000 hard-stop-after 6m log 127.0.0.1:5140 format rfc5424 local0 log-tag ingress lua-load /usr/local/etc/haproxy/lua/send-response.lua lua-load /usr/local/etc/haproxy/lua/auth-request.lua ssl-dh-param-file /ingress-controller/ssl/dhparam.pem ssl-default-bind-ciphers ... ssl-default-bind-options ssl-max-ver TLSv1.2 ssl-min-ver TLSv1.0 tune.bufsize 65536 defaults log global option redispatch option dontlognull option http-server-close option http-keep-alive timeout http-request5s timeout connect 5s timeout client 300s timeout client-fin 70s timeout queue 5s timeout server 300s timeout server-fin 70s timeout tunnel 1h timeout http-keep-alive 70s ~jm
DoS in h2 - from f5.com
Hi list, do you know if HAProxy wasn't mentioned here[1] because it isn't vulnerable (1.8 and 1.9) or because it wasn't tested? ~jm [1] https://www.f5.com/labs/articles/threat-intelligence/denial-of-service-vulnerabilities-discovered-in-http-2
Re: Tune HAProxy in front of a large k8s cluster
> Em 20 de fev de 2019, à(s) 02:51, Igor Cicimov
> escreveu:
>
>
> On Wed, 20 Feb 2019 3:39 am Joao Morais Hi Willy,
>
> > Em 19 de fev de 2019, à(s) 01:55, Willy Tarreau escreveu:
> >
> > use_backend foo if { var(req.host) ssl:www.example.com }
> >
> This is a nice trick that I’m planning to use with dynamic use_backend. I
> need to concat host (sometimes ssl_fc_sni) and path. The question is: how do
> I concatenate two strings?
>
> Something like this:
> http-request set-header X-Concat %[req.fhdr(Authorization),word(3,.)]_%[src]
>
Hi Igor, this is almost the same workaround I’m using - the difference is that
I’m moving the result to a req.var and removing the header after that.
Wondering if 1.8 has a better option
~jm
Re: Tune HAProxy in front of a large k8s cluster
> Em 20 de fev de 2019, à(s) 03:30, Baptiste escreveu: > > Hi Joao, > > I do have a question for you about your ingress controller design and the > "chained" frontends, summarized below: > * The first frontend is on tcp mode binding :443, inspecting sni and doing a > triage; >There is also a ssl-passthrough config - from the triage frontend straight > to a tcp backend. > * The second frontend is binding a unix socket with ca-file (tls > authentication); > * The last frontend is binding another unix socket, doing ssl-offload but > without ca-file. > > What feature is missing in HAProxy to allow switching these 3 frontends into > a single one? > I understand that the ability to do ssl deciphering and ssl passthrough on a > single bind line is one of them. Is there anything else we could improve? > I wonder if crt-list would be useful in your case: > https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#5.1-crt-list > Hi Baptiste, I’m changing the approach of the frontend creation - if the user configuration just need one, this one will listen :443 without need to chain another one. Regarding switch to more frontends - or at least more bind lines in the same frontend - and creating the mode-tcp one, here are the current rules: * conflict on timeout client - and perhaps on other frontend configs - distinct frontends will be created to each one * if one really want to use a certificate that doesn’t match its domain - crt-list sounds to solve this * tls auth (bind with ca-file) and no tls auth - I don’t want to mix then in the same frontend because of security - tls auth use sni, no tls auth use host header * ssl-passthrough as you have mentioned ~jm
Re: %[] in use-server directives
> Em 19 de fev de 2019, à(s) 17:51, Bruno Henc escreveu: > > On 2/19/19 9:45 PM, Joao Morais wrote: >> >>> Em 19 de fev de 2019, à(s) 05:57, Willy Tarreau escreveu: >>> >>> In the past it was not possible >>> to dynamically create servers >> I think I misunderstood something, but... how do one dynamically create a >> new server? >> > The following links should be able to help you out: > > https://www.haproxy.com/blog/dynamic-configuration-haproxy-runtime-api/#dynamically-scaling-backend-servers > > https://www.haproxy.com/blog/dynamic-scaling-for-microservices-with-runtime-api/#runtime-api > > You might need to build a development version of HAProxy to take advantage of > the latest features. > > > Let me know if you get stuck. > Hi Bruno, thanks! Updating servers via api I’m currently using. From Willy "in the past it was not possible to dynamically create servers" - so now I’m wondering if there is a way or future plan to create a new server on an existing backend and, even better, also dynamically create a new backend and its servers. ~jm
Re: %[] in use-server directives
> Em 19 de fev de 2019, à(s) 05:57, Willy Tarreau escreveu: > > In the past it was not possible > to dynamically create servers I think I misunderstood something, but... how do one dynamically create a new server?
Re: Tune HAProxy in front of a large k8s cluster
Hi Willy,
> Em 19 de fev de 2019, à(s) 01:55, Willy Tarreau escreveu:
>
> use_backend foo if { var(req.host) ssl:www.example.com }
>
This is a nice trick that I’m planning to use with dynamic use_backend. I need
to concat host (sometimes ssl_fc_sni) and path. The question is: how do I
concatenate two strings? Apparently there isn’t a concat converter and
http-request set-var() doesn’t support custom-log like expressions. There is a
usecase where I need to concatenate ssl_fc_sni and path before search in the
map.
> At this point I think that such heavy configs reach their limits and
> that the only right solution is the dynamic use_backend (possibly with
> a map).
>
Thanks for the detailed review! I’m going to the map route.
>> There are also a lot of other backends and
>> servers with health check enabled every 2s consuming some cpu and network.
>
> For this if you have many times the same server you can use the "track"
> directive, and only enable checks on a subset of servers and have all
> other track them. Typically you'd have a dummy backend dedicated to
> checks, and checks disabled in all other backends, replaced with track.
>
I’d say that currently about 98% are unique servers, but this is indeed a nice
implementation to the configuration builder.
>> Note also that I needed to add -no-pie otherwise gprof output was empty --
>> sounds a gcc issue. Let me know if this is good enough.
>
> Yes that's fine and the output was perfectly exploitable.
>
Great!
One final note - sorry about the flood yesterday. I can say with about 90% sure
I sent only one message =)
~jm
Re: Tune HAProxy in front of a large k8s cluster
> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau escreveu: > > If you have some time to run some extra tests, it would be nice to rebuild > haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not > ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls > to each function and a rough approximation of the time spent there. We may > find a huge number of calls to the culprit and possibly we could improve > it. Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 6000 requests. The request is empty (has only trivial headers) and output body has only 4 bytes. There are 12 servers on the backend which wait 200ms before sending its response. There are also a lot of other backends and servers with health check enabled every 2s consuming some cpu and network. Note also that I needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. Let me know if this is good enough. ~jm gprof.txt.gz Description: GNU Zip compressed data
Re: Tune HAProxy in front of a large k8s cluster
> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau escreveu: > > If you have some time to run some extra tests, it would be nice to rebuild > haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not > ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls > to each function and a rough approximation of the time spent there. We may > find a huge number of calls to the culprit and possibly we could improve > it. Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 6000 requests. The request is empty (has only trivial headers) and output body has only 4 bytes. There are 12 servers on the backend which wait 200ms before sending its response. There are also a lot of other backends and servers with health check enabled every 2s consuming some cpu and network. Note also that I needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. Let me know if this is good enough. ~jm gprof.txt.gz Description: GNU Zip compressed data
Re: Tune HAProxy in front of a large k8s cluster
> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau escreveu: > > If you have some time to run some extra tests, it would be nice to rebuild > haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not > ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls > to each function and a rough approximation of the time spent there. We may > find a huge number of calls to the culprit and possibly we could improve > it. Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 6000 requests. The request is empty (has only trivial headers) and output body has only 4 bytes. There are 12 servers on the backend which wait 200ms before sending its response. There are also a lot of other backends and servers with health check enabled every 2s consuming some cpu and network. Note also that I needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. Let me know if this is good enough. ~jm gprof.txt.gz Description: GNU Zip compressed data
Re: Tune HAProxy in front of a large k8s cluster
> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau escreveu: > > If you have some time to run some extra tests, it would be nice to rebuild > haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not > ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls > to each function and a rough approximation of the time spent there. We may > find a huge number of calls to the culprit and possibly we could improve > it. Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 6000 requests. The request is empty (has only trivial headers) and output body has only 4 bytes. There are 12 servers on the backend which wait 200ms before sending its response. There are also a lot of other backends and servers with health check enabled every 2s consuming some cpu and network. Note also that I needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. Let me know if this is good enough. ~jm gprof.txt.gz Description: GNU Zip compressed data
Re: Tune HAProxy in front of a large k8s cluster
> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau escreveu: > > If you have some time to run some extra tests, it would be nice to rebuild > haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not > ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls > to each function and a rough approximation of the time spent there. We may > find a huge number of calls to the culprit and possibly we could improve > it. Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 6000 requests. The request is empty (has only trivial headers) and output body has only 4 bytes. There are 12 servers on the backend which wait 200ms before sending its response. There are also a lot of other backends and servers with health check enabled every 2s consuming some cpu and network. Note also that I needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. Let me know if this is good enough. ~jm gprof.txt.gz Description: GNU Zip compressed data
Re: Tune HAProxy in front of a large k8s cluster
> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau escreveu: > > If you have some time to run some extra tests, it would be nice to rebuild > haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not > ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls > to each function and a rough approximation of the time spent there. We may > find a huge number of calls to the culprit and possibly we could improve > it. Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 6000 requests. The request is empty (has only trivial headers) and output body has only 4 bytes. There are 12 servers on the backend which wait 200ms before sending its response. There are also a lot of other backends and servers with health check enabled every 2s consuming some cpu and network. Note also that I needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. Let me know if this is good enough. ~jm gprof.txt.gz Description: GNU Zip compressed data
Re: Tune HAProxy in front of a large k8s cluster
> Em 16 de fev de 2019, à(s) 03:16, Willy Tarreau escreveu: > > If you have some time to run some extra tests, it would be nice to rebuild > haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not > ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls > to each function and a rough approximation of the time spent there. We may > find a huge number of calls to the culprit and possibly we could improve > it. Hi Willy, sure. Attached gprof output of a 300rps request during 20s, total of 6000 requests. The request is empty (has only trivial headers) and output body has only 4 bytes. There are 12 servers on the backend which wait 200ms before sending its response. There are also a lot of other backends and servers with health check enabled every 2s consuming some cpu and network. Note also that I needed to add -no-pie otherwise gprof output was empty — sounds a gcc issue. Let me know if this is good enough. ~jm gprof.txt.gz Description: GNU Zip compressed data
Re: Tune HAProxy in front of a large k8s cluster
> Em 15 de fev de 2019, à(s) 19:22, Aleksandar Lazic > escreveu: > > Am 15.02.2019 um 22:11 schrieb Joao Morais: >> >> Hey Aleks, this made my day. Thank you for remember me that map exist and a >> big thank you to The Author of map, map_beg and map_reg converters! Time to >> achieve a 5 digits rps. > > Wow you mean from 600 rps to 6 rps "just" to switch to map. > That's amazing 8-O, especially it was just a wild guess ;-) > This is just theory - and 5 digits starts on 1 =) . The problem is the " if " multiplied by 3000 or so. Moving everything to the backend or a map is a really big step forward. I suspect my frontend will have about 10 lines instead of 4000. > Would you like to share your config now? > I’d share but it has some sensitive data. But trust in me, this is as simple as one tcp frontend on :443 with about 100 use_backend -> one http frontend on an unix socket with about 3000 " if” with a few regex -> the backend. I'll let you know as soon as I've updated the config and ran the same tests again ;) ~jm
Re: Tune HAProxy in front of a large k8s cluster
> Em 15 de fev de 2019, à(s) 08:43, Aleksandar Lazic > escreveu: > > Hi Joao. > > Am 15.02.2019 um 11:15 schrieb Joao Morais: >> >> Hi Aleks, sure. Regarding the config, it has currently about 4k lines only >> in the largest frontend because of the number of hostnames and paths being >> supported. About 98% is acl declarations, http-request, reqrep, redirect >> scheme, use_backend. Most of them I'll move to the backend and this will >> already improve performance. The question is: what about the 2200+ >> `use_backend` - is there anything else that could be done? > > As I don't know the config, even a snippet could help, let me suggest you to > try > to use a map for lookup for the backends. > > https://www.haproxy.com/blog/introduction-to-haproxy-maps/ > Hey Aleks, this made my day. Thank you for remember me that map exist and a big thank you to The Author of map, map_beg and map_reg converters! Time to achieve a 5 digits rps. > Do you use DNS resolving for the hostnames? > Nops. My guess so far is the size of the frontend and my tests have confirmed this. >> / # haproxy -vv >> HA-Proxy version 1.8.17 2019/01/08 > > Event it's not critical, it would be nice when you can try 1.8.19 or better > 1.9.4 ;-) > Moving to 1.8.19 soon and waiting 1.9.5 =) ~jm
Re: HAProxy in front of Docker Enterprise problem
> Em 12 de fev de 2019, à(s) 21:21, Norman Branitsky > escreveu: > > Do I have to make HAProxy listen on 8443 and just do a tcp frontend/backend > for the Manager nodes? You can bind on another port, you can also bind on another IP address (change *:443 to some.ip.addr:443). But if you want or you need to share the same IP and port, a possible configuration is to create a tcp mode frontend which inspect sni extension and make a triage: manager hostname? Use a tcp mode backend and the manager nodes as servers - no data would be changed. This blog post[1] is of some help. In the triage, if the request isn't to a maanger node, use another tcp backend whose only server is a unix socket. Use also send-proxy-v2 in the server declaration. Create another http mode frontend, binding that unix socket and accept-proxy keyword to do the ssl offload of your worker nodes. hth. [1] https://www.haproxy.com/blog/enhanced-ssl-load-balancing-with-server-name-indication-sni-tls-extension/
Re: Tune HAProxy in front of a large k8s cluster
> Em 15 de fev de 2019, à(s) 07:44, Aleksandar Lazic
> escreveu:
>
> Hi Joao.
>
> Am 15.02.2019 um 10:21 schrieb Joao Morais:
>>
>> Hi list, I'm tuning some HAProxy instances in front of a large kubernetes
>> cluster. The config has about 500 hostnames (a la apache/nginx virtual
>> hosts), 3 frontends, 1500 backends and 4000 servers. The first frontend is on
>> tcp mode binding :443, inspecting sni and doing a triage; the second frontend
>> is binding a unix socket with ca-file (tls authentication); the last frontend
>> is binding another unix socket, doing ssl-offload but without ca-file. This
>> last one has about 80% of the hostnames. There is also a ssl-passthrough
>> config - from the triage frontend straight to a tcp backend.
>
> Please can you tell us which haproxy you use and show us the config, thanks.
Hi Aleks, sure. Regarding the config, it has currently about 4k lines only in
the largest frontend because of the number of hostnames and paths being
supported. About 98% is acl declarations, http-request, reqrep, redirect
scheme, use_backend. Most of them I'll move to the backend and this will
already improve performance. The question is: what about the 2200+
`use_backend` - is there anything else that could be done?
/ # haproxy -vv
HA-Proxy version 1.8.17 2019/01/08
Copyright 2000-2019 Willy Tarreau
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
-Wno-null-dereference -Wno-unused-label
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.2q 20 Nov 2018
Running on OpenSSL version : OpenSSL 1.0.2q 20 Nov 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
Tune HAProxy in front of a large k8s cluster
Hi list, I'm tuning some HAProxy instances in front of a large kubernetes cluster. The config has about 500 hostnames (a la apache/nginx virtual hosts), 3 frontends, 1500 backends and 4000 servers. The first frontend is on tcp mode binding :443, inspecting sni and doing a triage; the second frontend is binding a unix socket with ca-file (tls authentication); the last frontend is binding another unix socket, doing ssl-offload but without ca-file. This last one has about 80% of the hostnames. There is also a ssl-passthrough config - from the triage frontend straight to a tcp backend. I'm observing some latency on moderate loads (200+ rps per instance) - on my tests, the p95 was about 25ms only in the proxy, and the major issue is that I cannot have a throughput above 600 rps. This latency moves easily from 25ms on p95 to 1s or more on p50 with 700+ rps. The problem is of course the big amount of rules in the frontend: haproxy need to check every single bit of configuration for every single host and every single path. Moving the testing hostname to a dedicated frontend with only its own rules give me with about 5ms of p95 latency and more than 5000 rps. These are my ideas so far regarding tune such configuration: * Move all possible rules to the backend. Some txn vars should be created in order to be inspected there. This will of course help but there is still a lot of `use_backend if ` that cannot be removed, I think, which are being evaluated on every single request despite the hostname that I'm really interested. There are some hostnames without path acl, but there are also hostnames with 10+ different paths and its 10+ `use_backend`. * Create some more frontends and unix sockets with at most 50 hostnames or so. Pros: after the triage, each frontend will have the `use_backend if` of only another 49 hostnames. Cons: if some client doesn't send the sni extension, the right frontend couldn't be found. * Perhaps there is a hidden `if do done` that I'm missing which would improve performance, since I can help HAProxy to process only the keywords I'm really interested in that request. * Nbthreads was already tested, I'm using 3 that has the best performance on a 8 cores VM. 4+ threads doesn’t scale. Nbprocs will also be used, I'm tuning a per process configuration now. Is there any other approach I'm missing? Every single milisecond will help.
Re: Use acl on spoe events
> Em 27 de mai de 2018, à(s) 14:51, Joao Morais
> escreveu:
>
>> Em 27 de mai de 2018, à(s) 12:02, Daniel Corbett
>> escreveu:
>>
>> Hello Joao,
>>
>> On 05/26/2018 05:54 PM, Joao Morais wrote:
>>>
>>> There is no difference if I use acl like the example above, or use the `if
>>> {...}` syntax or remove the acl at all, my modsecurity agent always receive
>>> a new connection despite of the host I’m using.
>>>
>>> Is there a way to use a spoe filter only if some l7 conditions match?
>>
>> This appears to have been fixed in 1.9-dev with this commit:
>>
>> https://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=333694d7715952a9610a3e6f00807eaf5edd209a;hp=336d3ef0e77192582c98b3c578927a529ceadd9b
>
> Hi Daniel, I can confirm this fixes my 1.8.9 deployment, thanks!
>
> Is there any plan to backport to 1.8?
..
~jm
Right way to seamless reload a multi process cfg
Hi, taking this hypothetical cfg: === global daemon nbproc 3 stats socket unix@/tmp/haproxy1.sock expose-fd listeners process 1 stats socket unix@/tmp/haproxy2.sock expose-fd listeners process 2 stats socket unix@/tmp/haproxy3.sock expose-fd listeners process 3 listen f1 bind :80 process 1 server s1 192.168.99.1:8000 check listen f2 bind :8080 process 2-3 server s2 192.168.99.1:8000 check === What's the right or the better way to seamless reload (-x) this multi process configuration? Multi -x command-line parameter doesn't seem to work, and use one or the other socket will apparently leave some listening sockets without being reused. ~jm
Re: Use acl on spoe events
> Em 27 de mai de 2018, à(s) 12:02, Daniel Corbett
> escreveu:
>
> Hello Joao,
>
> On 05/26/2018 05:54 PM, Joao Morais wrote:
>>
>> There is no difference if I use acl like the example above, or use the `if
>> {...}` syntax or remove the acl at all, my modsecurity agent always receive
>> a new connection despite of the host I’m using.
>>
>> Is there a way to use a spoe filter only if some l7 conditions match?
>
> This appears to have been fixed in 1.9-dev with this commit:
>
> https://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=333694d7715952a9610a3e6f00807eaf5edd209a;hp=336d3ef0e77192582c98b3c578927a529ceadd9b
Hi Daniel, I can confirm this fixes my 1.8.9 deployment, thanks!
Is there any plan to backport to 1.8?
>
> Thanks,
> -- Daniel
~jm
Use acl on spoe events
Hi list, I’m trying to filter spoe events using acl, no success atm.
This is the relevant part of my configuration:
=== /etc/haproxy/haproxy.cfg ===
frontend bar
...
filter spoe engine modsecurity config /etc/haproxy/spoe-modsecurity.conf
http-request deny if { var(txn.modsec.code) -m int gt 0 }
...
backend spoe-modsecurity
mode tcp
server modsec-spoa1 127.0.0.1:12345
=== /etc/haproxy/spoe-modsecurity.conf
[modsecurity]
spoe-agent modsecurity-agent
messages check-request
option var-prefix modsec
timeout hello 100ms
timeout idle30s
timeout processing 1s
use-backend spoe-modsecurity
spoe-message check-request
aclhost_my.domain req.hdr(host) my.domain
args unique-id method path query req.ver req.hdrs_bin req.body_size
req.body
event on-frontend-http-request if host_my.domain
There is no difference if I use acl like the example above, or use the `if
{...}` syntax or remove the acl at all, my modsecurity agent always receive a
new connection despite of the host I’m using.
Is there a way to use a spoe filter only if some l7 conditions match?
~jm
Re: SPOE and modsecurity contrib
> Em 20 de mai de 2018, à(s) 11:59, Daniel Corbett
> escreveu:
>
> Hello Joao,
>
> While I haven't been able to get 'tcp-request content reject' to work with
> this configuration -- I am able to get 'http-request deny' to work:
>
> http-request deny if { var(txn.modsec.code) -m int gt 0 }
>
> Regarding txn.modsec.code -- I have been able to reproduce the
> "txn.modsec.code=-101" and "set variable code=4294967195" when crs-setup.conf
> / crs.setup.conf.example is missing the following SecDefaultAction lines:
>
> SecDefaultAction "phase:1,log,auditlog,deny,status:403"
> SecDefaultAction "phase:2,log,auditlog,deny,status:403"
>
> When those are in place -- I receive the following in logs:
>
> The txn.modsec.code is: 403
>
> Please let me know if that solves it for you.
>
Hi Daniel, yes, now everything makes a lot of sense - and it’s working.
Including the fact that `4294967195(uint32) == -101(int32)` and that I need to
read a lot more about how ModSecurity works.
Thanks!
>
> Thanks,
> —- Daniel
>
~jm
SPOE and modsecurity contrib
Hi list, I'm playing with SPOE and modsecurity contrib from HAProxy 1.8.9. I've
a couple of doubts and issues that I'll describe just below my config and some
loggings:
= haproxy.conf
listen my-front
log 127.0.0.1:514 format rfc5424 local0
timeout client 5s
timeout connect 5s
timeout server 5s
mode http
bind :8080
log-format "The txn.modsec.code is: %[var(txn.modsec.code)]"
filter spoe engine modsecurity config /tmp/haproxy/spoe-modsecurity.conf
tcp-request content reject if { var(txn.modsec.code) -m int gt 0 }
server local 192.168.95.102:8000
backend spoe-modsecurity
mode tcp
timeout connect 5s
timeout server 3m
server iprep1 127.0.0.1:12345
-
= spoe-modsecurity.conf
[modsecurity]
spoe-agent modsecurity-agent
messagescheck-request
option var-prefix modsec
timeout hello 100ms
timeout idle30s
timeout processing 1s
use-backend spoe-modsecurity
spoe-message check-request
args unique-id method path query req.ver req.hdrs_bin req.body_size req.body
event on-frontend-http-request
-
modsecurity SPOA is logging whenever I make a request to HAProxy so apparently
it's working as expected:
=log 1 -- curl -I localhost:8080?ref=abc
1526747591.637257 [01] Process SPOE Message 'check-request'
1526747591.638333 [01] Encode Agent ACK frame
1526747591.638867 [01] STREAM-ID=0 - FRAME-ID=1
1526747591.638887 [01] Add action : set variable code=4294967195
-
=log 2 -- curl -I 127.0.0.1:8080?ref=abc
1526741757.670146 [01] Process SPOE Message 'check-request'
1526741757.671956 [00] [client 127.0.0.1] ModSecurity: Warning. Pattern match
"^[\\d.:]+$" at REQUEST_HEADERS:Host. [file
"/owasp-modsecurity-crs-3.0.2/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"]
[line "810"] [id "920350"] [rev "2"] [msg "Host header is a numeric IP
address"] [data "127.0.0.1:8080"] [severity "WARNING"] [ver "OWASP_CRS/3.0.0"]
[maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"]
[tag "platform-multi"] [tag "attack-protocol"] [tag
"OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST"] [tag "WASCTC/WASC-21"] [tag
"OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "8ae61a5f0c2e"] [uri
"http://127.0.0.1:8080/";] [unique_id ""]
1526741757.672306 [01] Encode Agent ACK frame
1526741757.672333 [01] STREAM-ID=12 - FRAME-ID=1
1526741757.672351 [01] Add action : set variable code=4294967195
-
=log 3 -- curl -I 127.0.0.1:8080?ref=/../etc/passwd
1526741728.786700 [00] [client 127.0.0.1] ModSecurity: Warning. Matched phrase
"etc/passwd" at ARGS:ref. [file
"/owasp-modsecurity-crs-3.0.2/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"]
[line "448"] [id "932160"] [rev "1"] [msg "Remote Command Execution: Unix Shell
Code Found"] [data "Matched Data: etc/passwd found within ARGS:ref:
/etc/passwd"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"]
[accuracy "8"] [tag "application-multi"] [tag "language-shell"] [tag
"platform-unix"] [tag "attack-rce"] [tag
"OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag
"OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] [hostname "8ae61a5f0c2e"] [uri
"http://127.0.0.1:8080/";] [unique_id ""]
...
1526741728.787814 [01] Add action : set variable code=4294967195
-
Running modsecurity with: -d -n 1 -f /tmp/haproxy/modsec.conf
My environment is a `haproxy:alpine` container with all the stuff needed to
compile both modsecurity and it's SPOA contrib. The doc used was
`haproxy-1.8.9/contrib/modsecurity/README`.
Now a couple of doubts:
* If I `curl -v 127.0.0.1:12345` I can see a connection being opened on
modsecurity and `curl` waits for something it could understand. If I ctrl+C
`curl`, modsecurity starts an endless loop and use almost 100% of one CPU. I'm
aware modsecurity SPOA speaks SPOP instead of HTTP, but perhaps something could
be improved since this could also happen eg on a network failure.
* The `set variable code=` on logs has always the same value `4294967195`
despite what modsecurity found.
* A `log-format "%[var(txn.modsec.code)]"` logs always the same value `-101`
despite what modsecurity found.
* Perhaps a specific HAProxy doubt here: `tcp-request content reject` never
rejects a request even changing `gt` to `lt` or `eq`. Changing the acl to
`!{...}` the connection is always rejected despite of the operator.
~jm
Re: [ANNOUNCE] haproxy-1.8.3
> Em 30 de dez de 2017, à(s) 15:32, Willy Tarreau escreveu: > > - implemented the graceful shutdown on HTTP/2 connections during a reload >so that we can inform the client we're going to close, encouraging the >client to switch to a new connection. This avoids connections from >lasting forever after reloads on H2. I also noticed that it allows the >process to be replaced faster. Hi Willy, is there something like this on 1.7.x? This would be useful on eg websockets. ~jm
Re: Client cert verification on some paths
> Em 2 de dez de 2017, à(s) 08:47, Aleksandar Lazic
> escreveu:
>
> Von: "Joao Morais" gesendet: 02.12.2017 00:53:33
>
>> Hi, I have some apps that need to mimic an Apache httpd behavior on client
>> certificate verification: require certificate only on some paths.
>>
>> Apache does this implementing SSL renegotiation as briefly explained here[1].
>>
>> Of couse I can `mode tcp` proxy to an Apache instance to do that for me but
>> my topology would be simplified if I could implement SSL renegotiation on
>> HAProxy as soon as I can fetch the path sample.
>>
>> Is there a way to accomplish this without using Apache httpd?
> You can use the following line to full fill your request, untested.
>
> bind :443 ssl ca-file "${PATH_TO_CAFILE}" crl-file "${PATH_TO_CRLFILE}"
> verify "${VERIFY_MODE}"
>
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-ca-file
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-crl-file
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-verify
>
> You can add the following header to see if the client was successful verified.
>
> http-request set-header X-SSL-Client-Verify %[ssl_c_verify]
>
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-http-request
>
> When you start the haproxy with the environment variables PATH_TO_CAFILE and
> PATH_TO_CRLFILE set to your paths and VERIFY_MODE=optional can you test if
> the verification works.
>
> http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#2.3
Thanks for the detailed explanation.
This is actually very close to my current setup and I'm looking for a way to
avoid ask the certificate from the user on a browser if he doesn't request the
/path that require the certificate. HAProxy has a lot of L5/6 fetch samples,
and with some unknown (by me) keyword, perhaps I could implement a SSL
renegotiation (or something like that) just like Apache httpd already implement.
Just to name an example: HAProxy doesn't have native support for configuration
of a http response which explains to the user he need to provide a certificate
(on one page) - and signed by a known CA (on another page), but I got it
working using verify optional and fetching the right L5 samples. The actual
configuration however is far beyond my knowledge in such a way that I simply
cannot say this is even possible.
> I strongly suggest to go through the manual several times due to the fact
> that it's worth and you learn a lot about haproxy ;-)
Sure, the link to the doc is already on my favourites =)
~jm
Client cert verification on some paths
Hi, I have some apps that need to mimic an Apache httpd behavior on client certificate verification: require certificate only on some paths. Apache does this implementing SSL renegotiation as briefly explained here[1]. Of couse I can `mode tcp` proxy to an Apache instance to do that for me but my topology would be simplified if I could implement SSL renegotiation on HAProxy as soon as I can fetch the path sample. Is there a way to accomplish this without using Apache httpd? ~jm [1] http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslverifyclient
What can cause RST from HAProxy 1.7.9
Hi, HAProxy 1.7.9 is being used to route traffic to a Kubernetes cluster on AWS. It was observed periodic spikes of RST from HAProxy on active connections. Full details in the following issue from GitHub: https://github.com/jcmoraisjr/haproxy-ingress/issues/77 . In which circumstances HAProxy send RST? Any help would be very much appreciated. ~jm
[docs] about master-worker mode
Hi, from nbproc doc[1]: "This requires the 'daemon' mode”, but this is also the way to start more than one worker on master-worker mode, right? Still on the same doc: "USING MULTIPLE PROCESSES IS HARDER TO DEBUG AND IS REALLY DISCOURAGED”, is this still valid on master-worker? Both "harder to debug" and "is really discouraged" parts. From the blog post[2]: "Start the haproxy process with the command line argument -W <# of processes>" - is this <# of procs> correct? Couldn’t use this on the command line. ~jm [1] http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.1-nbproc [2] https://www.haproxy.com/blog/whats-new-haproxy-1-8/
Fetch DN according to RFC 2253
Hi list, is there a way to choose between pre and pos RFC 2253[1] format of DN from a client cert? Here are nginx[2] and Apache[3] docs about the subject. [1] https://tools.ietf.org/html/rfc2253 [2] http://nginx.org/en/docs/http/ngx_http_ssl_module.html#var_ssl_client_s_dn [3] http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#ssloptions (look for LegacyDNStringFormat option)
Re: Deny with 413 request too large
> Em 17 de mai de 2017, à(s) 19:34, Bryan Talbot
> escreveu:
>
>
>> On May 15, 2017, at May 15, 6:35 PM, Joao Morais
>> wrote:
>>
>> errorfile 413 /usr/local/etc/haproxy/errors/413.http
>> http-request deny deny_status 413 if { req.body_size gt 10485760 }
>>
>> ... HAProxy complains with:
>>
>> [WARNING] 135/001448 (27) : parsing [/etc/haproxy/haproxy.cfg:15] : status
>> code 413 not handled by 'errorfile', error customization will be ignored.
>> [WARNING] 135/001448 (27) : parsing [/etc/haproxy/haproxy.cfg:89] : status
>> code 413 not handled, using default code 403.
>>
>> How should I configure HAProxy in order to deny with 413?
>
>
> You’ve already found it. AFAIK, that’s the only way.
=(
In my understanding I should only use a 400badreq.http like message on an
errorfile 400 config line, otherwise if HAProxy need to issue a 400 status
code, my 413 status code would be issued instead.
Is this a valid feature request or there are technical reasons why this has
been done that way?
Hints are welcome.
~jm
Deny with 413 request too large
Hello list, I need to configure HAProxy to deny the request with 413 based on
the value of the content-length header. This is my actual configuration:
errorfile 400 /usr/local/etc/haproxy/errors/413.http
http-request deny deny_status 400 if { req.body_size gt 10485760 }
This is working but sounds a hacky workaround since I’m using another status
code. If I try to use:
errorfile 413 /usr/local/etc/haproxy/errors/413.http
http-request deny deny_status 413 if { req.body_size gt 10485760 }
... HAProxy complains with:
[WARNING] 135/001448 (27) : parsing [/etc/haproxy/haproxy.cfg:15] : status
code 413 not handled by 'errorfile', error customization will be ignored.
[WARNING] 135/001448 (27) : parsing [/etc/haproxy/haproxy.cfg:89] : status
code 413 not handled, using default code 403.
How should I configure HAProxy in order to deny with 413?
~jm
