Re: Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4
Hi Olivier, On Wed, May 06, 2020 at 06:29:46PM +0200, Olivier D wrote: > So to be clear : > I'm using 2.0.14 source code. In this version, patch 7f26391bc is already > applied and 02c88036a is not. > So applying 02c88036a did nothing (well, it triggers two different > non-working behaviour with Apache 2.4 patched, and a single behaviour > without the patch on remoteip). > And with clean source again, reverting 7f26391bc did the trick (on both > Apache 2.4 versions). Thanks. FYI I've reverted the faulty patch from 2.0 so if you use the latest snapshot or -git you'll be fine. I think I'll soon emit another set of 2.1 and 2.0 with more fixes. Cheers, Willy
Re: Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4
Hi again, Le mer. 6 mai 2020 à 17:47, Willy Tarreau a écrit : > Hi Olivier, > > On Wed, May 06, 2020 at 05:29:59PM +0200, Olivier D wrote: > > > Try applying this commit: > > > > > > > https://github.com/haproxy/haproxy/commit/02c88036a61e09d0676a2b6b4086af677b023b94 > > > > > > So this patch is not working for me, with or without patching Apache2 > with > > https://bz.apache.org/bugzilla/show_bug.cgi?id=63893 > > > > But "good news" : reverting 7f26391bc51 did the trick. > > This is sad. So this means that we've trained external components to > get used to our bugs and consider them to be the default behavior > despite what the doc says. > > > To make sure we are talking about the same things, I've attached both > > commits as patch files. > > - applying 7f26391bc.patch did not fix the issue > > - reverting 02c88036a.patch fixed the issue > > Then this is confusing because 7f2639 was already applied to your tree > and is supposed to be the one causing you the issue, and 02c88 was not > yet so you couldn't revert it. That's why I'd like to have a precise > description of the starting state and what operations you did which > worked and those which didn't. > My bad, I've inverted 7f26391bc and 02c88036a ... I should have used named patches instead of dealing with commit number. So to be clear : I'm using 2.0.14 source code. In this version, patch 7f26391bc is already applied and 02c88036a is not. So applying 02c88036a did nothing (well, it triggers two different non-working behaviour with Apache 2.4 patched, and a single behaviour without the patch on remoteip). And with clean source again, reverting 7f26391bc did the trick (on both Apache 2.4 versions). > > How safe is it to use 02c88036a reverted in production ? > > Either choice is safe for a given component. It's just that we send > wrong information on health checks, forcing other implementations to > implement bugs (see bug #511), that at least Dovecot doesn't support > a "relaxed" approach combining LOCAL with an address, and that from > what you're saying, Apache2 doesn't support LOCAL without an address. > > At this point I guess we'll have to revert all this from older branches > and provide a config option for 2.2+ to re-enable the old behavior for > compatibility with servers that got it wrong the first time. > Yes I understand. That's the price of fame :) HAProxy is now so widely used that softwares are implemented based on what HAProxy does instead of what the specs says. Olivier
Re: Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4
Hi Olivier, On Wed, May 06, 2020 at 05:29:59PM +0200, Olivier D wrote: > > Try applying this commit: > > > > https://github.com/haproxy/haproxy/commit/02c88036a61e09d0676a2b6b4086af677b023b94 > > > So this patch is not working for me, with or without patching Apache2 with > https://bz.apache.org/bugzilla/show_bug.cgi?id=63893 > > But "good news" : reverting 7f26391bc51 did the trick. This is sad. So this means that we've trained external components to get used to our bugs and consider them to be the default behavior despite what the doc says. > To make sure we are talking about the same things, I've attached both > commits as patch files. > - applying 7f26391bc.patch did not fix the issue > - reverting 02c88036a.patch fixed the issue Then this is confusing because 7f2639 was already applied to your tree and is supposed to be the one causing you the issue, and 02c88 was not yet so you couldn't revert it. That's why I'd like to have a precise description of the starting state and what operations you did which worked and those which didn't. > How safe is it to use 02c88036a reverted in production ? Either choice is safe for a given component. It's just that we send wrong information on health checks, forcing other implementations to implement bugs (see bug #511), that at least Dovecot doesn't support a "relaxed" approach combining LOCAL with an address, and that from what you're saying, Apache2 doesn't support LOCAL without an address. At this point I guess we'll have to revert all this from older branches and provide a config option for 2.2+ to re-enable the old behavior for compatibility with servers that got it wrong the first time. Willy
Re: Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4
Hello,
Le mer. 6 mai 2020 à 15:30, Tim Düsterhus a écrit :
> Olivier,
>
> > I was not aware there were any change in the way HAProxy was doing its
> > checks over proxy-protocol in 2.0.14 ... any hint ?
>
> This sounds like this issue we've seen with Dovecot:
> https://www.mail-archive.com/haproxy@formilux.org/msg36890.html
>
> Try applying this commit:
>
> https://github.com/haproxy/haproxy/commit/02c88036a61e09d0676a2b6b4086af677b023b94
So this patch is not working for me, with or without patching Apache2 with
https://bz.apache.org/bugzilla/show_bug.cgi?id=63893
But "good news" : reverting 7f26391bc51 did the trick.
To make sure we are talking about the same things, I've attached both
commits as patch files.
- applying 7f26391bc.patch did not fix the issue
- reverting 02c88036a.patch fixed the issue
How safe is it to use 02c88036a reverted in production ?
Olivier
--- src/connection.c
+++ src/connection.c
@@ -1247,6 +1247,7 @@ int make_proxy_line_v2(char *buf, int buf_len, struct
server *srv, struct connec
/* At least one of src or dst is not of AF_INET or AF_INET6 */
if ( !src
|| !dst
+ || conn_is_back(remote)
|| (src->ss_family != AF_INET && src->ss_family != AF_INET6)
|| (dst->ss_family != AF_INET && dst->ss_family != AF_INET6)) {
if (buf_len < PP2_HDR_LEN_UNSPEC)
@@ -1256,14 +1257,7 @@ int make_proxy_line_v2(char *buf, int buf_len, struct
server *srv, struct connec
ret = PP2_HDR_LEN_UNSPEC;
}
else {
- /* Note: due to historic compatibility with V1 which required
-* to send "PROXY" with local addresses for local connections,
-* we can end up here with the remote in fact being our outgoing
-* connection. We still want to send real addresses and LOCAL on
-* it.
-*/
- hdr->ver_cmd = PP2_VERSION;
- hdr->ver_cmd |= conn_is_back(remote) ? PP2_CMD_LOCAL :
PP2_CMD_PROXY;
+ hdr->ver_cmd = PP2_VERSION | PP2_CMD_PROXY;
/* IPv4 for both src and dst */
if (src->ss_family == AF_INET && dst->ss_family == AF_INET) {
if (buf_len < PP2_HDR_LEN_INET)
--- src/connection.c
+++ src/connection.c
@@ -1318,11 +1318,18 @@ int make_proxy_line_v2(char *buf, int buf_len, struct
server *srv, struct connec
ret = PP2_HDR_LEN_UNSPEC;
}
else {
+ /* Note: due to historic compatibility with V1 which required
+* to send "PROXY" with local addresses for local connections,
+* we can end up here with the remote in fact being our outgoing
+* connection. We still want to send real addresses and LOCAL on
+* it.
+*/
+ hdr->ver_cmd = PP2_VERSION;
+ hdr->ver_cmd |= conn_is_back(remote) ? PP2_CMD_LOCAL :
PP2_CMD_PROXY;
/* IPv4 for both src and dst */
if (src->ss_family == AF_INET && dst->ss_family == AF_INET) {
if (buf_len < PP2_HDR_LEN_INET)
return 0;
- hdr->ver_cmd = PP2_VERSION | PP2_CMD_PROXY;
hdr->fam = PP2_FAM_INET | PP2_TRANS_STREAM;
hdr->addr.ip4.src_addr = ((struct sockaddr_in
*)src)->sin_addr.s_addr;
hdr->addr.ip4.src_port = ((struct sockaddr_in
*)src)->sin_port;
@@ -1336,7 +1343,6 @@ int make_proxy_line_v2(char *buf, int buf_len, struct
server *srv, struct connec
if (buf_len < PP2_HDR_LEN_INET6)
return 0;
- hdr->ver_cmd = PP2_VERSION | PP2_CMD_PROXY;
hdr->fam = PP2_FAM_INET6 | PP2_TRANS_STREAM;
if (src->ss_family == AF_INET) {
v4tov6(, &((struct sockaddr_in
*)src)->sin_addr);
Re: Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4
Olivier, Am 06.05.20 um 15:15 schrieb Olivier D: > This morning I tried to upgrade HAProxy 2.0.13 to 2.0.14 but had to > rollback immediately : some backends checks started to fail. > Error reported was : SOCKERR - SSL handshake failure > > The backends failing have a specific configuration as follows (I removed > anything unnecessary to trigger the issue) > listen webtruc:443 > mode tcp > bind X.X.X.X:443 > server xxx X.X.X.X:443 check weight 5 send-proxy-v2-ssl-cn check-ssl > verify none > > Backend is an Apache 2.4 with "RemoteIPProxyProtocol On". > In apache logs I have : > [remoteip:error] [pid 1067 [client :26847] AH03507: > RemoteIPProxyProtocol: unsupported command 20 > > I can link this error to this bugreport : > https://bz.apache.org/bugzilla/show_bug.cgi?id=63893 > So I applied this patch to Apache 2.4 and then get this error : > HAproxy side: L7STS Bad request > Apache side : RemoteIPProxyProtocol data is missing, but required! Aborting > request. > > I was not aware there were any change in the way HAProxy was doing its > checks over proxy-protocol in 2.0.14 ... any hint ? This sounds like this issue we've seen with Dovecot: https://www.mail-archive.com/haproxy@formilux.org/msg36890.html Try applying this commit: https://github.com/haproxy/haproxy/commit/02c88036a61e09d0676a2b6b4086af677b023b94 Best regards Tim Düsterhus
Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4
Hello,
This morning I tried to upgrade HAProxy 2.0.13 to 2.0.14 but had to
rollback immediately : some backends checks started to fail.
Error reported was : SOCKERR - SSL handshake failure
The backends failing have a specific configuration as follows (I removed
anything unnecessary to trigger the issue)
listen webtruc:443
mode tcp
bind X.X.X.X:443
server xxx X.X.X.X:443 check weight 5 send-proxy-v2-ssl-cn check-ssl
verify none
Backend is an Apache 2.4 with "RemoteIPProxyProtocol On".
In apache logs I have :
[remoteip:error] [pid 1067 [client :26847] AH03507:
RemoteIPProxyProtocol: unsupported command 20
I can link this error to this bugreport :
https://bz.apache.org/bugzilla/show_bug.cgi?id=63893
So I applied this patch to Apache 2.4 and then get this error :
HAproxy side: L7STS Bad request
Apache side : RemoteIPProxyProtocol data is missing, but required! Aborting
request.
I was not aware there were any change in the way HAProxy was doing its
checks over proxy-protocol in 2.0.14 ... any hint ?
HAProxy -vv output :
HA-Proxy version 2.0.14 2020/04/02 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wno-implicit-fallthrough
-Wno-stringop-overflow -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_THREAD=0 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1
USE_ZLIB=1 USE_NS=
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE
-PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED
-REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE
+LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4
-MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO -NS +DL +RT -DEVICEATLAS
-51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=20).
Built with OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.44 2020-02-12
Running on PCRE version : 8.44 2020-02-12
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTXside=FE|BE mux=H2
h2 : mode=HTTP side=FEmux=H2
: mode=HTXside=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
