Re: regression in 1.6.13 - wrong http status in logs and stats

[Willy Tarreau]

Hi Michal,

On Wed, Jul 12, 2017 at 04:06:16PM +0200, Michal Pasierb wrote:
> Hi,
> 
> I would like to report a regression in HAProxy 1.6.13 after upgrading from
> 1.6.9 in production :(
> 
> Reproduce with config:
> 
> ---
> global
>   log 127.0.0.1   local0
>   log 127.0.0.1   local1 notice
>   maxconn 4096
>   chroot /usr/share/haproxy
>   uid 99
>   gid 99
>   daemon
> 
> defaults
>   log global
>   mode http
>   option httplog
>   option dontlognull
>   option redispatch
>   stats enable
>   stats refresh 5s
>   timeout connect 5s
>   timeout client  30s
>   timeout server  30s
>   balance roundrobin
>   option forwardfor
> 
> listen Statistics
>   bind 192.168.122.202:
>   mode http
>   stats refresh 5s
>   stats uri /
> 
> frontend http-in
>   bind 192.168.122.202:9000
> 
>   acl is_root_path  path /
>   redirect location /my_custom_page if is_root_path
> 
> backend nomatch
>   redirect location http://www.google.com
> ---
> 
> Send a request with curl -v 192.168.122.202:9000/ - response is OK and has
> status code 302 but logs and statistics have 503 instead:
> 
> 192.168.122.202:37880 [12/Jul/2017:15:54:49.573] http-in http-in/
> 0/-1/-1/-1/2 503 305 - - LR-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"
> 
> git bisect shows this commit to be the culprit:
> 
> commit b12d699543adb84fa543297d12b64fce7ec94803
> Author: Christopher Faulet 
> Date:   Tue Mar 28 11:51:33 2017 +0200
> 
> BUG/MINOR: http: Fix conditions to clean up a txn and to handle the
> next request
> 
> I also tested 1.7.8 and 1.8-dev2 - they are OK. So it seems it is a
> backport issue.

I know what caused this, indeed a fix was backported and had this side
effect. I can produce a new version with a backport of the latest fix
that went into 1.7.8. Sorry for the mess :-(

Willy

DNS suffix for resolver

[Aleksandar Lazic]

Hi,

I have used today again my haproxy image 
https://hub.docker.com/r/me2digital/haproxy17/ in openshift.

There is a variable SERVICE_DEST which have the destination hostname for 
the server line in haproxy.

When I use just mongodb, that's the service name in openshift, it will 
not resolve because the resolver does not respect the 'search' line.

###
oc rsh haproxy17-3-gd94c cat /etc/resolv.conf
search 1-mongodb-test.svc.cluster.local svc.cluster.local cluster.local 
esrv.local
nameserver 10.40.96.55
nameserver 10.40.96.55
options ndots:5
###

will this be better handled with 1.8 dns code?

###
oc rsh haproxy17-3-gd94c haproxy -vv
HA-Proxy version 1.7.8 2017/07/07
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
  OPTIONS = USE_LINUX_SPLICE=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 
USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_TFO=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : yes
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe
###

-- 
Best Regards
Aleks

Re: [PATCH] BUG/MINOR: ssl: Be sure that SSLv3 connection methods exist for openssl < 1.1.0

[Willy Tarreau]

On Wed, Jul 12, 2017 at 03:54:28PM +0200, Emmanuel Hocdet wrote:
> Yes i'm confident because i worked a lot to abstract tls version/api support 
> with
> older/newer openssl versions. It's what i do with haproxy's methodVersions 
> table
> for ssl-min/max-ver support.
> What i'm missing is OPENSSL_NO_SSL3 define...

OK.

> > $ grep -rF methodVersions openssl-1.0.2k/
> > $ echo $?
> > 1
> 
> methodVersions is in ssl_sock.c, it will break nothing.

Ah sorry, thank you :-) As you can see I try to be careful these days!

Willy

regression in 1.6.13 - wrong http status in logs and stats

[Michał Pasierb]

Hi,

I would like to report a regression in HAProxy 1.6.13 after upgrading from
1.6.9 in production :(

Reproduce with config:

---
global
  log 127.0.0.1   local0
  log 127.0.0.1   local1 notice
  maxconn 4096
  chroot /usr/share/haproxy
  uid 99
  gid 99
  daemon

defaults
  log global
  mode http
  option httplog
  option dontlognull
  option redispatch
  stats enable
  stats refresh 5s
  timeout connect 5s
  timeout client  30s
  timeout server  30s
  balance roundrobin
  option forwardfor

listen Statistics
  bind 192.168.122.202:
  mode http
  stats refresh 5s
  stats uri /

frontend http-in
  bind 192.168.122.202:9000

  acl is_root_path  path /
  redirect location /my_custom_page if is_root_path

backend nomatch
  redirect location http://www.google.com
---

Send a request with curl -v 192.168.122.202:9000/ - response is OK and has
status code 302 but logs and statistics have 503 instead:

192.168.122.202:37880 [12/Jul/2017:15:54:49.573] http-in http-in/
0/-1/-1/-1/2 503 305 - - LR-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"

git bisect shows this commit to be the culprit:

commit b12d699543adb84fa543297d12b64fce7ec94803
Author: Christopher Faulet 
Date:   Tue Mar 28 11:51:33 2017 +0200

BUG/MINOR: http: Fix conditions to clean up a txn and to handle the
next request

I also tested 1.7.8 and 1.8-dev2 - they are OK. So it seems it is a
backport issue.

Regards,
Michal

Re: [PATCH] BUG/MINOR: ssl: Be sure that SSLv3 connection methods exist for openssl < 1.1.0

[Emmanuel Hocdet]

> Le 12 juil. 2017 à 15:23, Willy Tarreau  a écrit :
> 
> Hi Manu!
> 
> Please don't forget to CC Emeric and keep in mind that I still don't
> understand anything about openssl, so for me it's always a huge pain
> each time to try to have an opinion on openssl related changes.
> 

oops indeed

> On Wed, Jul 12, 2017 at 02:54:16PM +0200, Emmanuel Hocdet wrote:
>> 
>> Hi Willy,
>> 
>> I would like you consider this patches because Christopher's patch is false 
>> and
>> doesn't support other ssl libs and openssl >= 1.1.0.
> 
> OK so I guess we need to take it. Are you confident that it doesn't break
> older versions ? I'm asking because since we started to add support for
> openssl derivatives, we've probably had as many patches to fix build with
> them as patches needed to fix the build with openssl due to these patches,
> to the point that sometimes I'm wondering why we still make so many efforts
> supporting these libs given the amount of incompatibilities they cause :-(
> 

Yes i’m confident because i worked a lot to abstract tls version/api support 
with
older/newer openssl versions. It’s what i do with haproxy’s methodVersions table
for ssl-min/max-ver support.
What i’m missing is OPENSSL_NO_SSL3 define...


>> I sent my original patch with more comments and another with a little 
>> cleanup:
> 
> This one will definitely break :
> 
> Subject: [PATCH 2/2] MINOR: ssl: remove an unecessary SSL_OP_NO_* dependancy
> 
> Use methodVersions table to display "OpenSSL library supports".
> (...)
> - memprintf(, "%s\nOpenSSL library supports : "
> -#if SSL_OP_NO_SSLv3
> -   "SSLv3 "
> -#endif
> -#if SSL_OP_NO_TLSv1
> -   "TLSv1.0 "
> -#endif
> -#if SSL_OP_NO_TLSv1_1
> -   "TLSv1.1 "
> -#endif
> -#if SSL_OP_NO_TLSv1_2
> -   "TLSv1.2 "
> -#endif
> -#if SSL_OP_NO_TLSv1_3
> -   "TLSv1.3"
> -#endif
> -"", ptr);
> + memprintf(, "%s\nOpenSSL library supports :", ptr);
> + for (i = CONF_TLSV_MIN; i <= CONF_TLSV_MAX; i++)
> + if (methodVersions[i].option)
> + memprintf(, "%s %s", ptr, methodVersions[i].name);
> 
> $ grep -rF methodVersions openssl-1.0.2k/
> $ echo $?
> 1

methodVersions is in ssl_sock.c, it will break nothing.

++
Manu

Re: [PATCH] BUG/MINOR: ssl: Be sure that SSLv3 connection methods exist for openssl < 1.1.0

[Emeric Brun]

Hi Manu,

On 07/12/2017 03:23 PM, Willy Tarreau wrote:
> Hi Manu!
> 
> Please don't forget to CC Emeric and keep in mind that I still don't
> understand anything about openssl, so for me it's always a huge pain
> each time to try to have an opinion on openssl related changes.
> 
> On Wed, Jul 12, 2017 at 02:54:16PM +0200, Emmanuel Hocdet wrote:
>>
>> Hi Willy,
>>
>> I would like you consider this patches because Christopher's patch is false 
>> and
>> doesn't support other ssl libs and openssl >= 1.1.0.
> 
> OK so I guess we need to take it. Are you confident that it doesn't break
> older versions ? I'm asking because since we started to add support for
> openssl derivatives, we've probably had as many patches to fix build with
> them as patches needed to fix the build with openssl due to these patches,
> to the point that sometimes I'm wondering why we still make so many efforts
> supporting these libs given the amount of incompatibilities they cause :-(
> 
>> I sent my original patch with more comments and another with a little 
>> cleanup:

Same worries, the openssl 0.9.8 is still maintained in redhat 5 so we should be 
able to compile with this version.


> This one will definitely break :
> 
> Subject: [PATCH 2/2] MINOR: ssl: remove an unecessary SSL_OP_NO_* dependancy
> 
> Use methodVersions table to display "OpenSSL library supports".
> (...)
> - memprintf(, "%s\nOpenSSL library supports : "
> -#if SSL_OP_NO_SSLv3
> -   "SSLv3 "
> -#endif
> -#if SSL_OP_NO_TLSv1
> -   "TLSv1.0 "
> -#endif
> -#if SSL_OP_NO_TLSv1_1
> -   "TLSv1.1 "
> -#endif
> -#if SSL_OP_NO_TLSv1_2
> -   "TLSv1.2 "
> -#endif
> -#if SSL_OP_NO_TLSv1_3
> -   "TLSv1.3"
> -#endif
> -"", ptr);
> + memprintf(, "%s\nOpenSSL library supports :", ptr);
> + for (i = CONF_TLSV_MIN; i <= CONF_TLSV_MAX; i++)
> + if (methodVersions[i].option)
> + memprintf(, "%s %s", ptr, methodVersions[i].name);
> 
> $ grep -rF methodVersions openssl-1.0.2k/
> $ echo $?
> 1
> 
> Thanks,
> Willy
> 

R,
Emeric

Re: [PATCH] BUG/MINOR: ssl: Be sure that SSLv3 connection methods exist for openssl < 1.1.0

[Willy Tarreau]

Hi Manu!

Please don't forget to CC Emeric and keep in mind that I still don't
understand anything about openssl, so for me it's always a huge pain
each time to try to have an opinion on openssl related changes.

On Wed, Jul 12, 2017 at 02:54:16PM +0200, Emmanuel Hocdet wrote:
> 
> Hi Willy,
> 
> I would like you consider this patches because Christopher's patch is false 
> and
> doesn't support other ssl libs and openssl >= 1.1.0.

OK so I guess we need to take it. Are you confident that it doesn't break
older versions ? I'm asking because since we started to add support for
openssl derivatives, we've probably had as many patches to fix build with
them as patches needed to fix the build with openssl due to these patches,
to the point that sometimes I'm wondering why we still make so many efforts
supporting these libs given the amount of incompatibilities they cause :-(

> I sent my original patch with more comments and another with a little cleanup:

This one will definitely break :

Subject: [PATCH 2/2] MINOR: ssl: remove an unecessary SSL_OP_NO_* dependancy

Use methodVersions table to display "OpenSSL library supports".
(...)
-   memprintf(, "%s\nOpenSSL library supports : "
-#if SSL_OP_NO_SSLv3
- "SSLv3 "
-#endif
-#if SSL_OP_NO_TLSv1
- "TLSv1.0 "
-#endif
-#if SSL_OP_NO_TLSv1_1
- "TLSv1.1 "
-#endif
-#if SSL_OP_NO_TLSv1_2
- "TLSv1.2 "
-#endif
-#if SSL_OP_NO_TLSv1_3
- "TLSv1.3"
-#endif
-  "", ptr);
+   memprintf(, "%s\nOpenSSL library supports :", ptr);
+   for (i = CONF_TLSV_MIN; i <= CONF_TLSV_MAX; i++)
+   if (methodVersions[i].option)
+   memprintf(, "%s %s", ptr, methodVersions[i].name);

$ grep -rF methodVersions openssl-1.0.2k/
$ echo $?
1

Thanks,
Willy

Re: [PATCH] BUG/MINOR: ssl: Be sure that SSLv3 connection methods exist for openssl < 1.1.0

[Emmanuel Hocdet]

Hi Willy,

I would like you consider this patches because Christopher’s patch is false and
doesn’t support other ssl libs and openssl >= 1.1.0.

I sent my original patch with more comments and another with a little cleanup:

++
Manu




0001-BUG-MINOR-ssl-remove-haproxy-SSLv3-support-when-ssl-.patch
Description: Binary data


0002-MINOR-ssl-remove-an-unecessary-SSL_OP_NO_-dependancy.patch
Description: Binary data

Re: Question about "show servers state" and changing backends

[Baptiste]

On Tue, Jul 11, 2017 at 6:59 PM, Joseph Lynch  wrote:

> Ah, I didn't realize that id was the puid! Let me try that out!
>
> So just curious, if the puid is what we're using to uniquely identify
> servers, why can't we update the name dynamically? It seems like
> use-server and the like should use these puids rather than the names?
>
> -Joey
>
>
Because most configurations doesn't enforce the configuration 'id' (aka
server->puid). So the most reliable piece of information we have is what
the human has entered, hence the server "name" (aka server->id).
We may explore a way to improve things like, adding a flag when a
server->puid was enforced by configuration, so we know it's more reliable
than server->id and we should prefer using it first.

In a general manner, I think we should step back a bit and watch this
"server-state" feature from a different angle, because I have the feeling
that if we carry on like this, we may end up with a patchwork of exceptions
that may be boring to manage and understand at some point.
"server-state" is supposed to be used to give a new HAProxy process the
status the servers had in a previous running process.

Baptiste

Re: X-Real-IP = X-Forwarded-For

[Aleksandar Lazic]

Hi Andrey Zakabluk,

Andrey Zakabluk wrote on 12.07.2017:

> Hi! I Use
> HA-Proxy version 1.5.12 2015/05/02
> .
> Need add in http package option X-Real-IP.  X-Real-IP should be equal
> X-Forwarded-For. X-Forwarded-For be in package.
> I tried

> frontend http-in
> bind *:4016
> default_backend servers
> mode http
> option httplog
> log global
>capture cookie SERVERID len 32
>capture request header Host len 15
>capture request header X-Forwarded-For len 15
>capture request header Referrer len 15
>capture response header Content-length len 9
>capture response header Location len 15
>http-request set-header X-Client-IP req.hdr_ip([X-Forwarded-For])]

My naive solution would be

   http-request set-header X-Real-IP req.hdr_ip([X-Forwarded-For])]

> not help. Help me pls. 

-- 
Best Regards
Aleks

X-Real-IP = X-Forwarded-For

[Andrey Zakabluk]

Hi! I Use
HA-Proxy version 1.5.12 2015/05/02
.
Need add in http package option X-Real-IP.  X-Real-IP should be equal 
X-Forwarded-For. X-Forwarded-For be in package.
I tried

frontend http-in
bind *:4016
default_backend servers
mode http
option httplog
log global
   capture cookie SERVERID len 32
   capture request header Host len 15
   capture request header X-Forwarded-For len 15
   capture request header Referrer len 15
   capture response header Content-length len 9
   capture response header Location len 15
   http-request set-header X-Client-IP req.hdr_ip([X-Forwarded-For])]

not help. Help me pls.