date:20171103

Logging errors during reload of haproxy

2017-11-03 Thread Veiko Kukk


Hi,

I noticed, while trying to reproduce conditions for another bug about 
processes never closing after restart, that sometimes reload causes 
logging errors displayed.


Following config section might be relevant:

global
  log /dev/log local0
  nbproc 3

defaults
  log /dev/log local0

frontend foo
  log /dev/log local1

...

# service haproxy reload
[ALERT] 306/110738 (29225) : sendmsg logger #1 failed: Resource 
temporarily unavailable (errno=11)
[ALERT] 306/110738 (29225) : sendmsg logger #1 failed: Resource 
temporarily unavailable (errno=11)
[ALERT] 306/110738 (29225) : sendmsg logger #1 failed: Resource 
temporarily unavailable (errno=11)


# haproxy -vv
HA-Proxy version 1.7.9 2017/08/18
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing 
-Wdeclaration-after-statement -fwrapv

  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3
Running on zlib version : 1.2.3
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
Running on PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND


Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[TRACE] trace
[COMP] compression

Veiko

Re: Logging errors during reload of haproxy

2017-11-03 Thread Veiko Kukk


On 11/03/2017 01:21 PM, Veiko Kukk wrote:

Hi,

I noticed, while trying to reproduce conditions for another bug about 
processes never closing after restart, that sometimes reload causes 
logging errors displayed.


Should read here "never closing after *reload*".

Veiko

[PATCH]: BUG/MINOR

2017-11-03 Thread David CARLIER

Hi

Here is a tiny diff while I was trying to build under OpenBSD (not sure if
few other oses are affected like Solaris but this change is not overkill I
think).

Kind regards.
From deb8a3aca9aec73d2edd3461e5bf2d0f7c9153f5 Mon Sep 17 00:00:00 2001
From: David Carlier 
Date: Fri, 3 Nov 2017 12:00:26 +
Subject: [PATCH] BUG/MINOR: stdarg.h inclusion

Needed for the memvprintf part, the va_list type.
Spotted during OpenBSD build.
---
 include/common/standard.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/common/standard.h b/include/common/standard.h
index 59809b4c..a06fb844 100644
--- a/include/common/standard.h
+++ b/include/common/standard.h
@@ -26,6 +26,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
-- 
2.14.2

Re: [ANNOUNCE] haproxy-1.8-rc1 : the last mile

2017-11-03 Thread Dmitry Sivachenko


> On 01 Nov 2017, at 02:20, Willy Tarreau  wrote:
> 
> Hi all!
> 


Hello,

several new warnings from clang, some look meaningful:

cc -Iinclude -Iebtree -Wall  -O2 -pipe  -fstack-protector -fno-strict-aliasing  
-fno-strict-aliasing -Wdeclaration-after-statement -fwrapv  
-Wno-address-of-packed-member -Wno-null-dereference -Wno-unused-label   
-DFREEBSD_PORTS-DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB  
-DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 
-DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL  -DUSE_PCRE -I/usr/local/include 
-DUSE_PCRE_JIT  -DCONFIG_HAPROXY_VERSION=\"1.8-rc1-901f75c\" 
-DCONFIG_HAPROXY_DATE=\"2017/10/31\" -c -o src/standard.o src/standard.c
src/server.c:875:14: warning: address of array 'check->desc' will always
  evaluate to 'true' [-Wpointer-bool-conversion]
if (check->desc)
~~  ~~~^~~~
src/server.c:914:14: warning: address of array 'check->desc' will always
  evaluate to 'true' [-Wpointer-bool-conversion]
if (check->desc)
~~  ~~~^~~~
src/server.c:958:14: warning: address of array 'check->desc' will always
  evaluate to 'true' [-Wpointer-bool-conversion]
if (check->desc)
~~  ~~~^~~~
src/cfgparse.c:5044:34: warning: implicit conversion from 'int' to 'char'
  changes value from 130 to -126 [-Wconstant-conversion]
  ...curproxy->check_req[5] = 130;
~ ^~~
src/cfgparse.c:5070:33: warning: implicit conversion from 'int' to 'char'
  changes value from 128 to -128 [-Wconstant-conversion]
  ...curproxy->check_req[5] = 128;
~ ^~~



cc -Iinclude -Iebtree -Wall  -O2 -pipe  -fstack-protector -fno-strict-aliasing  
-fno-strict-aliasing -Wdeclaration-after-statement -fwrapv  
-Wno-address-of-packed-member -Wno-null-dereference -Wno-unused-label   
-DFREEBSD_PORTS-DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB  
-DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 
-DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL  -DUSE_PCRE -I/usr/local/include 
-DUSE_PCRE_JIT  -DCONFIG_HAPROXY_VERSION=\"1.8-rc1-901f75c\" 
-DCONFIG_HAPROXY_DATE=\"2017/10/31\" -c -o src/sample.o src/sample.c
src/peers.c:255:16: warning: implicit conversion from 'int' to 'char' changes
  value from 133 to -123 [-Wconstant-conversion]
*msg_type = PEER_MSG_STKT_UPDATE_TIMED;
  ~ ^~
src/peers.c:257:16: warning: implicit conversion from 'int' to 'char' changes
  value from 134 to -122 [-Wconstant-conversion]
*msg_type = PEER_MSG_STKT_INCUPDATE_TIMED;
  ~ ^
src/peers.c:261:16: warning: implicit conversion from 'int' to 'char' changes
  value from 128 to -128 [-Wconstant-conversion]
*msg_type = PEER_MSG_STKT_UPDATE;
  ~ ^~~~
src/peers.c:263:16: warning: implicit conversion from 'int' to 'char' changes
  value from 129 to -127 [-Wconstant-conversion]
*msg_type = PEER_MSG_STKT_INCUPDATE;
  ~ ^~~
src/peers.c:450:11: warning: implicit conversion from 'int' to 'char' changes
  value from 130 to -126 [-Wconstant-conversion]
msg[1] = PEER_MSG_STKT_DEFINE;
   ~ ^~~~
src/peers.c:486:11: warning: implicit conversion from 'int' to 'char' changes
  value from 132 to -124 [-Wconstant-conversion]
msg[1] = PEER_MSG_STKT_ACK;
   ~ ^



cc -Iinclude -Iebtree -Wall  -O2 -pipe  -fstack-protector -fno-strict-aliasing  
-fno-strict-aliasing -Wdeclaration-after-statement -fwrapv  
-Wno-address-of-packed-member -Wno-null-dereference -Wno-unused-label   
-DFREEBSD_PORTS-DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB  
-DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 
-DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL  -DUSE_PCRE -I/usr/local/include 
-DUSE_PCRE_JIT  -DCONFIG_HAPROXY_VERSION=\"1.8-rc1-901f75c\" 
-DCONFIG_HAPROXY_DATE=\"2017/10/31\" -c -o src/freq_ctr.o src/freq_ctr.c
src/mux_h2.c:1734:15: warning: implicit conversion from enumeration type
  'enum h2_ss' to different enumeration type 'enum h2_cs'
  [-Wenum-conversion]
h2c->st0 = H2_SS_ERROR;
 ~ ^~~
src/mux_h2.c:2321:15: warning: implicit conversion from enumeration type
  'enum h2_ss' to different enumeration type 'enum h2_cs'
  [-Wenum-conversion]
h2c->st0 = H2_SS_ERROR;
 ~ ^~~
src/mux_h2.c:2435:15: warning: implicit conversion from enumeration type
  'enum h2_ss' to different enumeration type 'enum h2_cs'
  [-Wenum-conversion]
h2c->st0 = H2_SS_ERROR;

Quick question re errorloc urls

2017-11-03 Thread Franks Andy (IT Technical Architecture Manager)

Hi all,
  We have a test haproxy instance that is using a letscrypt certificate, and is 
in a DMZ zone. We have internal network servers delivering an application via 
https using an internal CA certificate, and all works fine - the client 
connects, sees the letscrypt cert and since the load balancer o/s has the 
internal CA in its trusted root certs, the verification works there too.

The query I have is where we fail both backend servers we're using and have the 
client redirect to an internal website errorloc directive via https to deliver 
a "sorry the service isn't available" :

Here's the config:

frontend ft_web_ssl
  bind 0.0.0.0:443 name https ssl crt 
/usr/local/bin/dehydrated/certs/portal.fqdn/concat.pem
  mode http
  option httplog
  option forwardfor
  tcp-request connection track-sc0 src table connections
  default_backend bk_portal_ssl

backend bk_portal_ssl
  mode http
  option httplog
  option httpchk GET /Login/Heartbeat HTTP/1.0\r\nHost:\ 
portal.sath.nhs.uk
  http-check expect rstatus 200
  balance roundrobin
  stick on src table connections
  cookie SERVERID insert nocache indirect
  default-server inter 2s rise 10 fall 20 on-marked-down 
shutdown-sessions
  errorloc 503 https://intranet/clinicalportal/clinicalportalholding.htm
  server RSH-CP-IIS1 192.168.176.175:443 cookie 1 check ssl
  server RSH-CP-IIS2 192.168.176.176:443 cookie 2 check ssl

First of all, the errorloc "redirection" from a 503 works fine but since this 
intranet page is configured using an internal CA certificate and for some 
reason the client doesn't see the letscrypt certificate, rather the one from 
the website itself, it therefore warns on the client browser if they don't have 
the internal CA root cert installed as trusted.

I would have thought that the client should get the letscrypt frontend 
certificate in this case? How come not, and is there any way of fixing this? 
I'd rather keep ssl between DMZ and intranet zone if possible; obviously using 
plain http for the errorloc 503 is one way around, but would rather not do this.

Thanks
Andy

Throughput issue after moving between kernels.

2017-11-03 Thread Mark Brookes

Hi All,

We have been investigating an issue with reduced throughput. (its quite
possible that its nothing to do with HAProxy.)
I thought I would just check here to see if this rings a bell with anyone.

We are currently looking to update our kernel from 3.10.18 to 4.4.49. It
appears that in the move from 3.x.x to 4.x.x at some point the kernel devs
change the tcp_mem calculation which results in halving the values based on
the same amount of RAM. Although that isnt the problem it just highlighted
it.

Our test setup is

Mutiple Clients --> Haproxy --> Real Server.

If I run a fairly heavy load using iperf through haproxy using the 3.10.18
kernel and I check -


cat proc/net/sockstat
sockets: used 193
TCP: inuse 116 orphan 0 tw 17 alloc 118 mem 25591
UDP: inuse 12 mem 3
UDPLITE: inuse 0
RAW: inuse 1
FRAG: inuse 0 memory 0

cat /proc/sys/net/ipv4/tcp_mem
89544 119392 179088

When I reboot into the 4.4.49 kernel and run the same test I get -

cat proc/net/sockstat
sockets: used 198
TCP: inuse 115 orphan 0 tw 18 alloc 117 mem 43957
UDP: inuse 12 mem 2
UDPLITE: inuse 0
RAW: inuse 1
FRAG: inuse 0 memory 0

cat /proc/sys/net/ipv4/tcp_mem
44721 59631 89442

Haproxy --
Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
-Wdeclaration-after-statement -fwrapv
  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_STATIC_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3
Running on zlib version : 1.2.3
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2j-fips  26 Sep 2016
Running on OpenSSL version : OpenSSL 1.0.2j-fips  26 Sep 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
Running on PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[TRACE] trace
[COMP] compression

Ive tried 4.4.95 kernel and get the same result Ive also tried the 4.9.59.

Ive tried the latest 1.7.9 HAProxy too.

Does anyone have any ideas?

Thanks
-- 

Mark Brookes
Loadbalancer.org Ltd.
www.loadbalancer.org


+44 (0)330 380 1064
m...@loadbalancer.org

Re: Logging errors during reload of haproxy

2017-11-03 Thread Lukas Tribus

Hello Veiko,


2017-11-03 12:21 GMT+01:00 Veiko Kukk :
> Hi,
>
> I noticed, while trying to reproduce conditions for another bug about
> processes never closing after restart, that sometimes reload causes logging
> errors displayed.
>
> Following config section might be relevant:
>
> global
>   log /dev/log local0
>   nbproc 3
>
> defaults
>   log /dev/log local0
>
> frontend foo
>   log /dev/log local1
>
> ...
>
> # service haproxy reload
> [ALERT] 306/110738 (29225) : sendmsg logger #1 failed: Resource temporarily
> unavailable (errno=11)

Well the destination logging socket is unavailable. I don't think
there is a lot to do here
on the haproxy side, this mostly depends on the destination socket and
the kernel.

I would suggest you use a UDP destination instead. That should be
better suited to
handle logging at this rate.



> another bug about processes never closing after reload

Unless you are hitting a bug already fixed (make sure you use a
current stable release), it's
likely that long running sessions keep haproxy running.

Use the hard-stop-after directive to limit the time haproxy spends in
this state:
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#3.1-hard-stop-after



Regards,
Lukas

Re: Logging errors during reload of haproxy

2017-11-03 Thread Veiko Kukk


Hi Lukas,

On 11/03/2017 02:53 PM, Lukas Tribus wrote:

# service haproxy reload
[ALERT] 306/110738 (29225) : sendmsg logger #1 failed: Resource temporarily
unavailable (errno=11)


Well the destination logging socket is unavailable. I don't think
there is a lot to do here
on the haproxy side, this mostly depends on the destination socket and
the kernel.

I would suggest you use a UDP destination instead. That should be
better suited to
handle logging at this rate.


This is a test system with not much load other than my little 'ab -c 10 
...' is creating. We have unix logging everywhere locally, works even 
under heavy load.


First i suspected change in config where i added 'log /dev/log local0', 
but after commenting that, those messages are still appear. Once per 
process after reload, every time when doing quick reloads e.g. for i in 
{1..10}; do service haproxy reload; done But sometimes even when not 
restarting quickly. I have cronjob that runs after every 3 minutes and 
reloads haproxy, then this error appears sometimes, not each time.



another bug about processes never closing after reload


Unless you are hitting a bug already fixed (make sure you use a
current stable release), it's
likely that long running sessions keep haproxy running.

Use the hard-stop-after directive to limit the time haproxy spends in
this state:
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#3.1-hard-stop-after


I would not comment hanging process bug more under this thread, because 
it's off topic. Will create new thread for that. Planned anyway, but 
wanted first create reproduction instructions. So far, it's quite random...


Regards,
Veiko

Re: [PATCH]: BUG/MINOR

2017-11-03 Thread Willy TARREAU

Hi David,

On Fri, Nov 03, 2017 at 12:03:53PM +, David CARLIER wrote:
> Hi
> 
> Here is a tiny diff while I was trying to build under OpenBSD (not sure if
> few other oses are affected like Solaris but this change is not overkill I
> think).

Cool, thanks, now applied. By the way since you're testing on openbsd,
could you try with "make USE_THREAD=1" to see if threads build correctly
or not ? If so we can enable them by default in the makefile (you can do
it by setting USE_THREAD=implicit in the respective section, look for
linux2628 and freebsd to have an idea).

Willy

Re: Logging errors during reload of haproxy

2017-11-03 Thread Lukas Tribus

Hello,



> This is a test system with not much load other than my little 'ab -c 10 ...'
> is creating. We have unix logging everywhere locally, works even under heavy
> load.

Be that as it may, this is a syscall returning an error:

http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/log.c;h=88e0d07a281d8aeb7cbdf0613f8ad6e74c99f8e5;hb=HEAD#l1296

I'm not sure what we can do to improve the situation ... your kernel
is telling you that the socket /dev/log is unavailable.


Read this thread for some tuning advice on the kernel side:
http://comments.gmane.org/gmane.comp.web.haproxy/4716



>> Use the hard-stop-after directive to limit the time haproxy spends in
>> this state:
>>
>> https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#3.1-hard-stop-after
>
>
> I would not comment hanging process bug more under this thread, because it's
> off topic. Will create new thread for that

Agreed.



Regards,
Lukas

Re: [PATCH]: BUG/MINOR

2017-11-03 Thread David CARLIER

Sure thing. and it does :-)

On 3 November 2017 at 14:05, Willy TARREAU  wrote:

> Hi David,
>
> On Fri, Nov 03, 2017 at 12:03:53PM +, David CARLIER wrote:
> > Hi
> >
> > Here is a tiny diff while I was trying to build under OpenBSD (not sure
> if
> > few other oses are affected like Solaris but this change is not overkill
> I
> > think).
>
> Cool, thanks, now applied. By the way since you're testing on openbsd,
> could you try with "make USE_THREAD=1" to see if threads build correctly
> or not ? If so we can enable them by default in the makefile (you can do
> it by setting USE_THREAD=implicit in the respective section, look for
> linux2628 and freebsd to have an idea).
>
> Willy
>
From a9bc473aa4da66115f4a57a35b17077f48ebf826 Mon Sep 17 00:00:00 2001
From: David Carlier 
Date: Fri, 3 Nov 2017 14:35:44 +
Subject: [PATCH] BUILD: enable USE_THREAD for OpenBSD build.

---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index f066f310..c897c025 100644
--- a/Makefile
+++ b/Makefile
@@ -335,6 +335,7 @@ ifeq ($(TARGET),openbsd)
   USE_KQUEUE = implicit
   USE_TPROXY = implicit
   USE_ACCEPT4= implicit
+  USE_THREAD = implicit
 else
 ifeq ($(TARGET),netbsd)
   # This is for NetBSD
-- 
2.14.2

Re: [ANNOUNCE] haproxy-1.8-rc1 : the last mile

2017-11-03 Thread Willy Tarreau

Hi Dmitry,

On Fri, Nov 03, 2017 at 03:11:21PM +0300, Dmitry Sivachenko wrote:
> Hello,
> 
> several new warnings from clang, some look meaningful:

Thanks, Olivier also reported some of them. Some are valid or easy
to address, others might need some -Wno-something I guess.

> src/server.c:875:14: warning: address of array 'check->desc' will always
>   evaluate to 'true' [-Wpointer-bool-conversion]
> if (check->desc)
> ~~  ~~~^~~~

This one needs to be fixed, I think we moved from a pointer to a char[]
and the test is not needed anymore.

> src/cfgparse.c:5044:34: warning: implicit conversion from 'int' to 'char'
>   changes value from 130 to -126 [-Wconstant-conversion]
>   ...curproxy->check_req[5] = 130;
> ~ ^~~
> src/cfgparse.c:5070:33: warning: implicit conversion from 'int' to 'char'
>   changes value from 128 to -128 [-Wconstant-conversion]
>   ...curproxy->check_req[5] = 128;
> ~ ^~~

We've seen this one the other way around in the past, you assing a negative
number to shut it up and it complains on arm where chars are unsigned. I
*seem* to remember that it doesn't complain when entered in hexadecimal
form though, we'll have to try (if you want to, feel free to send a patch
if it works using 0x82 and 0x80).

> cc -Iinclude -Iebtree -Wall  -O2 -pipe  -fstack-protector 
> -fno-strict-aliasing  -fno-strict-aliasing -Wdeclaration-after-statement 
> -fwrapv  -Wno-address-of-packed-member -Wno-null-dereference 
> -Wno-unused-label   -DFREEBSD_PORTS-DTPROXY -DCONFIG_HAP_CRYPT 
> -DUSE_GETADDRINFO -DUSE_ZLIB  -DENABLE_POLL -DENABLE_KQUEUE 
> -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 -DUSE_THREAD 
> -DUSE_OPENSSL  -DUSE_PCRE -I/usr/local/include -DUSE_PCRE_JIT  
> -DCONFIG_HAPROXY_VERSION=\"1.8-rc1-901f75c\" 
> -DCONFIG_HAPROXY_DATE=\"2017/10/31\" -c -o src/sample.o src/sample.c
> src/peers.c:255:16: warning: implicit conversion from 'int' to 'char' changes
>   value from 133 to -123 [-Wconstant-conversion]
> *msg_type = PEER_MSG_STKT_UPDATE_TIMED;
>   ~ ^~

Same here, though this one is an enum and I would hate to have to start
to use casts for this, casts are only made to secretly hide bugs :-(
I think the pointer could be an unsigned char though. We'll have to
check if it yells somewhere else.

> cc -Iinclude -Iebtree -Wall  -O2 -pipe  -fstack-protector 
> -fno-strict-aliasing  -fno-strict-aliasing -Wdeclaration-after-statement 
> -fwrapv  -Wno-address-of-packed-member -Wno-null-dereference 
> -Wno-unused-label   -DFREEBSD_PORTS-DTPROXY -DCONFIG_HAP_CRYPT 
> -DUSE_GETADDRINFO -DUSE_ZLIB  -DENABLE_POLL -DENABLE_KQUEUE 
> -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 -DUSE_THREAD 
> -DUSE_OPENSSL  -DUSE_PCRE -I/usr/local/include -DUSE_PCRE_JIT  
> -DCONFIG_HAPROXY_VERSION=\"1.8-rc1-901f75c\" 
> -DCONFIG_HAPROXY_DATE=\"2017/10/31\" -c -o src/freq_ctr.o src/freq_ctr.c
> src/mux_h2.c:1734:15: warning: implicit conversion from enumeration type
>   'enum h2_ss' to different enumeration type 'enum h2_cs'
>   [-Wenum-conversion]
> h2c->st0 = H2_SS_ERROR;
>  ~ ^~~

I'm aware (and ashamed) of these ones already, I need to understand what I
wanted to set to know how to fix it (currently dealing with a much uglier
one).

> src/mux_h2.c:526:24: warning: unused function 'h2_get_n64' [-Wunused-function]

Ah now it starts to report this also for inlines ? It'll get pretty annoying
then as it prevents one from creating all the useful tools at once for later
use :-/  No idea how to shut up this one, maybe in the long term we'll also
need -Wno-unused-function.

> src/cache.c:176:7: warning: variable 'ret' is used uninitialized whenever 'if'
>   condition is false [-Wsometimes-uninitialized]
> if (len) {
> ^~~
> src/cache.c:202:7: note: uninitialized use occurs here
> if ((ret != len) ||
>  ^~~
> src/cache.c:176:3: note: remove the 'if' if its condition is always true
> if (len) {
> ^
> src/cache.c:151:9: note: initialize the variable 'ret' to silence this warning
> int ret;
>^
> = 0

Thanks, these ones may indeed be useful.

> src/cache.c:566:9: warning: implicit conversion from enumeration type
>   'enum act_return' to different enumeration type 'enum act_parse_ret'
>   [-Wenum-conversion]
> return ACT_RET_ERR;
> ~~ ^~~

I noted these ones as well.

> In file included from ../../ebtree/ebtree.c:21:
> ../../ebtree/ebtree.h:469:35: warning: taking address of packed member
>   'branches' of class or structure 'eb_node' may result in an unaligned
>   pointer value [-Waddress-of-packed-member]
> eb_troot_t *new_left = eb_dotag(&new->branches, EB_LEFT);
>

Re: [PATCH]: BUG/MINOR

2017-11-03 Thread Willy TARREAU

On Fri, Nov 03, 2017 at 02:37:24PM +, David CARLIER wrote:
> Sure thing. and it does :-)

thank you!
Willy

[PATCH] : BUG/MINOR

2017-11-03 Thread David CARLIER

Hi again,

while testing a build, clang spotted the health desc field improper check
so here a little patch proposal. That should be all after it :-).

Kind regards.
From e96a3190ada573dc39eb16e721dd1b9c297f9c45 Mon Sep 17 00:00:00 2001
From: David Carlier 
Date: Fri, 3 Nov 2017 14:41:46 +
Subject: [PATCH] BUG/MINOR: server: check desc field is not a pointer.

The desc field was checked a pointer rather than an array.
Triggered by clang build.
---
 src/server.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/server.c b/src/server.c
index 37f90d8c..2ade4137 100644
--- a/src/server.c
+++ b/src/server.c
@@ -872,7 +872,7 @@ void srv_set_stopped(struct server *s, const char *reason, struct check *check)
 		strlcpy2(s->op_st_chg.reason, reason, sizeof(s->op_st_chg.reason));
 	}
 	else if (check) {
-		if (check->desc)
+		if (check->desc[0] != '\0')
 			strlcpy2(s->op_st_chg.reason, check->desc, sizeof(s->op_st_chg.reason));
 		s->op_st_chg.code = check->code;
 		s->op_st_chg.status = check->status;
@@ -911,7 +911,7 @@ void srv_set_running(struct server *s, const char *reason, struct check *check)
 		strlcpy2(s->op_st_chg.reason, reason, sizeof(s->op_st_chg.reason));
 	}
 	else if (check) {
-		if (check->desc)
+		if (check->desc[0] != '\0')
 			strlcpy2(s->op_st_chg.reason, check->desc, sizeof(s->op_st_chg.reason));
 		s->op_st_chg.code = check->code;
 		s->op_st_chg.status = check->status;
@@ -955,7 +955,7 @@ void srv_set_stopping(struct server *s, const char *reason, struct check *check)
 		strlcpy2(s->op_st_chg.reason, reason, sizeof(s->op_st_chg.reason));
 	}
 	else if (check) {
-		if (check->desc)
+		if (check->desc[0] != '\0')
 			strlcpy2(s->op_st_chg.reason, check->desc, sizeof(s->op_st_chg.reason));
 		s->op_st_chg.code = check->code;
 		s->op_st_chg.status = check->status;
-- 
2.14.2

Re: [PATCH] : BUG/MINOR

2017-11-03 Thread Willy Tarreau

On Fri, Nov 03, 2017 at 02:45:43PM +, David CARLIER wrote:
> Hi again,
> 
> while testing a build, clang spotted the health desc field improper check
> so here a little patch proposal. That should be all after it :-).

I *think* that in fact we just have to remove the test. We need to
look through history to find the original need, but I guess that the
description field used to be allocated using malloc() or strdup() in
the past and changed to an array afterwards.

Willy

Re: Quick question re errorloc urls

2017-11-03 Thread Lukas Tribus

Hello Andy,


2017-11-03 13:44 GMT+01:00 Franks Andy (IT Technical Architecture
Manager) :
> First of all, the errorloc “redirection” from a 503 works fine but since
> this intranet page is configured using an internal CA certificate and for
> some reason the client doesn’t see the letscrypt certificate, rather the one
> from the website itself, it therefore warns on the client browser if they
> don’t have the internal CA root cert installed as trusted.

Its is a redirection. Not sure why you put it in quotes, this is a
proper redirection which
means the browser will close the connection to haproxy and opens a
direct connection to
your intranet page, which then, of course will serve the internal certificate.

If you open the intranet page in your browser:
https://intranet/clinicalportal/clinicalportalholding.htm

it won't use the letsencrypt certificate either, because the request
does not go through
haproxy.



> I would have thought that the client should get the letscrypt frontend
> certificate in this case? How come not, and is there any way of fixing this?

You need to figure out what you would like haproxy to do. Clealry a
redirect is not
what you want.



> I’d rather keep ssl between DMZ and intranet zone if possible; obviously
> using plain http for the errorloc 503 is one way around, but would rather
> not do this.

So no redirect, no plain http error either. Seems like you need to handle
it in a different backend then.


Something like this (first line in the frontend before default_backend):
use_backend bk_clinicalportalholding if nbsrv(bk_portal_ssl) eq 0

backend bk_clinicalportalholding
 mode http
 http-request set-uri /clinicalportal/clinicalportalholding.htm
 server intranet intranet:443 ssl verify none




Lukas

Error ‘NAME_MAX’ undeclared in HAProxy 1.8 on Solaris 11.3 (64-bit)

2017-11-03 Thread Daniel Heitepriem

Hi everyone,

I tried to compile the recent HAProxy 1.8 (pulled from the git
repository) on Solaris 11.3 (x86) with these settings:
gmake TARGET=solaris CPU=generic USE_TPROXY=1 USE_ZLIB=1 USE_OPENSSL=1
USE_PCRE=1 USE_GETADDRINFO=1 USE_REGPARM=1 DEFINE="-D_XOPEN_SOURCE=600"

"-D_XOPEN_SOURCE=600" is necessary on my Solaris installation as we're
running GCC5 and the default Makefile ships with XOPEN_SOURCE=500

Despite a few warnings regarding SPIN_LOCK

:0:0: note: this is the location of the previous
definition
In file included from include/common/memory.h:30:0,
 from include/common/chunk.h:29,
 from include/common/standard.h:36,
 from include/common/ticks.h:56,
 from src/ev_poll.c:22:
include/common/hathreads.h:75:0: warning: "SPIN_LOCK" redefined
 #define SPIN_LOCK(lbl, l)    do { /* do nothing */ } while(0)
 ^
In file included from /usr/include/sys/t_lock.h:15:0,
 from /usr/include/sys/stream.h:18,
 from /usr/include/netinet/in.h:66,
 from /usr/include/sys/socket.h:29,
 from include/common/compat.h:29,
 from src/ev_poll.c:20:
/usr/include/sys/machlock.h:60:0: note: this is the location of the
previous definition
 #define SPIN_LOCK(pl) ((pl) > ipltospl(LOCK_LEVEL))


I get an error which doesn't let me compile this version of HAProxy:

src/haproxy.c: In function ‘get_old_sockets’:
src/haproxy.c:971:31: error: ‘NAME_MAX’ undeclared (first use in
this function)
  tmpbuf = malloc(fd_nb * (1 + NAME_MAX + 1 + IFNAMSIZ + sizeof(int)));
   ^
src/haproxy.c:971:31: note: each undeclared identifier is reported
only once for each function it appears in
gmake: *** [src/haproxy.o] Error 1


Is this a known issue so far?

Thanks and regards,
Daniel

Re: [PATCH] : BUG/MINOR

2017-11-03 Thread David CARLIER

Sure makes sense in this case. Cheers.

On 3 November 2017 at 14:50, Willy Tarreau  wrote:

> On Fri, Nov 03, 2017 at 02:45:43PM +, David CARLIER wrote:
> > Hi again,
> >
> > while testing a build, clang spotted the health desc field improper check
> > so here a little patch proposal. That should be all after it :-).
>
> I *think* that in fact we just have to remove the test. We need to
> look through history to find the original need, but I guess that the
> description field used to be allocated using malloc() or strdup() in
> the past and changed to an array afterwards.
>
> Willy
>
From 8c66a15fe7300f90280ca1e047aeada03a9fe54e Mon Sep 17 00:00:00 2001
From: David Carlier 
Date: Fri, 3 Nov 2017 14:41:46 +
Subject: [PATCH] BUG/MINOR: server: removing desc field check.

Seemingly, desc field was dynamically allocated before, now
it is an array we jsut remove the check.
---
 src/server.c | 9 +++--
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/src/server.c b/src/server.c
index 37f90d8c..35115f25 100644
--- a/src/server.c
+++ b/src/server.c
@@ -872,8 +872,7 @@ void srv_set_stopped(struct server *s, const char *reason, struct check *check)
 		strlcpy2(s->op_st_chg.reason, reason, sizeof(s->op_st_chg.reason));
 	}
 	else if (check) {
-		if (check->desc)
-			strlcpy2(s->op_st_chg.reason, check->desc, sizeof(s->op_st_chg.reason));
+		strlcpy2(s->op_st_chg.reason, check->desc, sizeof(s->op_st_chg.reason));
 		s->op_st_chg.code = check->code;
 		s->op_st_chg.status = check->status;
 		s->op_st_chg.duration = check->duration;
@@ -911,8 +910,7 @@ void srv_set_running(struct server *s, const char *reason, struct check *check)
 		strlcpy2(s->op_st_chg.reason, reason, sizeof(s->op_st_chg.reason));
 	}
 	else if (check) {
-		if (check->desc)
-			strlcpy2(s->op_st_chg.reason, check->desc, sizeof(s->op_st_chg.reason));
+		strlcpy2(s->op_st_chg.reason, check->desc, sizeof(s->op_st_chg.reason));
 		s->op_st_chg.code = check->code;
 		s->op_st_chg.status = check->status;
 		s->op_st_chg.duration = check->duration;
@@ -955,8 +953,7 @@ void srv_set_stopping(struct server *s, const char *reason, struct check *check)
 		strlcpy2(s->op_st_chg.reason, reason, sizeof(s->op_st_chg.reason));
 	}
 	else if (check) {
-		if (check->desc)
-			strlcpy2(s->op_st_chg.reason, check->desc, sizeof(s->op_st_chg.reason));
+		strlcpy2(s->op_st_chg.reason, check->desc, sizeof(s->op_st_chg.reason));
 		s->op_st_chg.code = check->code;
 		s->op_st_chg.status = check->status;
 		s->op_st_chg.duration = check->duration;
-- 
2.14.2

Re: Throughput issue after moving between kernels.

2017-11-03 Thread Krishna Kumar (Engineering)

Though it would not cause your problem, the reason for this is:

In 3.10.18:
https://elixir.free-electrons.com/linux/v3.10.18/source/net/ipv4/tcp.c

void tcp_init_mem
(struct
net *net){
unsigned long limit
 =
nr_free_buffer_pages
()
/ 8;
limit 
= max (limit
,
128UL);
net->ipv4.sysctl_tcp_mem
[0]
= limit 
/ 4 * 3;
net->ipv4.sysctl_tcp_mem
[1]
= limit ;
net->ipv4.sysctl_tcp_mem
[2]
= net->ipv4.sysctl_tcp_mem
[0]
* 2;}

In 4.4.49:
https://elixir.free-electrons.com/linux/v4.4.49/source/net/ipv4/tcp.c

static void __init

tcp_init_mem 
(void){
unsigned long limit
 =
nr_free_buffer_pages
()
/ 16;

limit  =
max (limit
, 128UL);
sysctl_tcp_mem
[0]
= limit 
/ 4 * 3;/* 4.68 % */
sysctl_tcp_mem
[1]
= limit ;  
/*
6.25 % */
sysctl_tcp_mem
[2]
= sysctl_tcp_mem
[0]
* 2;/* 9.37 % */}

As you can see, the limit is halved in 4.4.49. The relevant commit:

commit b66e91ccbc34ebd5a2f90f9e1bc1597e2924a500
Author: Eric Dumazet 
Date:   Fri May 15 12:39:30 2015 -0700

tcp: halves tcp_mem[] limits

Allowing tcp to use ~19% of physical memory is way too much,
and allowed bugs to be hidden. Add to this that some drivers use a full
page per incoming frame, so real cost can be twice the advertized one.

Reduce tcp_mem by 50 % as a first step to sanity.

tcp_mem[0,1,2] defaults are now 4.68%, 6.25%, 9.37% of physical memory.
\
Signed-off-by: Eric Dumazet 
Signed-off-by: David S. Miller 


Regards,
- Krishna


On Fri, Nov 3, 2017 at 6:19 PM, Mark Brookes  wrote:

> Hi All,
>
> We have been investigating an issue with reduced throughput. (its quite
> possible that its nothing to do with HAProxy.)
> I thought I would just check here to see if this rings a bell with anyone.
>
> We are currently looking to update our kernel from 3.10.18 to 4.4.49. It
> appears that in the move from 3.x.x to 4.x.x at some point the kernel devs
> change the tcp_mem calculation which results in halving the values based on
> the same amount of RAM. Although that isnt the problem it just highlighted
> it.
>
> Our test setup is
>
> Mutiple Clients --> Haproxy --> Real Server.
>
> If I run a fairly heavy load using iperf through haproxy using the 3.10.18
> kernel and I check -
>
>
> cat proc/net/sockstat
> sockets: used 193
> TCP: inuse 116 orphan 0 tw 17 alloc 118 mem 25591
> UDP: inuse 12 mem 3
> UDPLITE: inuse 0
> RAW: inuse 1
> FRAG: inuse 0 memory 0
>
> cat /proc/sys/net/ipv4/tcp_mem
> 89544 119392 179088
>
> When I reboot into the 4.4.49 kernel and run the same test I get -
>
> cat proc/net/sockstat
> sockets: used 198
> TCP: inuse 115 orphan 0 tw 18 alloc 117 mem 43957
> UDP: inuse 12 mem 2
> UDPLITE: inuse 0
> RAW: inuse 1
> FRAG: inuse 0 memory 0
>
> cat /proc/sys/net/ipv4/tcp_mem
> 44721 59631 89442
>
> Haproxy --
> Build options :
>   TARGET  = linux2628
>   CPU = generic
>   CC  = gcc
>   CFLAGS  = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
> -Wdeclaration-after-statement -fwrapv
>   OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_STATIC_PCRE=1
>
> Default settings :
>   maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>
> Encrypted password support via crypt(3): yes
> Built with zlib version : 1.2.3
> Running on zlib version : 1.2.3
> Compressi

[PATCHES] TLS 1.3 session resumption and early data to servers

2017-11-03 Thread Olivier Houchard

Hi,

The attached patches makes TLS 1.3 session resumption work (it is a bit
different than the previous version, as the session is created after the
handshake), and enable sending early data to the server, as long as the
client used early data (we can't afford to send early data without knowing
if the client can handle a 425 too early, as that's the only thing we can
return if the server rejected the early data, as we no longer have them to
resend them).
Any feedback is welcome, as it's been mostly tested with only OpenSSL 1.1.1.

Regards,

Olivier
>From 7db328b4e5028a80c9817049108f5625513a87e8 Mon Sep 17 00:00:00 2001
From: Olivier Houchard 
Date: Thu, 2 Nov 2017 19:04:38 +0100
Subject: [PATCH 1/4] BUG/MINOR; ssl: Don't assume we have a ssl_bind_conf
 because a SNI is matched.

We only have a ssl_bind_conf if crt-list is used, however we can still
match a certificate SNI, so don't assume we have a ssl_bind_conf.
---
 src/ssl_sock.c | 12 +++-
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index b1d39dbbd..66930d220 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -2267,11 +2267,13 @@ static int ssl_sock_switchctx_cbk(SSL *ssl, int *al, 
void *arg)
/* switch ctx */
struct ssl_bind_conf *conf = container_of(node, struct sni_ctx, 
name)->conf;
ssl_sock_switchctx_set(ssl, container_of(node, struct sni_ctx, 
name)->ctx);
-   methodVersions[conf->ssl_methods.min].ssl_set_version(ssl, 
SET_MIN);
-   methodVersions[conf->ssl_methods.max].ssl_set_version(ssl, 
SET_MAX);
-   if (conf->early_data)
-   allow_early = 1;
-   goto allow_early;
+   if (conf) {
+   
methodVersions[conf->ssl_methods.min].ssl_set_version(ssl, SET_MIN);
+   
methodVersions[conf->ssl_methods.max].ssl_set_version(ssl, SET_MAX);
+   if (conf->early_data)
+   allow_early = 1;
+   }
+   goto allow_early;
}
 #if (!defined SSL_NO_GENERATE_CERTIFICATES)
if (s->generate_certs && ssl_sock_generate_certificate(trash.str, s, 
ssl)) {
-- 
2.13.0

>From c2728dd7ee527c1eb99588b17eb4f917d622fef1 Mon Sep 17 00:00:00 2001
From: Olivier Houchard 
Date: Fri, 3 Nov 2017 13:43:35 +0100
Subject: [PATCH 2/4] MINOR: ssl: Handle session resumption with TLS 1.3

With TLS 1.3, session aren't established until after the main handshake
has completed. So we can't just rely on calling SSL_get1_session(). Instead,
we now register a callback for the "new session" event. This should work for
previous versions of TLS as well.
---
 src/ssl_sock.c | 30 --
 1 file changed, 20 insertions(+), 10 deletions(-)

diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 66930d220..1dd3a4488 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -3852,6 +3852,23 @@ static int sh_ssl_sess_store(unsigned char *s_id, 
unsigned char *data, int data_
return 1;
 }
 
+/* SSL callback used when a new session is created while connecting to a 
server */
+static int ssl_sess_new_srv_cb(SSL *ssl, SSL_SESSION *sess)
+{
+   struct connection *conn = SSL_get_app_data(ssl);
+
+   /* check if session was reused, if not store current session on server 
for reuse */
+   if (objt_server(conn->target)->ssl_ctx.reused_sess[tid]) {
+   
SSL_SESSION_free(objt_server(conn->target)->ssl_ctx.reused_sess[tid]);
+   objt_server(conn->target)->ssl_ctx.reused_sess[tid] = NULL;
+   }
+
+   if (!(objt_server(conn->target)->ssl_ctx.options & SRV_SSL_O_NO_REUSE))
+   objt_server(conn->target)->ssl_ctx.reused_sess[tid] = 
SSL_get1_session(conn->xprt_ctx);
+
+   return 1;
+}
+
 /* SSL callback used on new session creation */
 int sh_ssl_sess_new_cb(SSL *ssl, SSL_SESSION *sess)
 {
@@ -4580,7 +4597,9 @@ int ssl_sock_prepare_srv_ctx(struct server *srv)
 #endif
}
 
-   SSL_CTX_set_session_cache_mode(srv->ssl_ctx.ctx, SSL_SESS_CACHE_OFF);
+   SSL_CTX_set_session_cache_mode(srv->ssl_ctx.ctx, SSL_SESS_CACHE_CLIENT |
+   SSL_SESS_CACHE_NO_INTERNAL_STORE);
+   SSL_CTX_sess_set_new_cb(srv->ssl_ctx.ctx, ssl_sess_new_srv_cb);
if (srv->ssl_ctx.ciphers &&
!SSL_CTX_set_cipher_list(srv->ssl_ctx.ctx, 
srv->ssl_ctx.ciphers)) {
Alert("Proxy '%s', server '%s' [%s:%d] : unable to set SSL 
cipher list to '%s'.\n",
@@ -5208,15 +5227,6 @@ reneg_ok:
update_freq_ctr(&global.ssl_be_keys_per_sec, 1);
if (global.ssl_be_keys_per_sec.curr_ctr > 
global.ssl_be_keys_max)
global.ssl_be_keys_max = 
global.ssl_be_keys_per_sec.curr_ctr;
-
-   /* check if session was reused, if not store current 
session on server for reuse */
-

Re: [PATCH] : BUG/MINOR

2017-11-03 Thread Willy Tarreau

On Fri, Nov 03, 2017 at 02:59:52PM +, David CARLIER wrote:
> Sure makes sense in this case. Cheers.

nice, thanks!
Willy

Re: log-format in defaults section in 1.7

2017-11-03 Thread Thayne McCombs

Thank you! That's good to know.

On Thu, Nov 2, 2017 at 5:36 PM Cyril Bonté  wrote:

> Hi Thayne,
>
> Le 02/11/2017 à 23:08, Thayne McCombs a écrit :
> > So, I looked into using `no log` in non http frontends. But that isn't
> > sufficient.
> >
> > For example, if I have:
> >
> > global
> >log-tag "test"
> >log localhost:514 len 65535 local2 info info
> >
> > defaults
> >mode http
> >timeout connect 100
> >timeout server 3
> >timeout client 3
> >log-format "%Tq"
> >
> > listen mine
> >log global
> >bind :80
> >server localhost localhost:8080
> >
> > listen health_url
> >bind :27000
> >mode health
> >option httpchk
> >no log
> >
> >
> > I still get [ALERT] 305/160229 (21975) : Parsing [test.cfg:10]: failed
> > to parse log-format : format variable 'Tq' is reserved for HTTP mode.
>
> You can specify several "defaults" sections in your configuration : one
> for http, and one for tcp frontends.
>
> global
>log-tag "test"
>log localhost:514 len 65535 local2 info info
>
> defaults
>mode http
>timeout connect 100
>timeout server 3
>timeout client 3
>log-format "%Tq"
>
> listen mine
>log global
>bind :8080
>server localhost localhost:80
>
> # ...
> # Other HTTP frontends
> # ...
>
> defaults
>mode tcp
>timeout connect 100
>timeout server 3
>timeout client 3
>
> listen health_url
>bind :27000
>mode health
>option httpchk
>
> # ...
> # Other TCP frontends
> # ...
>
>
> > However, if I add `log-format "GARBAGE"` to the health_url listener,
> > then the error goes away.
>
> Or you can specify "option tcplog" in your "health_url" section (or any
> other tcp sections).
>
>
> --
> Cyril Bonté
>
-- 
*Thayne McCombs*
*Senior Software Engineer*
Lucid Software, Inc.

RE: HTTP DELETE command failing

2017-11-03 Thread Norman Branitsky

This particular DELETE was designed to return a 204 – no payload expected.
So the test which insisted on payload was incorrect.
Problem solved.
Thanks.

From: Igor Cicimov [mailto:ig...@encompasscorporation.com]
Sent: November-02-17 8:56 PM
To: Norman Branitsky 
Cc: Moemen MHEDHBI ; haproxy@formilux.org
Subject: Re: HTTP DELETE command failing

On Fri, Nov 3, 2017 at 11:29 AM, Norman Branitsky 
mailto:norman.branit...@micropact.com>> wrote:
I have this included in the configuration:

# Filter nasty input

acl missing_cl hdr_cnt(Content-length) eq 0

acl METH_PUT method PUT

acl METH_GET method GET HEAD

acl METH_PATCH method PATCH

##acl METH_DELETE method DELETE

http-request deny if HTTP_URL_STAR !METH_OPTIONS || METH_POST missing_cl || 
METH_PUT missing_cl || METH_PATCH missing_cl

|| METH_DELETE missing_cl

http-request deny if METH_GET HTTP_CONTENT

http-request deny unless METH_GET or METH_POST or METH_OPTIONS or 
METH_PATCH or METH_DELETE or METH_PUT

My colleague commented out the METH_DELETE acl.
It appears that in HAProxy 1.7 a number of acls are predefined
and we could delete the METH_PUT, METH_GET, and METH_PATCH acls also.
So is one of the http-request deny statements causing the problem?

Maybe check the DELETE RFC https://tools.ietf.org/html/rfc7231#section-4.3.5

and think about what to do with your conditions. Start by removing "|| 
METH_DELETE missing_cl"
 from the first one.

Re: Throughput issue after moving between kernels.

2017-11-03 Thread Aaron West

I think I understand that with faster networks giving shorter RTT you
need less buffer space and then as either RTT or throughput
increases(Maybe 40G+) then you will need more, am I right?

So maybe it was changed to take into account modern internet links,
however, that doesn't explain the observed throughput issue as yet...
I wonder what else might have changed.

Aaron West

Loadbalancer.org Ltd.

www.loadbalancer.org

+1 888 867 9504 / +44 (0)330 380 1064
aa...@loadbalancer.org

LEAVE A REVIEW | DEPLOYMENT GUIDES | BLOG

Re: Error 'NAME_MAX' undeclared in HAProxy 1.8 on Solaris 11.3 (64-bit)

2017-11-03 Thread Willy Tarreau

Hi Daniel,

On Fri, Nov 03, 2017 at 04:00:08PM +0100, Daniel Heitepriem wrote:
> Hi everyone,
> 
> I tried to compile the recent HAProxy 1.8 (pulled from the git
> repository) on Solaris 11.3 (x86) with these settings:
> gmake TARGET=solaris CPU=generic USE_TPROXY=1 USE_ZLIB=1 USE_OPENSSL=1
> USE_PCRE=1 USE_GETADDRINFO=1 USE_REGPARM=1 DEFINE="-D_XOPEN_SOURCE=600"
> 
> "-D_XOPEN_SOURCE=600" is necessary on my Solaris installation as we're
> running GCC5 and the default Makefile ships with XOPEN_SOURCE=500
> 
> Despite a few warnings regarding SPIN_LOCK
> 
> :0:0: note: this is the location of the previous
> definition
> In file included from include/common/memory.h:30:0,
>  from include/common/chunk.h:29,
>  from include/common/standard.h:36,
>  from include/common/ticks.h:56,
>  from src/ev_poll.c:22:
> include/common/hathreads.h:75:0: warning: "SPIN_LOCK" redefined
>  #define SPIN_LOCK(lbl, l)    do { /* do nothing */ } while(0)
>  ^
> In file included from /usr/include/sys/t_lock.h:15:0,
>  from /usr/include/sys/stream.h:18,
>  from /usr/include/netinet/in.h:66,
>  from /usr/include/sys/socket.h:29,
>  from include/common/compat.h:29,
>  from src/ev_poll.c:20:
> /usr/include/sys/machlock.h:60:0: note: this is the location of the
> previous definition
>  #define SPIN_LOCK(pl) ((pl) > ipltospl(LOCK_LEVEL))

Ah nice one, I suspected it but was glad it didn't appear. At least
we didn't test on solaris :-) I'm CCing Emeric and Christopher. Guys,
I think that we'll have to rename all of these to HA_SPIN_LOCK() as
you did for the wrappers for atomic ops.

> I get an error which doesn't let me compile this version of HAProxy:
> 
> src/haproxy.c: In function 'get_old_sockets':
> src/haproxy.c:971:31: error: 'NAME_MAX' undeclared (first use in
> this function)
>   tmpbuf = malloc(fd_nb * (1 + NAME_MAX + 1 + IFNAMSIZ + sizeof(int)));
>    ^
> src/haproxy.c:971:31: note: each undeclared identifier is reported
> only once for each function it appears in
> gmake: *** [src/haproxy.o] Error 1

Oh too bad, we've already dealt with this a long time ago but now it
reappeared in new code. CCing Olivier. Olivier, in 2012 we already
faced this issue and addressed it in commit ee2663b ("BUILD: ssl:
NAME_MAX is not portable, use MAXPATHLEN instead") but resorting to
MAXPATHLEN.

> Is this a known issue so far?

It wasn't and that was exactly the purpose of -rc1 to collect such
precious feedback! For NAME_MAX I think it can easily be redefined
by building with :

make ... DEFINE="-DNAME_MAX=MAXPATHLEN"

But for the spinlocks you'll have to modify each occurrence in many
files, which will be a real pain, so we'll have to raise this change
as a high priority in order not to wait too long without being able
to test some OSes.

Thanks very much!
Willy

Re: Error 'NAME_MAX' undeclared in HAProxy 1.8 on Solaris 11.3 (64-bit)

2017-11-03 Thread Daniel Heitepriem


Hi Willy,

when including "DEFINE="-DNAME_MAX=MAXPATHLEN"" in the build statement, 
HAProxy compiles fine (leaving the warning aside).


   ./haproxy -vv
   HA-Proxy version 1.8-rc1 2017/10/31
   Copyright 2000-2017 Willy Tarreau 

   Build options :
  TARGET  = solaris
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing
   -Wdeclaration-after-statement -fwrapv -Wno-unused-label
   -fomit-frame-pointer -DFD_SETSIZE=65536 -D_REENTRANT
   -D_XOPEN_SOURCE=500 -D__EXTENSIONS__ -D_XOPEN_SOURCE=600
   -DNAME_MAX=MAXPATHLEN
  OPTIONS = USE_TPROXY=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1
   USE_OPENSSL=1 USE_PCRE=1

   Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents
   = 200

   Built with OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017
   Running on OpenSSL version : OpenSSL 1.0.2k  26 Jan 2017
   OpenSSL library supports TLS extensions : yes
   OpenSSL library supports SNI : yes
   OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
   Built with transparent proxy support using:
   Built with network namespace support.
   Built with zlib version : 1.2.11
   Running on zlib version : 1.2.11
   Compression algorithms supported : identity("identity"),
   deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
   Encrypted password support via crypt(3): yes
   Built with PCRE version : 8.39 2016-06-14
   Running on PCRE version : 8.39 2016-06-14
   PCRE library supports JIT : no (USE_PCRE_JIT not set)

   Available polling systems :
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
   Total: 2 (2 usable), will use poll.

   Available filters :
    [SPOE] spoe
    [COMP] compression
    [TRACE] trace


Have a nice weekend,
Daniel

Am 03.11.2017 um 23:08 schrieb Willy Tarreau:

Hi Daniel,

It wasn't and that was exactly the purpose of -rc1 to collect such
precious feedback! For NAME_MAX I think it can easily be redefined
by building with :

 make ... DEFINE="-DNAME_MAX=MAXPATHLEN"

But for the spinlocks you'll have to modify each occurrence in many
files, which will be a real pain, so we'll have to raise this change
as a high priority in order not to wait too long without being able
to test some OSes.

Thanks very much!
Willy

Re: Throughput issue after moving between kernels.

2017-11-03 Thread Willy Tarreau

Hi Aaron,

On Fri, Nov 03, 2017 at 07:23:20PM +, Aaron West wrote:
> I think I understand that with faster networks giving shorter RTT you
> need less buffer space and then as either RTT or throughput
> increases(Maybe 40G+) then you will need more, am I right?

no :-) It's the opposite, the BDP (bandwidth*delay product) increases
because the bandwidth increases much faster than the delay shrinks, thus
you have more (unacked) bytes in flight. In fact the vast majority of
the delay comes from cable length and switch ports these days. The
serialization time (impacted by the bit rate) is minimal. It's not on
short lengths due to the coding in use but it's still small.

Also on receipt you need to take the time to deliver to the application
into account. When you take one millisecond to wake up an application
to deliver Rx data because it was computing an RSA key for example, you
quickly realize that within this short millisecond you've received 5 MB
of data that has to be buffered somewhere... And a 1ms pause is very
short compared to what can be observed on an application server making
disk accesses.

> So maybe it was changed to take into account modern internet links,
> however, that doesn't explain the observed throughput issue as yet...
> I wonder what else might have changed.

Something else has changed recently, TCP pacing was deployed somewhere
but I don't remember what exact kernel, though I'm almost sure it was
after 3.10 (or at least more aggressive after 3.10). You should notice
a slightly lower bandwidth with a single stream but higher sustained
bandwidth with many streams (due to less losses on switch ports and
routers).

BTW you really need to hurry up switching your kernel, as I'm going to
emit the last 3.10.108 tomorrow and after that the 3.10.x branch is dead!

> Aaron West
> 
> Loadbalancer.org Ltd.
> 
> www.loadbalancer.org
> 
> +1 888 867 9504 / +44 (0)330 380 1064
> aa...@loadbalancer.org

Cheers,
Willy

Re: Error 'NAME_MAX' undeclared in HAProxy 1.8 on Solaris 11.3 (64-bit)

2017-11-03 Thread Willy Tarreau

On Fri, Nov 03, 2017 at 11:17:25PM +0100, Daniel Heitepriem wrote:
> Hi Willy,
> 
> when including "DEFINE="-DNAME_MAX=MAXPATHLEN"" in the build statement,
> HAProxy compiles fine (leaving the warning aside).

Cool, thank you for the report.

willy

Re: [PATCH] send-proxy-v2-ssl-crypto parameter

2017-11-03 Thread Willy Tarreau

Hi Manu,

On Thu, Nov 02, 2017 at 02:57:10PM +0100, Emmanuel Hocdet wrote:
> 
> Hi Willy,
> 
> This patches implement send-proxy-v2-ssl-crypto to add CIPHER
> SIG_ALG and KEY_ALG to send-proxy-v2-ssl as describe in proxy-protocol.txt

I'm delaying this a bit, we first need to stabilize rc1. New stuff
should be queued for 1.9 now, though we can possibly reconsider it
during the cleanup phase at the end if it's very low risk/impact.

Thanks,
Willy

Re: [ANNOUNCE] haproxy-1.8-rc1 : the last mile

2017-11-03 Thread Willy Tarreau

Hi Robert,

On Thu, Nov 02, 2017 at 03:58:47PM +, Robert Samuel Newson wrote:
> Hi,
> 
> I think the "cert bundle" feature from 1.7 is broken in 1.8-rc1. My exact 
> config works with 1.7 but says this for 1.8-rc1;
> 
> unable to stat SSL certificate from file '/path/to/foo.pem': No such file or 
> directory.
> 
> That is, it's attempting to load foo.pem, not foo.pem.rsa or foo.pem.ecdsa 
> like 1.7 does.

Oh bad, that's totally unexpected. I'm CCing Emeric and Manu, the former
being the SSL maintainer (in case he has a quick idea about it) and the
latter having upgraded a large part of the cert management code, possibly
having an idea about anything that could have gone wrong.

> I also tried asking the mailing list daemon for help by emailing
> haproxy+h...@formilux.org as the signup confirmation specifies, the full body
> of that help is just "Hello,". I was hoping to ask the daemon to send me the
> initial message in this thread so I could reply into the thread properly.
> Sadly the mailing list archive doesn't show any of the headers I might have
> injected to get threading working that way, so sorry for breaking the thread
> but I really tried not to.

I was not even aware of the feature :-)

> I am very excited about many of the new features in 1.8 and am itching to try
> them.

As long as you're very careful that's useful. I'm going to issue an rc2 with
the most painful bugs fixed.

Thanks for the report,
Willy

[ANNOUNCE] haproxy-1.8-rc2

2017-11-03 Thread Willy Tarreau

Hi,

1.8-rc1 had quite a few painful issues that we managed to sort out, so
in order to facilitate testing by most users, here comes 1.8-rc2. Here's
an overview of the main changes :
  - fixed the deadlock that prevented us from using checks and threads at
the same time
  - fixed the bug with the lock on the freq counters causing a CPU loop
when retrieving some table entries over peers.
  - fixed an ugly memory corruption on H2 due to a bug on a buffer
manipulation function
  - fixed a use-after-free bug in the passthrough mux causing random
crashes
  - fixed another loop when the cache is used over H2
  - and fixed another cache issue related to filters (I don't
remember which one, I'm sorry).
  - a bunch of missing locks in the DNS code, checks and Lua
were addressed during a code review.

There are still some issues we're working on but they require a bit
less efforts to work around :
 
 - the cache currently must only used in a frontend and with logs
   enabled (to inherit txn->uri), but it's currently being addressed.
   Since it's not a big deal, we preferred to make it testable anyway.
   Please don't use cache rules in the backend and frontend at the
   same time for now, the behaviour will likely be undefined.

 - I've been running this version on www.haproxy.org (explaining the
   short outages a few of you might have noticed while trying to spot
   the passthrough mux bug). And yes, haproxy.org was delivered in H2
   this afternoon :-) But I faced an infinite loop once and unfortunately
   the core I managed to trigger was unexploitable so I'm trying again
   without success for now (I'll stop it for the night here, as I want
   to be in front when it happens to debug it but not to prevent anyone
   from downloading or accessing the site). Thus, please don't leave it
   unattended for too long if you don't have an automatic switch-over,
   even if you think it looks pretty at first glance.

 - we know that there are some build issues on Solaris which can be
   worked around, though I think threads will not work due to naming
   conflicts between macros.

Threads are expected to be usable now. I'm going to re-enable them on
haproxy.org as soon as I spot the bug causing the rare loop (don't want
to mix the two for now).

Feel free to test. Coredumps (and their associated executable) are
welcome, but only if built with -g (the default). And gdb backtraces
(bt full) are very helpful as well, especially if you attach to a
stuck or looping process.

Please find the usual URLs below :
   Site index   : http://www.haproxy.org/
   Discourse: http://discourse.haproxy.org/
   Sources  : http://www.haproxy.org/download/1.8/src/
   Git repository   : http://git.haproxy.org/git/haproxy-1.8.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy-1.8.git
   Changelog: http://www.haproxy.org/download/1.8/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Willy
---
Complete changelog :
David Carlier (2):
  BUG/MINOR: stdarg.h inclusion
  BUILD: enable USE_THREAD for OpenBSD build.

Emeric Brun (4):
  BUG/MAJOR: fix deadlock on healthchecks.
  BUG/MINOR: freq: fix infinite loop on freq_ctr_period.
  BUG/MINOR: dns: fix missing lock protection on server.
  BUG/MINOR: lua: fix missing lock protection on server.

Emmanuel Hocdet (2):
  BUG/MINOR: send-proxy-v2: fix dest_len in make_tlv call
  BUG/MINOR: send-proxy-v2: string size must include ('\0')

Olivier Houchard (3):
  MINOR: mux: Only define pipe functions on linux.
  MINOR: cache: Remove useless test for nonzero.
  MINOR: cache: Don't confuse act_return and act_parse_ret.

William Lallemand (2):
  BUILD: shctx: allow to be built without openssl
  BUG/MEDIUM: cache: don't try to resolve wrong filters

Willy Tarreau (6):
  BUG/MEDIUM: h2: don't try to parse incomplete H1 responses
  BUG/MEDIUM: checks/mux: always enable send-polling after connecting
  BUG/MINOR: thread: fix a typo in the debug code
  BUG/MAJOR: buffers: fix get_buffer_nc() for data at end of buffer
  BUG/MAJOR: mux_pt: don't dereference a connstream after ->wake()
  MINOR: thread: report multi-thread support in haproxy -vv

---

Re: [ANNOUNCE] haproxy-1.8-rc2

2017-11-03 Thread Aleksandar Lazic


The new version is on docker hub

https://hub.docker.com/r/me2digital/haproxy18/

###
HA-Proxy version 1.8-rc2-a8d8d6e 2017/11/03
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET = linux2628
  CPU = generic
  CC = gcc
  CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement 
-fwrapv -Wno-unused-label
  OPTIONS = USE_LINUX_SPLICE=1 USE_GETADDRINFO=1 USE_ZLIB=1 
USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1 
USE_TFO=1


Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 
200


Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND

Built with network namespace support.
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Encrypted password support via crypt(3): yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : yes
Built with multi-threading support.

Available polling systems :
  epoll : pref=300, test result OK
   poll : pref=200, test result OK
 select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
 [SPOE] spoe
 [COMP] compression
 [TRACE] trace
###

regards
Aleks

-- Originalnachricht --
Von: "Willy Tarreau" 
An: haproxy@formilux.org
Gesendet: 04.11.2017 00:15:49
Betreff: [ANNOUNCE] haproxy-1.8-rc2


Hi,

1.8-rc1 had quite a few painful issues that we managed to sort out, so
in order to facilitate testing by most users, here comes 1.8-rc2. 
Here's

an overview of the main changes :
 - fixed the deadlock that prevented us from using checks and threads 
at

   the same time
 - fixed the bug with the lock on the freq counters causing a CPU loop
   when retrieving some table entries over peers.
 - fixed an ugly memory corruption on H2 due to a bug on a buffer
   manipulation function
 - fixed a use-after-free bug in the passthrough mux causing random
   crashes
 - fixed another loop when the cache is used over H2
 - and fixed another cache issue related to filters (I don't
   remember which one, I'm sorry).
 - a bunch of missing locks in the DNS code, checks and Lua
   were addressed during a code review.

There are still some issues we're working on but they require a bit
less efforts to work around :

- the cache currently must only used in a frontend and with logs
  enabled (to inherit txn->uri), but it's currently being addressed.
  Since it's not a big deal, we preferred to make it testable anyway.
  Please don't use cache rules in the backend and frontend at the
  same time for now, the behaviour will likely be undefined.

- I've been running this version on www.haproxy.org (explaining the
  short outages a few of you might have noticed while trying to spot
  the passthrough mux bug). And yes, haproxy.org was delivered in H2
  this afternoon :-) But I faced an infinite loop once and 
unfortunately

  the core I managed to trigger was unexploitable so I'm trying again
  without success for now (I'll stop it for the night here, as I want
  to be in front when it happens to debug it but not to prevent anyone
  from downloading or accessing the site). Thus, please don't leave it
  unattended for too long if you don't have an automatic switch-over,
  even if you think it looks pretty at first glance.

- we know that there are some build issues on Solaris which can be
  worked around, though I think threads will not work due to naming
  conflicts between macros.

Threads are expected to be usable now. I'm going to re-enable them on
haproxy.org as soon as I spot the bug causing the rare loop (don't want
to mix the two for now).

Feel free to test. Coredumps (and their associated executable) are
welcome, but only if built with -g (the default). And gdb backtraces
(bt full) are very helpful as well, especially if you attach to a
stuck or looping process.

Please find the usual URLs below :
  Site index   : http://www.haproxy.org/
  Discourse: http://discourse.haproxy.org/
  Sources  : http://www.haproxy.org/download/1.8/src/
  Git repository   : http://git.haproxy.org/git/haproxy-1.8.git/
  Git Web browsing : http://git.haproxy.org/?p=haproxy-1.8.git
  Changelog: http://www.haproxy.org/download/1.8/src/CHANGELOG
  Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Willy
---
Complete changelog :
David Carlier (2):
 BUG/MINOR: stdarg.h inclusion
 BUILD: enable USE_THREAD for OpenBSD build.

Emeric Brun (4):
 BUG/MAJOR: fix deadlock on healthchecks.
 BUG/MINOR: freq: fix infinite loop on freq_ctr_period.
 BUG/MI

Re: [ANNOUNCE] haproxy-1.8-rc2

2017-11-03 Thread Willy Tarreau

On Fri, Nov 03, 2017 at 11:54:09PM +, Aleksandar Lazic wrote:
> The new version is on docker hub

Thanks Aleks!

By the way, I've added this line to report whether threads are
enabled or not, it will help during bug reports :

> haproxy -vv
> (...)
> Built with PCRE version : 8.32 2012-11-30
> Running on PCRE version : 8.32 2012-11-30
> PCRE library supports JIT : yes
> Built with multi-threading support.
  ^^^

(it may happen anywhere like other lines here as it depends on the
link order).

Willy

Logging errors during reload of haproxy

Re: Logging errors during reload of haproxy

[PATCH]: BUG/MINOR

Re: [ANNOUNCE] haproxy-1.8-rc1 : the last mile

Quick question re errorloc urls

Throughput issue after moving between kernels.

Re: Logging errors during reload of haproxy

Re: Logging errors during reload of haproxy

Re: [PATCH]: BUG/MINOR

Re: Logging errors during reload of haproxy

Re: [PATCH]: BUG/MINOR

Re: [ANNOUNCE] haproxy-1.8-rc1 : the last mile

Re: [PATCH]: BUG/MINOR

[PATCH] : BUG/MINOR

Re: [PATCH] : BUG/MINOR

Re: Quick question re errorloc urls

Error ‘NAME_MAX’ undeclared in HAProxy 1.8 on Solaris 11.3 (64-bit)

Re: [PATCH] : BUG/MINOR

Re: Throughput issue after moving between kernels.

[PATCHES] TLS 1.3 session resumption and early data to servers

Re: [PATCH] : BUG/MINOR

Re: log-format in defaults section in 1.7

RE: HTTP DELETE command failing

Re: Throughput issue after moving between kernels.

Re: Error 'NAME_MAX' undeclared in HAProxy 1.8 on Solaris 11.3 (64-bit)

Re: Error 'NAME_MAX' undeclared in HAProxy 1.8 on Solaris 11.3 (64-bit)

Re: Throughput issue after moving between kernels.

Re: Error 'NAME_MAX' undeclared in HAProxy 1.8 on Solaris 11.3 (64-bit)

Re: [PATCH] send-proxy-v2-ssl-crypto parameter

Re: [ANNOUNCE] haproxy-1.8-rc1 : the last mile

[ANNOUNCE] haproxy-1.8-rc2

Re: [ANNOUNCE] haproxy-1.8-rc2

Re: [ANNOUNCE] haproxy-1.8-rc2

33 matches

Site Navigation

Mail list logo

Footer information