Re: 3 HA proxy instances on 3 ECS clusters hang on and off

2017-08-03 Thread Cyril Bonté 

Hi,

Le 03/08/2017 à 19:17, Ransika Desilva a écrit :

Hello,

Thanks for a wonderful product. We have an issue as off now, hoping that 
you will be able to help us.


We are having 3 clusters (dev/staging/prod) based on AWS ECS and we 
deploy the HA Proxy as docker containers on them. Each cluster has 1 
instance of the HA Proxy running.


We have noticed that even during low volume, all the 3 clusters getting 
hang. The instances are running but traffic is not forwarded. A simple 
restart works. We have added aws resolvers to handle the LB IP address 
changes.


The issue is some what similar to 
https://discourse.haproxy.org/t/haproxy-crashes-on-3-nodes-at-exactly-the-same-time/1039, 
but we are not using FreeBSD.


I have attached the HA Proxy config for your kind reference.

Thanks and looking forward to hear from you soon.


The RTF file is a pain, please send the configuration as plain text in 
your mail next time ;-)


The issue is on your server lines :
  server foo-a fs-sim-alb-external-XX.com:39997 resolvers awsvpc
You have configured a resolver but you didn't enable health checks. This 
is (currently) mandatory to name updates :

http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#resolvers%20(Server%20and%20default-server%20options)


--
Cyril Bonté

Re: HAProxy and Exchange 2016 MAPI/RPC over HTTP

2017-08-03 Thread shouldbe q931 
On Thu, Aug 3, 2017 at 9:01 AM, Philipp Zeitschel  wrote:
> Hi,
>
>
>
> i have haproxy 1.7.8 @Ubuntu 16.04 up and running.
>
> Outlook Webaccess and the Administration Panel are working fine but I can’t
> get Outlook to work, it repeatly asks for credentials (it is working if I
> try it directly without the loadbalancer).
>
> Microsofts Connectivity Analyzer tells:
>
> Testing HTTP Authentication Methods for URL
> https://xxx/rpc/rpcproxy.dll?xxx:6002.
>
>The HTTP authentication methods are correct.
>
>
>
> Additional Details
>
>
>
> The Microsoft Connectivity Analyzer found all expected authentication
> methods and no disallowed methods. Methods found: Basic, Negotiate, NTLM
>
> HTTP Response Headers:
>
> request-id: b57cf3ce-4d29-4a15-9246-7527db63bea1
>
> Server: Microsoft-IIS/8.5
>
> WWW-Authenticate: Negotiate,NTLM,Basic realm="xxx"
>
> Date: Thu, 03 Aug 2017 07:57:54 GMT
>
> Content-Length: 0
>
> Elapsed Time: 1502 ms.
>
>
>
>
>
> Attempting to ping RPC proxy xxx.
>
>RPC Proxy can't be pinged.
>
>
>
> Additional Details
>
>
>
> An unexpected network-level exception was encountered.
>
>
>
> This is the log output of haproxy:
>
> Aug  3 09:50:51 localhost haproxy[1880]: 13.67.59.89:14546
> [03/Aug/2017:09:50:50.774] ft_exch~ oa/exch02 377/0/9/4/390 401 269 - - 
> 1/1/0/1/0 0/0 {xxx|MSRPC} {0} {TLSv1.2/ECDHE-RSA-AES256-SHA384/xxx/-}
> RPC_IN_DATA xxx/rpc/rpcproxy.dll HTTP/1.1
>
> Aug  3 09:50:51 localhost haproxy[1880]: 13.67.59.89:14547
> [03/Aug/2017:09:50:51.519] ft_exch~ oa/exch02 176/0/7/5/188 401 269 - - 
> 2/2/0/1/0 0/0 {xxx|MSRPC} {0} {TLSv1/ECDHE-RSA-AES256-SHA/xxx/-} RPC_IN_DATA
> xxx/rpc/rpcproxy.dll?xxx:6002 HTTP/1.1
>
> Aug  3 09:50:51 localhost haproxy[1880]: 13.67.59.89:14547
> [03/Aug/2017:09:50:51.708] ft_exch~ oa/exch02 175/0/0/4/180 401 269 - - 
> 2/2/0/1/0 0/0 {xxx|MSRPC} {0} {TLSv1/ECDHE-RSA-AES256-SHA/xxx/-} RPC_IN_DATA
> xxx/Rpc/RpcProxy.dll?xxx:6001 HTTP/1.1
>
> Aug  3 09:50:52 localhost haproxy[1880]: 13.67.59.89:14549
> [03/Aug/2017:09:50:52.239] ft_exch~ oa/exch02 182/0/7/4/193 401 582 - - 
> 3/3/0/1/0 0/0 {xxx|MSRPC} {0} {TLSv1/ECDHE-RSA-AES256-SHA/xxx/-} RPC_IN_DATA
> xxx/Rpc/RpcProxy.dll?xxx:6001 HTTP/1.1
>
> Aug  3 09:50:52 localhost haproxy[1880]: 13.67.59.89:14549
> [03/Aug/2017:09:50:52.433] ft_exch~ oa/exch02 177/0/0/169/346 404 282 - -
>  3/3/0/1/0 0/0 {xxx|MSRPC} {0} {TLSv1/ECDHE-RSA-AES256-SHA/xxx/-}
> RPC_IN_DATA xxx/Rpc/RpcProxy.dll?xxx:6001 HTTP/1.1
>
>
>
> Firewall is deaktivated
>
>
>
> And this is my configuration:
>
> global
>
> log 127.0.0.1 local0 debug
>
> log /var/lib/haproxy/dev/loglocal0 debug
>
> log /var/lib/haproxy/dev/loglocal1 notice
>
>
>
> chroot /var/lib/haproxy
>
> stats socket /run/haproxy/admin.sock mode 660 level admin
>
> stats timeout 30s
>
> user haproxy
>
> group haproxy
>
> daemon
>
> ssl-server-verify none
>
> # Default SSL material locations
>
> #ca-base /etc/ssl/certs
>
> #crt-base /etc/ssl/private
>
> crt-base /etc/ssl/ca/certs
>
> ca-base /etc/ssl/ca/intermediate/certs
>
>
>
>
>
> # Default ciphers to use on SSL-enabled listening sockets.
>
> # For more information, see ciphers(1SSL). This list is from:
>
> #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
>
> ssl-default-bind-ciphers
> ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
>
> ssl-default-bind-options no-sslv3
>
> tune.ssl.default-dh-param 2048
>
>
>
> defaults
>
> log global
>
> modehttp
>
> option  httplog
>
> option  dontlognull
>
> option http-keep-alive
>
> option prefer-last-server
>
> option forwardfor
>
> option http-server-close
>
> no option httpclose
>
> no option forceclose
>
> no option http-tunnel
>
> balance leastconn
>
> default-server inter 3s rise 2 fall 3
>
> timeout client 600s
>
> timeout http-request 10s
>
> timeout connect 4s
>
> timeout server 60s
>
> errorfile 400 /etc/haproxy/errors/400.http
>
> errorfile 403 /etc/haproxy/errors/403.http
>
> errorfile 408 /etc/haproxy/errors/408.http
>
> errorfile 500 /etc/haproxy/errors/500.http
>
> errorfile 502 /etc/haproxy/errors/502.http
>
> errorfile 503 /etc/haproxy/errors/503.http
>
> errorfile 504 /etc/haproxy/errors/504.http
>
>
>
> frontend ft_exch
>
> bind 0.0.0.0:443 name https ssl crt /etc/ssl/z/bundle.pem no-sslv3
>
> capture request header Host len 32
>
> capture request header User-Agent len 64
>
> capture response header Content-Length len 10
>
> log /var/lib/haproxy/dev/log local4 debug
>
> log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\
> %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\
> {%sslv/%sslc/%[ssl_fc_sni]/%[ss
> l_fc_session_id]}\

3 HA proxy instances on 3 ECS clusters hang on and off

2017-08-03 Thread Ransika Desilva 
Hello,

Thanks for a wonderful product. We have an issue as off now, hoping that
you will be able to help us.

We are having 3 clusters (dev/staging/prod) based on AWS ECS and we deploy
the HA Proxy as docker containers on them. Each cluster has 1 instance of
the HA Proxy running.

We have noticed that even during low volume, all the 3 clusters getting
hang. The instances are running but traffic is not forwarded. A simple
restart works. We have added aws resolvers to handle the LB IP address
changes.

The issue is some what similar to
https://discourse.haproxy.org/t/haproxy-crashes-on-3-nodes-at-exactly-the-same-time/1039,
but we are not using FreeBSD.

I have attached the HA Proxy config for your kind reference.

Thanks and looking forward to hear from you soon.

Regards,
Ransika


ha_proxy.rtf
Description: RTF file

Re: server ports - server state file

2017-08-03 Thread William Lallemand 
On Thu, Aug 03, 2017 at 02:32:55PM +0200, Willy Tarreau wrote:
> On Tue, Aug 01, 2017 at 09:00:20AM +0200, Frederic Lecaille wrote:
> > Hello Haproxy ML,
> > 
> > Here is a simple patch to add server ports to server state file.
> 
> I missed this one, but it's now merged. thanks Fred!
> Willy
> 

Thanks Fred, that will be really useful!

-- 
William Lallemand

Re: server ports - server state file

2017-08-03 Thread Willy Tarreau 
On Tue, Aug 01, 2017 at 09:00:20AM +0200, Frederic Lecaille wrote:
> Hello Haproxy ML,
> 
> Here is a simple patch to add server ports to server state file.

I missed this one, but it's now merged. thanks Fred!
Willy

Re: PIDs are not dying on a reload

2017-08-03 Thread Arnaud B. 
2 production days, using Vincent's haproxy.debian.net build
(1.7.8-1~bpo9+1), I'm not having the issue on 4 out of 5 load balancers.

There is still one though. I'll see if it evolves over time.


Le 01/08/2017 à 12:14, Vincent Bernat a écrit :
>  ❦  1 août 2017 12:00 +0200, "Arnaud B."  :
>
>> I'm not using peers, it's a feature that I've discovered as you
>> mentionned it, seems worth the try.
>>
>> I'll upgrade to the latest bpo package from Vincent's repository and see
>> if it fixes my issue, I'll get back to you if it succeeds or not.
> You can use the "official" backport from Debian repositories. You just
> have to know it could be updated to 1.8 at some point (while the
> packages served directly by haproxy.debian.net won't).

Re: Fix building haproxy with recent LibreSSL

2017-08-03 Thread Willy Tarreau 
Hi Bernard,

I'm CCing Emeric since this affects SSL. I have some comments below.

On Tue, Jul 25, 2017 at 05:03:10PM +0200, Bernard Spil wrote:
> On 2017-07-04 10:18, Willy Tarreau wrote:
> > On Tue, Jul 04, 2017 at 11:12:20AM +0300, Dmitry Sivachenko wrote:
> > > >> https://www.mail-archive.com/haproxy@formilux.org/msg25819.html
> > > >
> > > >
> > > > Do you know if the patch applies to 1.8 (it was mangled so I didn't 
> > > > try).
> > > 
> > > 
> > > Sorry, hit reply too fast:  no, one chunk fails against 1.8-dev2
> > > (the one
> > > dealing with #ifdef SSL_CTX_get_tlsext_status_arg, it requires
> > > analysis
> > > because it is not simple surrounding context change).
> > 
> > OK thanks. Bernard, care to have a look and ensure it works for you ?
> > 
> > Thanks,
> > Willy
> 
> I've just committed a patch to FreeBSD's ports tree for haproxy-devel
> (1.8-dev2). This would be a good candidate to include.
> 
> Not sure if attachments work for the mailing-list...
> https://svnweb.freebsd.org/ports/head/net/haproxy-devel/files/patch-src_ssl__sock.c

The attachment works, though we only apply git patches (with proper
commit messages explaining the nature of the change and its impact,
please check CONTRIBUTING for this). However for the discussion, your
patch is fine.

> --- src/ssl_sock.c.orig   2017-06-02 13:59:51 UTC
> +++ src/ssl_sock.c
> @@ -56,7 +56,7 @@
>  #include 
>  #endif
>  
> -#if OPENSSL_VERSION_NUMBER >= 0x101fL
> +#if (OPENSSL_VERSION_NUMBER >= 0x101fL) && 
> !defined(LIBRESSL_VERSION_NUMBER)
>  #include 
>  #endif
(...)

OK so all the ones related to LibreSSL not supporting async should go
into one patch.

> @@ -1034,10 +1034,13 @@ static int ssl_sock_load_ocsp(SSL_CTX *c
>   ocsp = NULL;
>  
>  #ifndef SSL_CTX_get_tlsext_status_cb
> -# define SSL_CTX_get_tlsext_status_cb(ctx, cb) \
> - *cb = (void (*) (void))ctx->tlsext_status_cb;
> +#ifndef SSL_CTRL_GET_TLSEXT_STATUS_REQ_CB
> +#define SSL_CTRL_GET_TLSEXT_STATUS_REQ_CB 128
>  #endif
> + callback = SSL_CTX_ctrl(ctx, SSL_CTRL_GET_TLSEXT_STATUS_REQ_CB, 0, 
> callback);
> +#else
>   SSL_CTX_get_tlsext_status_cb(ctx, );
> +#endif
>  
>   if (!callback) {
>   struct ocsp_cbk_arg *cb_arg = calloc(1, sizeof(*cb_arg));

Here I'm having a problem with this change, but maybe Emeric will tell
me I'm wrong. From what I'm seeing, by hard-coding the callback number,
it will simply be ignored by older openssl versions (look at SSL_CTX_ctrl()
in ssl/ssl_lib.c). So it will indeed build but break them. In my opinion
we should avoid doing this and instead proceed like this :

  #ifdef SSL_CTX_get_tlsext_status_cb
/* what versions are covered here */
SSL_CTX_get_tlsext_status_cb(ctx, );
  #elif defined(SSL_CTRL_GET_TLSEXT_STATUS_REQ_CB)
/* what versions are covered here */
callback = SSL_CTX_ctrl(ctx, SSL_CTRL_GET_TLSEXT_STATUS_REQ_CB, 0, 
callback);
  #else
/* what versions are covered here */
callback = (void (*) (void))ctx->tlsext_status_cb;
  #endif

It presents the benefit of *not* changing what currently works, and of
allowing us to get rid of some of these parts in a future release when
we drop support for older openssl versions.

> @@ -1063,7 +1066,10 @@ static int ssl_sock_load_ocsp(SSL_CTX *c
>   int key_type;
>   EVP_PKEY *pkey;
>  
> -#ifdef SSL_CTX_get_tlsext_status_arg
> +#if defined(SSL_CTX_get_tlsext_status_arg) || (LIBRESSL_VERSION_NUMBER >= 
> 0x2050100fL)
> +#ifndef SSL_CTRL_GET_TLSEXT_STATUS_REQ_CB_ARG
> +#define SSL_CTRL_GET_TLSEXT_STATUS_REQ_CB_ARG 129
> +#endif
>   SSL_CTX_ctrl(ctx, SSL_CTRL_GET_TLSEXT_STATUS_REQ_CB_ARG, 0, 
> _arg);
>  #else

Here are you sure that libressl on this version supports this specific
arg value even if it's not defined ? It's very possible but I'm a bit
worried about the fact that recently, for the sake of supporting some
of the openssl derivatives, we've started to hide the dust under the
carpet by adding missing defines.

>   cb_arg = ctx->tlsext_status_arg;
> @@ -3403,7 +3409,7 @@ int ssl_sock_load_cert_list_file(char *f
>  #define SSL_MODE_SMALL_BUFFERS 0
>  #endif
>  
> -#if (OPENSSL_VERSION_NUMBER < 0x101fL) && !defined(OPENSSL_IS_BORINGSSL)
> +#if (OPENSSL_VERSION_NUMBER < 0x101fL) && !defined(OPENSSL_IS_BORINGSSL) 
> || defined(LIBRESSL_VERSION_NUMBER)
>  static void ssl_set_SSLv3_func(SSL_CTX *ctx, int is_server)
>  {
>  #if SSL_OP_NO_SSLv3

This one seems more related to the drop of SSLv3 support on this lib,
please use a separate patch for it.

I think everything here is reasonable. Regarding the one you have for
1.7 in your tree, I don't know if this is valid or not :

-  empty_handshake = !((SSL *)conn->xprt_ctx)->packet_length;
+  empty_handshake = SSL_state((SSL *)conn->xprt_ctx) == 
SSL_ST_BEFORE;

Emeric tends to spend a lot of time picking the appropriate openssl functions
so he might have had a good reason for

HAProxy and Exchange 2016 MAPI/RPC over HTTP

2017-08-03 Thread Philipp Zeitschel 
Hi,

i have haproxy 1.7.8 @Ubuntu 16.04 up and running.
Outlook Webaccess and the Administration Panel are working fine but I can't get 
Outlook to work, it repeatly asks for credentials (it is working if I try it 
directly without the loadbalancer).
Microsofts Connectivity Analyzer tells:
Testing HTTP Authentication Methods for URL 
https://xxx/rpc/rpcproxy.dll?xxx:6002.
   The HTTP authentication methods are correct.

Additional Details

The Microsoft Connectivity Analyzer found all expected authentication methods 
and no disallowed methods. Methods found: Basic, Negotiate, NTLM
HTTP Response Headers:
request-id: b57cf3ce-4d29-4a15-9246-7527db63bea1
Server: Microsoft-IIS/8.5
WWW-Authenticate: Negotiate,NTLM,Basic realm="xxx"
Date: Thu, 03 Aug 2017 07:57:54 GMT
Content-Length: 0
Elapsed Time: 1502 ms.


Attempting to ping RPC proxy xxx.
   RPC Proxy can't be pinged.

Additional Details

An unexpected network-level exception was encountered.

This is the log output of haproxy:
Aug  3 09:50:51 localhost haproxy[1880]: 13.67.59.89:14546 
[03/Aug/2017:09:50:50.774] ft_exch~ oa/exch02 377/0/9/4/390 401 269 - -  
1/1/0/1/0 0/0 {xxx|MSRPC} {0} {TLSv1.2/ECDHE-RSA-AES256-SHA384/xxx/-} 
RPC_IN_DATA xxx/rpc/rpcproxy.dll HTTP/1.1
Aug  3 09:50:51 localhost haproxy[1880]: 13.67.59.89:14547 
[03/Aug/2017:09:50:51.519] ft_exch~ oa/exch02 176/0/7/5/188 401 269 - -  
2/2/0/1/0 0/0 {xxx|MSRPC} {0} {TLSv1/ECDHE-RSA-AES256-SHA/xxx/-} RPC_IN_DATA 
xxx/rpc/rpcproxy.dll?xxx:6002 HTTP/1.1
Aug  3 09:50:51 localhost haproxy[1880]: 13.67.59.89:14547 
[03/Aug/2017:09:50:51.708] ft_exch~ oa/exch02 175/0/0/4/180 401 269 - -  
2/2/0/1/0 0/0 {xxx|MSRPC} {0} {TLSv1/ECDHE-RSA-AES256-SHA/xxx/-} RPC_IN_DATA 
xxx/Rpc/RpcProxy.dll?xxx:6001 HTTP/1.1
Aug  3 09:50:52 localhost haproxy[1880]: 13.67.59.89:14549 
[03/Aug/2017:09:50:52.239] ft_exch~ oa/exch02 182/0/7/4/193 401 582 - -  
3/3/0/1/0 0/0 {xxx|MSRPC} {0} {TLSv1/ECDHE-RSA-AES256-SHA/xxx/-} RPC_IN_DATA 
xxx/Rpc/RpcProxy.dll?xxx:6001 HTTP/1.1
Aug  3 09:50:52 localhost haproxy[1880]: 13.67.59.89:14549 
[03/Aug/2017:09:50:52.433] ft_exch~ oa/exch02 177/0/0/169/346 404 282 - -  
3/3/0/1/0 0/0 {xxx|MSRPC} {0} {TLSv1/ECDHE-RSA-AES256-SHA/xxx/-} RPC_IN_DATA 
xxx/Rpc/RpcProxy.dll?xxx:6001 HTTP/1.1

Firewall is deaktivated

And this is my configuration:
global
log 127.0.0.1 local0 debug
log /var/lib/haproxy/dev/loglocal0 debug
log /var/lib/haproxy/dev/loglocal1 notice

chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
ssl-server-verify none
# Default SSL material locations
#ca-base /etc/ssl/certs
#crt-base /etc/ssl/private
crt-base /etc/ssl/ca/certs
ca-base /etc/ssl/ca/intermediate/certs


# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers 
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
tune.ssl.default-dh-param 2048

defaults
log global
modehttp
option  httplog
option  dontlognull
option http-keep-alive
option prefer-last-server
option forwardfor
option http-server-close
no option httpclose
no option forceclose
no option http-tunnel
balance leastconn
default-server inter 3s rise 2 fall 3
timeout client 600s
timeout http-request 10s
timeout connect 4s
timeout server 60s
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend ft_exch
bind 0.0.0.0:443 name https ssl crt /etc/ssl/z/bundle.pem no-sslv3
capture request header Host len 32
capture request header User-Agent len 64
capture response header Content-Length len 10
log /var/lib/haproxy/dev/log local4 debug
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ 
%CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ 
{%sslv/%sslc/%[ssl_fc_sni]/%[ss 
l_fc_session_id]}\ 
"%[capture.req.method]\ %[capture.req.hdr(0)]%[capture.req.uri]\ HTTP/1.1"
option http-keep-alive
option socket-stats
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth admin:xxx

maxconn 1000
acl ssl_connection ssl_fc
acl host_mail hdr(Host) -i xxx
acl path_slash path /
acl path_autodiscover path_beg -i

Re: maxconn without queue?

2017-08-03 Thread Claudio Kuenzler 
Hi Lukas


> However I doubt that this is a proper solution. Any other ideas?
>
> The proper solution depends on the problem you are trying to solve.
>
> What is the problem you are trying to solve?
>


Basically the following:

1) Each backend server only allows a maximum of 6 concurrent sessions.
2) Higher number (7+) of concurrent sessions/connections are to be a)
balanced to other backend servers or b) refused and send a HTTP error back
to the client
3) Any new request can be balanced to whatever available backend server.
4) Once a HTTP request landed on a backend, all follow-up requests (this is
a keep-alive connection) must go to the same backend again
5) If backend is down and a request is supposed to land on it, rather send
a HTTP error back to the client than to failover to another (available)
backend

The points 1 and 2 were solved with the combination "maxqueue 1" and
"timeout queue" as written before.
For 4 and 5 I was still scratching my head because some of the requests
were landing on the "wrong" backend server when (for reason X) a backend
server was down.

But you already gave me the solution:


You probably want "option persist" [1] instead?
>


I didn't see that option before. I just tested it, it works as I need it in
that setup.

Thanks a lot!

cheers,
ck