Re: Sharing SSL information via PROXY protocol or HAProxy internally

[Dennis Jacobfeuerborn]

On 15.04.2016 16:01, Christian Ruppert wrote:
> Hi,
> 
> would it be possible to inherit the SSL information from a SSL
> listener/frontend via PROXY protocol?
> So for example:
> 
> listen ssl-relay
> mode tcp
> 
> ...
> 
> server rsa unix@/var/run/haproxy_ssl_rsa.sock send-proxy-v2
> 
> listen ssl-rsa_ecc
> mode tcp
> 
> ...
> 
> bind unix@/var/run/haproxy_ssl_rsa.sock accept-proxy ssl crt
> SSl-RSA.PEM user haproxy
> 
> frontend http_https
> bind :80 # http
> bind unix@/var/run/haproxy_ssl.sock accept-proxy user haproxy # https
> 
> redirect scheme https code 301 if !{ssl_fc}
> 
> 
> Here the ssl_fc and other SSL related ACLs do not work because the
> actual SSL termination has been done in the above ssl-rsa_ecc listener.
> Sharing that either internally or via the PROXY protocol would be really
> handy, if that's possible.
> For now we use the bind "id" to check whether it's the proxy connection
> or not but the above would be much easier/better IMHO.

For this specific case of http to https redirect I use the
X-Forwarded-Proto header. In the ssl frontend I do this:

http-request set-header X-Forwarded-Proto https

and in the plain http frontend I do this:

http-request redirect scheme https if !{ req.hdr(X-Forwarded-Proto) https }

You usually need to set this header anyway so the application knows it
needs to generate https URLs in the generated HTML.

Regards,
  Dennis

Re: 100% cpu , epoll_wait()

[Jim Freeman]

Did a bit more digging on the most recent instance, and found that the
haproxy pid doing the hogging was handling a connection to the stats
port :

listen haproxy_stats :5
stats enable
stats uri /
no log

, with this 'netstat -pantlu' entry :
tcp0  99756 10.34.176.98:5  10.255.247.189:54484
ESTABLISHED 9499/haproxy

I'm suspecting that a connection to the stats port goes wonky with a
'-sf' reload, but I'll have to wait for it to re-appear to poke
further.  I'll look first for a stats port connection handled by the
pegged process, then use 'tcpkill' to kill just that connection
(rather than the whole process, which may be handling other
connections).

Its been happening 2 to 3 times a week, and I now have alerting around
the event - I'll post more info as I get it ...


On Fri, Apr 15, 2016 at 4:28 PM, Cyril Bonté  wrote:
> Hi Jim,
>
> Le 15/04/2016 23:20, Jim Freeman a écrit :
>>
>> I have haproxy slaved to 2d cpu (CPU1), with frequent config changes
>> and a '-sf' soft-stop with the now-old non-listening process nannying
>> old connections.
>>
>> Sometimes CPU1 goes to %100, and then a few minutes later request
>> latencies suffer across multiple haproxy peers.
>>
>> An strace of the nanny haproxy process shows a tight loop of :
>>
>> epoll_wait(0, {}, 200, 0)   = 0
>> epoll_wait(0, {}, 200, 0)   = 0
>> epoll_wait(0, {}, 200, 0)   = 0
>>
>> I've searched the archives and found similar but old-ish complaints
>> about similar circumstances, but with fixes/patches mentioned.
>>
>> This has happened with both 1.5.3 and 1.5.17.
>>
>> Insights ?
>
>
> Can you provide your configuration (without sensible data) ?
> Are you using peers ?
>
> Also, do you have a reproductible testcase that we can play with, or is it
> absolutely random ?
>
>
>
>>
>> ===
>>
>> # cat  /proc/version
>> Linux version 3.16.0-0.bpo.4-amd64 (debian-ker...@lists.debian.org)
>> (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian
>> 3.16.7-ckt25-1~bpo70+1 (2016-04-02)
>>
>> # haproxy -vv
>> HA-Proxy version 1.5.17 2016/04/13
>> Copyright 2000-2016 Willy Tarreau 
>>
>> Build options :
>>TARGET  = linux2628
>>CPU = generic
>>CC  = gcc
>>CFLAGS  = -g -O2 -fstack-protector --param=ssp-buffer-size=4
>> -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2
>>OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1
>>
>> Default settings :
>>maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
>>
>> Encrypted password support via crypt(3): yes
>> Built with zlib version : 1.2.7
>> Compression algorithms supported : identity, deflate, gzip
>> Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
>> Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
>> OpenSSL library supports TLS extensions : yes
>> OpenSSL library supports SNI : yes
>> OpenSSL library supports prefer-server-ciphers : yes
>> Built with PCRE version : 8.30 2012-02-04
>> PCRE library supports JIT : no (USE_PCRE_JIT not set)
>> Built with transparent proxy support using: IP_TRANSPARENT
>> IPV6_TRANSPARENT IP_FREEBIND
>>
>> Available polling systems :
>>epoll : pref=300,  test result OK
>> poll : pref=200,  test result OK
>>   select : pref=150,  test result OK
>> Total: 3 (3 usable), will use epoll.
>>
>
>
> --
> Cyril Bonté

Re: 100% cpu , epoll_wait()

[Cyril Bonté]

Hi Jim,

Le 15/04/2016 23:20, Jim Freeman a écrit :

I have haproxy slaved to 2d cpu (CPU1), with frequent config changes
and a '-sf' soft-stop with the now-old non-listening process nannying
old connections.

Sometimes CPU1 goes to %100, and then a few minutes later request
latencies suffer across multiple haproxy peers.

An strace of the nanny haproxy process shows a tight loop of :

epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 0)   = 0

I've searched the archives and found similar but old-ish complaints
about similar circumstances, but with fixes/patches mentioned.

This has happened with both 1.5.3 and 1.5.17.

Insights ?


Can you provide your configuration (without sensible data) ?
Are you using peers ?

Also, do you have a reproductible testcase that we can play with, or is 
it absolutely random ?





===

# cat  /proc/version
Linux version 3.16.0-0.bpo.4-amd64 (debian-ker...@lists.debian.org)
(gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian
3.16.7-ckt25-1~bpo70+1 (2016-04-02)

# haproxy -vv
HA-Proxy version 1.5.17 2016/04/13
Copyright 2000-2016 Willy Tarreau 

Build options :
   TARGET  = linux2628
   CPU = generic
   CC  = gcc
   CFLAGS  = -g -O2 -fstack-protector --param=ssp-buffer-size=4
-Wformat -Werror=format-security -D_FORTIFY_SOURCE=2
   OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
   maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.30 2012-02-04
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
   epoll : pref=300,  test result OK
poll : pref=200,  test result OK
  select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.




--
Cyril Bonté

Coding style for coonfig files

[Michael Rennecke]

Hello,

I know this question is stupid. Is there a coding style for config
files, like this: http://www.haproxy.org/coding-style.html ?

Cheers,
Michael




signature.asc
Description: OpenPGP digital signature

100% cpu , epoll_wait()

[Jim Freeman]

I have haproxy slaved to 2d cpu (CPU1), with frequent config changes
and a '-sf' soft-stop with the now-old non-listening process nannying
old connections.

Sometimes CPU1 goes to %100, and then a few minutes later request
latencies suffer across multiple haproxy peers.

An strace of the nanny haproxy process shows a tight loop of :

epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 0)   = 0

I've searched the archives and found similar but old-ish complaints
about similar circumstances, but with fixes/patches mentioned.

This has happened with both 1.5.3 and 1.5.17.

Insights ?

===

# cat  /proc/version
Linux version 3.16.0-0.bpo.4-amd64 (debian-ker...@lists.debian.org)
(gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian
3.16.7-ckt25-1~bpo70+1 (2016-04-02)

# haproxy -vv
HA-Proxy version 1.5.17 2016/04/13
Copyright 2000-2016 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fstack-protector --param=ssp-buffer-size=4
-Wformat -Werror=format-security -D_FORTIFY_SOURCE=2
  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.30 2012-02-04
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Re: use of variables in ACL

[David Birdsong]

Oh, great thanks!

On Thu, Apr 14, 2016 at 11:50 PM Willy Tarreau  wrote:

> Hi David,
>
> On Fri, Apr 15, 2016 at 03:35:11AM +, David Birdsong wrote:
> > Is there anyway to use:
> >
> > http-request set-var()
> >
> > to set a var for later use in forming an ACL?
> >
> > I've tried all the prefixes to make the variable survive past http
> > processing, but the ACL is always rejected by the config check.
> >
> > 
> > http-request set-var(txn.my_v) base,sdbm,mod(100)
> > acl test_threshold txn.my_v lt 10
> > 
>
> The "var()" sample fetch should be use for this :
>
>  acl test_threshold var(txn.my_v) lt 10
>
> regards,
> willy
>
>

TTL-based DNS resolution ?

[Ben Tisdall]

Hi,

are there are plans to support DNS resolution based on TTL a la NGINX? This
would be helpful for use cases where the upstream is an ELB or similar
system. I've pasted a reply from AWS support based on some observations
from a couple of our services that use HAProxy 1.6 in front of ELBs. Note
that I am not contending that the issue of uneven distribution of upstream
IPs is HAProxy's fault (that is a consequence of our design), but the
cycling of ELB nodes when retirement occurs is something that NGINX would
seem to handle in a more satisfactory way.

"
 I think an explanation of what happens when ELB scales will be helpful as
background at this point. ELB employs what we term "Graceful Scaling". When
a scaling trigger is breached, let's say for sake of argument this is a
scale-up event, then ELBs controller immediately begins the process of
provisioning new more performant ELB nodes. This usually takes a few
minutes, and once these new nodes pass the controller health-checks, we
remove the old node IPs from the DNS record set, and add in the new ELB
node IPs to the DNS set. Since the TTL with this DNS record is published in
60 seconds, after about 2 - 3 minutes, most traffic will migrate over to
the new node. We do however, do not de-provision the old ELB nodes, but
instead we begin to monitor them to determine when traffic received by
these nodes drops to below a certain threshold level, or a maximum age has
expired (this is several days). This happens to cater for the case where
some clients are caching DNS longer than the TTL value.

Given the way that HA proxy works, when it starts up, it resolves the ELB
name, and obtains the current IPs. HealthChecks are a requirement for the
resolver clause, so HA also begins to perform the configured health-check
on the nodes it learned about at startup.

If the ELB were to scale now, the new nodes would come online but HA proxy
would never learn of them, as the old nodes will continue to pass
health-checks. If traffic continues to increase, at some point the older
ELB nodes will become overwhelmed and will fail a health-check on HA proxy,
at which point that HA proxy node on which the health-check failed, will
learn of the new ELB nodes from DNS, and start to send traffic to the new
one.

Should traffic not increase sufficiently to cause the old nodes to fail a
health-check, then only new HA proxy instances in your fleet will learn of
the new nodes. Eventually the maximum graceful node lifetime will be
reached, and we will terminate the old nodes, at this point all your HA
proxy instances will fail health-checks on their upstream and learn of the
new nodes at the same time.

This process happens in such a fashion that over a time, its conceivable
that each of your HA proxy nodes may know of different back-end IPs. As a
result, traffic on the inside ELB nodes will not be symmetrically
distributed by the HA proxy nodes over time. This is somewhat mitigated on
the back-end by the use of cross-zone load-balancing, so the asymmetry is
not propagated to the back-ends. We do monitor each ELB node individually,
thus the ELB will scale on the monitoring of a single node, rather than the
entire ELB, which further mitigates the effects of any asymmetry on your
ELB nodes.

There is no easy way to make HA proxy work perfectly in front of an ELB,
due to the nature of how HA proxy have implemented DNS resolution.

We often recommend to customers using a reverse proxy in front of ELB, to
rather use Nginx, as this does have the ability to follow DNS TTLs of its
upstreams perfectly. In this case, given the way you have implemented it
means that HA will learn of failed ELB nodes, and eventually learn of
scaling, and the ELB mitigates the imbalance to your back-ends through
cross zone. So, in summary, as of now the only possible way to overcome
this behavior would be to consider using a different reverse proxy solution
between the two ELB tiers instead of HA proxy. I apologize for any
inconvenience. I hope the above information was helpful. Please let us know
if you have any other questions or concerns and we will be happy to assist
you.
"

Regards,

-- 
Ben

-- 




*This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure.If you are not the intended recipient 
you must not copy this message or attachment or disclose the contents to 
any other person.For more information please contact:  
**groupsecur...@photobox.com 
*

*PhotoBox Group, Unit 7, Metal Box Factory, 30 Great Guildford Street, 
London, SE1 0HS*

*http://group.photobox.com *

Re: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection

[David Martin]

On Apr 15, 2016 4:24 AM, "Janusz Dziemidowicz" 
wrote:
>
> 2016-04-14 17:39 GMT+02:00 David Martin :
> > Here's a revised patch, it throws a fatal config error if
> > SSL_CTX_set1_curves_list() fails.  The default echde option is used so
> > current configurations should not be impacted.
> >
> > Sorry Janusz, forgot the list on my reply.
>
> I believe that now it is wrong as SSL_CTX_set_ecdh_auto works
> differently than this code implies.
> From what I was able to tell from OpenSSL code (always a pleasure
> looking at) it works as follows:
> - SSL_CTX_set_ecdh_auto turns on negotiation of curves, without this
> no curves will be negotiated (and only one configured curve will be
> used, "the old way")
> - the list of curves that are considered during negotiation contain
> all of the OpenSSL supported curves
> - unless you also call SSL_CTX_set1_curves_list() and narrow it down
> to the list you prefer
>
> Right now you patch either calls SSL_CTX_set_ecdh_auto or
> SSL_CTX_set1_curves_list, but not both. Unless I'm mistaken, this
> kinda is not how it is supposed to be used.
> Have you tested behavior of the server with any command line client?

I have tested the current patch with the HAProxy default, a list of curves,
a single curve and also an incorrect curve.  All seem to behave correctly.
The conditional should only skip calling ecdh_auto() if curves_list()
returns 0 in which case HAProxy exits anyway.

Maybe I'm missing something obvious, this has been a learning experience
for me.

>
> I believe this should be something like:
> #if new OpenSSL
>SSL_CTX_set_ecdh_auto(... 1)
>SSL_CTX_set1_curves_list() with user supplied ecdhe or
> ECDHE_DEFAULT_CURVE by default
> #elif ...
>SSL_CTX_set_tmp_ecdh() with user supplied ecdhe or
> ECDHE_DEFAULT_CURVE by default
> #endif
>
> This way haproxy behaves exactly the same with default configuration
> and any version of OpenSSL. User can configure multiple curves if
> there is sufficiently new OpenSSL.
>
> Changes to the documentation would also be nice in the patch :)
>
> --
> Janusz Dziemidowicz

Just to be clear I have no intention of running anything other than
prime256v1 on my systems nor do I think anyone else should. I had x25519 in
mind when looking over the code.

Perhaps something like this should wait until haproxy is ready for openssl
1.1.0 as it has little value without an alternative curve.

Love the project by the way, thanks for all the awesome work.

Sharing SSL information via PROXY protocol or HAProxy internally

[Christian Ruppert]

Hi,

would it be possible to inherit the SSL information from a SSL 
listener/frontend via PROXY protocol?

So for example:

listen ssl-relay
mode tcp

...

server rsa unix@/var/run/haproxy_ssl_rsa.sock send-proxy-v2

listen ssl-rsa_ecc
mode tcp

...

bind unix@/var/run/haproxy_ssl_rsa.sock accept-proxy ssl crt 
SSl-RSA.PEM user haproxy


frontend http_https
bind :80 # http
bind unix@/var/run/haproxy_ssl.sock accept-proxy user haproxy # 
https


redirect scheme https code 301 if !{ssl_fc}


Here the ssl_fc and other SSL related ACLs do not work because the 
actual SSL termination has been done in the above ssl-rsa_ecc listener. 
Sharing that either internally or via the PROXY protocol would be really 
handy, if that's possible.
For now we use the bind "id" to check whether it's the proxy connection 
or not but the above would be much easier/better IMHO.


--
Regards,
Christian Ruppert

Re: Q: about HTTP/2

[Steven Le Roux]

Hi,

I have a concret use case that need http/2 : one ssl frontend with
multiple heterogeneous (h2,http/1.1) backends

gRPC is http2 only and if you have to use it behind a single domain
you need to be able to route http/2 traffic among http/1.1.

In this case, I need to apply some http specific rules (e.g. path_beg)
that require mode http to be enabled.
In the same time, I need to route http2 traffic to the right backend.

The problem is that the http2 backend is not mode http compliant so
there is a conflict between backends given the routing issue I have.

What I did to do this is having :

frontend f_https
  mode tcp
  bind ... alpn h2,http/1.1
  ...
  acl is_h2 ssl_fc_alpn -i h2
  use_backend b_grpc if h2
  use_backend b_http if !h2

frontend f_http
  mode http
  bind 127.0.0.1:80
  acl p_1 path_beg /1
  acl p_2 path_beg /2
  use_backend b_1 if p_1
  use_backend b_2 if p_2

backend b_http
   server internal 127.0.0.1:80

backend b_1
  mode http
  server api_1 10.0.0.1:8081
  server api_2 10.0.0.2:8081
backend b_2
  mode http
  server ws_1  10.0.0.1:8082
  server ws_2 10.0.0.2:8082
backend b_grpc
  mode tcp
  server grpc_1  10.0.0.1:8082
  server grpc_2  10.0.0.2:8082


Having HPACK parsing with mode http over h2 would simplify this by
just using HTTP ACLs and ALPN detection in the same frontend.

On Sat, Apr 9, 2016 at 11:30 AM, Pavlos Parissis
 wrote:
> On 07/04/2016 06:01 μμ, Willy Tarreau wrote:
>> Hi Aleks,
>>
>> On Fri, Apr 01, 2016 at 12:18:54PM +0200, Aleksandar Lazic wrote:
>>> Hi Willy & other core devs/pms.
>>>
>>> I know that HTTP/2 is on the road-map but not ready yet.
>>>
>>> Would you be so kind and share some of your thoughts, stats and plans for
>>> HTTP/2.
>>
>> Well, the plan is to have *at least* HTTP/2 with the client and HTTP/1 with
>> the server. Maybe we'll find that doing H2->H2 is easy enough but given our
>> past experiences guessing that two sides are only slightly more difficult
>> than a single one, I'd prefer to remain careful.
>>
>> Regarding the timing, I'm trying hard to get something for 1.7 next fall.
>> But to be very honnest, 90% of my time spent on haproxy is spent chasing
>> bugs these days, the last 10% are spent on code review.
>
> And this is very much appreciated by a lot of people(including my self).
> It increases the confidence level on the software when you know that
> devs are actively working on properly fixing bugs and avoid putting
> yet another monkey patch, so they can go back to development of new
> cool things.
>
> Having said, I would love to see H2 on client side and a _more_ proper
> connection pooling on the server side in 1.7 version.
>
> Cheers,
> Pavlos
>
>



-- 
Steven Le Roux
Jabber-ID : ste...@jabber.fr
0x39494CCB 
2FF7 226B 552E 4709 03F0  6281 72D7 A010 3949 4CCB

Re: nbproc 1 vs >1 performance

[Pavlos Parissis]

On 15/04/2016 11:44 πμ, Willy Tarreau wrote:
> Hi Christian,
> 
> On Fri, Apr 15, 2016 at 11:26:18AM +0200, Christian Ruppert wrote:
>> Just in case someone is interested in this setup:
>> Don't put the two SSL binds into the frontend. Add a second listener for the
>> two SSL binds and from there via send-proxy-v2 to the frontend.
>> Why? Because having >1 processes in the frontend in combination with a
>> backend with active checks is a bad idea. If you have for example 10
>> loadbalancer with 40 process each you'll basically DDoS yourself.
> 
> Definitely. That's why the recommended setup is indeed to have two layers,
> one for SSL and one for L7, possibly connected over unix or abstract sockets :
> 
>   listen ssl-offload
>   bind-process 2-40
>   bind :443 ssl crt foo ... process 2
>   ...
>   bind :443 ssl crt foo ... process 40
>   server clear abns@clear send-proxy-v2
> 
>   frontend clear
>   bind-process 1
>   bind :80 ... # clear traffic
>   bind abns@clear accept-proxy
>   ...
>   default_backend application...
> 
>   etc.
> 

But in case of HTTPs on the backend side, you can't use the above as you
need CPU power on the backend.


Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: nbproc 1 vs >1 performance

[Willy Tarreau]

On Fri, Apr 15, 2016 at 11:56:20AM +0200, Pavlos Parissis wrote:
> nbproc > 1 introduces also 2 other __issues__ which has been discussed
> several times in this ML:
> 
> 1) the aggregation of statistics
> 2) performing operations such as disable/enable/change weight.
> In setups where the frontends/backends is unevenly distributed across
> all process, this becomes more apparent as you have to track which
> process manages which frontend.

All of them are properly solved by having the backend in a single process.
Even peers work fine this way :-) You just need to consider the front SSL
processes as dumb off-loaders.

Cheers,
Willy

Re: nbproc 1 vs >1 performance

[Pavlos Parissis]

On 15/04/2016 11:26 πμ, Christian Ruppert wrote:
> On 2016-04-14 11:06, Christian Ruppert wrote:
>> Hi Willy,
>>
>> On 2016-04-14 10:17, Willy Tarreau wrote:
>>> On Thu, Apr 14, 2016 at 08:55:47AM +0200, Lukas Tribus wrote:
 Le me put it this way:

 frontend haproxy_test
  bind-process 1-8
  bind :12345 process 1
  bind :12345 process 2
  bind :12345 process 3
  bind :12345 process 4


 Leads to 8 processes, and the master process binds the socket 4
 times (PID
 16509):

>>> (...)
 lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap
 tcp0  0 *:12345 *:* LISTEN 
 16509/haproxy
 tcp0  0 *:12345 *:* LISTEN 
 16509/haproxy
 tcp0  0 *:12345 *:* LISTEN 
 16509/haproxy
 tcp0  0 *:12345 *:* LISTEN 
 16509/haproxy
 lukas@ubuntuvm:~/haproxy-1.5$
>>>
>>> OK so it's netstat which gives a wrong report, I have the same here.
>>> I verified
>>> in /proc/$PID/fd/ and I properly saw the FDs. Next, "ss -anp" also
>>> shows all the
>>> process list :
>>>
>>>   LISTEN 0  128   *:12345
>>>   *:*
>>> users:(("haproxy",25360,7),("haproxy",25359,7),("haproxy",25358,7),("haproxy",25357,7),("haproxy",25356,7),("haproxy",25355,7),("haproxy",25354,7),("haproxy",25353,7))
>>>
>>>   LISTEN 0  128   *:12345
>>>   *:*
>>> users:(("haproxy",25360,6),("haproxy",25359,6),("haproxy",25358,6),("haproxy",25357,6),("haproxy",25356,6),("haproxy",25355,6),("haproxy",25354,6),("haproxy",25353,6))
>>>
>>>   LISTEN 0  128   *:12345
>>>   *:*
>>> users:(("haproxy",25360,5),("haproxy",25359,5),("haproxy",25358,5),("haproxy",25357,5),("haproxy",25356,5),("haproxy",25355,5),("haproxy",25354,5),("haproxy",25353,5))
>>>
>>>   LISTEN 0  128   *:12345
>>>   *:*
>>> users:(("haproxy",25360,4),("haproxy",25359,4),("haproxy",25358,4),("haproxy",25357,4),("haproxy",25356,4),("haproxy",25355,4),("haproxy",25354,4),("haproxy",25353,4))
>>>
>>>
>>> A performance test also shows a fair distribution of the load :
>>>
>>>   25353 willy 20   0 21872 4216 1668 S   26  0.1   0:04.54 haproxy
>>>   25374 willy 20   0  7456  1080 S   25  0.0   0:02.26
>>> injectl464
>>>   25376 willy 20   0  7456  1080 S   25  0.0   0:02.27
>>> injectl464
>>>   25377 willy 20   0  7456  1080 S   25  0.0   0:02.26
>>> injectl464
>>>   25375 willy 20   0  7456  1080 S   24  0.0   0:02.26
>>> injectl464
>>>   25354 willy 20   0 21872 4168 1620 R   22  0.1   0:04.51 haproxy
>>>   25356 willy 20   0 21872 4216 1668 R   22  0.1   0:04.21 haproxy
>>>   25355 willy 20   0 21872 4168 1620 S   21  0.1   0:04.38 haproxy
>>>
>>> However, as you can see these sockets are still bound to all
>>> processes and
>>> that's not a good idea in the multi-queue mode.
>>>
>>> I have added a few debug lines in enable_listener() like this :
>>>
>>> $ git diff
>>> diff --git a/src/listener.c b/src/listener.c
>>> index 5abeb80..59c51a1 100644
>>> --- a/src/listener.c
>>> +++ b/src/listener.c
>>> @@ -49,6 +49,7 @@ static struct bind_kw_list bind_keywords = {
>>>   */
>>>  void enable_listener(struct listener *listener)
>>>  {
>>> +   fddebug("%d: enabling fd %d\n", getpid(), listener->fd);
>>> if (listener->state == LI_LISTEN) {
>>> if ((global.mode & (MODE_DAEMON | MODE_SYSTEMD)) &&
>>> listener->bind_conf->bind_proc &&
>>> @@ -57,6 +58,7 @@ void enable_listener(struct listener *listener)
>>>  * want any fd event to reach it.
>>>  */
>>> fd_stop_recv(listener->fd);
>>> +   fddebug("%d: pausing fd %d\n", getpid(),
>>> listener->fd);
>>> listener->state = LI_PAUSED;
>>> }
>>> else if (listener->nbconn < listener->maxconn) {
>>>
>>> And we're seeing this upon startup for processes 25746..25755 :
>>>
>>> Thus as you can see that FDs are properly enabled and paused for the
>>> unavailable ones.
>>>
>>> willy@wtap:haproxy$ grep 4294967295 log | grep 25746
>>> 25746 write(4294967295, "25746: enabling fd 4\n", 21 
>>> 25746 write(4294967295, "25746: enabling fd 5\n", 21 
>>> 25746 write(4294967295, "25746: pausing fd 5\n", 20) = -1 EBADF (Bad
>>> file descriptor)
>>> 25746 write(4294967295, "25746: enabling fd 6\n", 21) = -1 EBADF (Bad
>>> file descriptor)
>>> 25746 write(4294967295, "25746: pausing fd 6\n", 20) = -1 EBADF (Bad
>>> file descriptor)
>>> 25746 write(4294967295, "25746: enabling fd 7\n", 21 
>>> 25746 write(4294967295, "25746: pausing fd 7\n", 20 
>>> willy@wtap:haproxy$ grep 4294967295 log | grep 25747
>>> 25747 write(4294967295, "25747: enabling fd 4\n", 21 
>>> 25747 write(4294967295, "25747: pausing fd 4\n", 20 
>>> 25747 write(

Re: nbproc 1 vs >1 performance

[Willy Tarreau]

Hi Christian,

On Fri, Apr 15, 2016 at 11:26:18AM +0200, Christian Ruppert wrote:
> Just in case someone is interested in this setup:
> Don't put the two SSL binds into the frontend. Add a second listener for the
> two SSL binds and from there via send-proxy-v2 to the frontend.
> Why? Because having >1 processes in the frontend in combination with a
> backend with active checks is a bad idea. If you have for example 10
> loadbalancer with 40 process each you'll basically DDoS yourself.

Definitely. That's why the recommended setup is indeed to have two layers,
one for SSL and one for L7, possibly connected over unix or abstract sockets :

  listen ssl-offload
  bind-process 2-40
  bind :443 ssl crt foo ... process 2
  ...
  bind :443 ssl crt foo ... process 40
  server clear abns@clear send-proxy-v2

  frontend clear
  bind-process 1
  bind :80 ... # clear traffic
  bind abns@clear accept-proxy
  ...
  default_backend application...

  etc.

> We'll have
> to wait for multi-threaded for the setup above or some kind of dedicated
> check process that shares the information with all other processes.

No, you really don't want to multiply your backends, it causes pain
for checks, for monitoring, for maxconn etc. Prefer to have a single
instance if that sustains the load.

Willy

Re: nbproc 1 vs >1 performance

[Christian Ruppert]

On 2016-04-14 11:06, Christian Ruppert wrote:

Hi Willy,

On 2016-04-14 10:17, Willy Tarreau wrote:

On Thu, Apr 14, 2016 at 08:55:47AM +0200, Lukas Tribus wrote:

Le me put it this way:

frontend haproxy_test
 bind-process 1-8
 bind :12345 process 1
 bind :12345 process 2
 bind :12345 process 3
 bind :12345 process 4


Leads to 8 processes, and the master process binds the socket 4 times 
(PID

16509):


(...)

lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap
tcp0  0 *:12345 *:* LISTEN  
16509/haproxy
tcp0  0 *:12345 *:* LISTEN  
16509/haproxy
tcp0  0 *:12345 *:* LISTEN  
16509/haproxy
tcp0  0 *:12345 *:* LISTEN  
16509/haproxy

lukas@ubuntuvm:~/haproxy-1.5$


OK so it's netstat which gives a wrong report, I have the same here. I 
verified
in /proc/$PID/fd/ and I properly saw the FDs. Next, "ss -anp" also 
shows all the

process list :

  LISTEN 0  128   *:12345
  *:*
users:(("haproxy",25360,7),("haproxy",25359,7),("haproxy",25358,7),("haproxy",25357,7),("haproxy",25356,7),("haproxy",25355,7),("haproxy",25354,7),("haproxy",25353,7))
  LISTEN 0  128   *:12345
  *:*
users:(("haproxy",25360,6),("haproxy",25359,6),("haproxy",25358,6),("haproxy",25357,6),("haproxy",25356,6),("haproxy",25355,6),("haproxy",25354,6),("haproxy",25353,6))
  LISTEN 0  128   *:12345
  *:*
users:(("haproxy",25360,5),("haproxy",25359,5),("haproxy",25358,5),("haproxy",25357,5),("haproxy",25356,5),("haproxy",25355,5),("haproxy",25354,5),("haproxy",25353,5))
  LISTEN 0  128   *:12345
  *:*
users:(("haproxy",25360,4),("haproxy",25359,4),("haproxy",25358,4),("haproxy",25357,4),("haproxy",25356,4),("haproxy",25355,4),("haproxy",25354,4),("haproxy",25353,4))

A performance test also shows a fair distribution of the load :

  25353 willy 20   0 21872 4216 1668 S   26  0.1   0:04.54 haproxy
  25374 willy 20   0  7456  1080 S   25  0.0   0:02.26 
injectl464
  25376 willy 20   0  7456  1080 S   25  0.0   0:02.27 
injectl464
  25377 willy 20   0  7456  1080 S   25  0.0   0:02.26 
injectl464
  25375 willy 20   0  7456  1080 S   24  0.0   0:02.26 
injectl464

  25354 willy 20   0 21872 4168 1620 R   22  0.1   0:04.51 haproxy
  25356 willy 20   0 21872 4216 1668 R   22  0.1   0:04.21 haproxy
  25355 willy 20   0 21872 4168 1620 S   21  0.1   0:04.38 haproxy

However, as you can see these sockets are still bound to all processes 
and

that's not a good idea in the multi-queue mode.

I have added a few debug lines in enable_listener() like this :

$ git diff
diff --git a/src/listener.c b/src/listener.c
index 5abeb80..59c51a1 100644
--- a/src/listener.c
+++ b/src/listener.c
@@ -49,6 +49,7 @@ static struct bind_kw_list bind_keywords = {
  */
 void enable_listener(struct listener *listener)
 {
+   fddebug("%d: enabling fd %d\n", getpid(), listener->fd);
if (listener->state == LI_LISTEN) {
if ((global.mode & (MODE_DAEMON | MODE_SYSTEMD)) &&
listener->bind_conf->bind_proc &&
@@ -57,6 +58,7 @@ void enable_listener(struct listener *listener)
 * want any fd event to reach it.
 */
fd_stop_recv(listener->fd);
+   fddebug("%d: pausing fd %d\n", getpid(), 
listener->fd);

listener->state = LI_PAUSED;
}
else if (listener->nbconn < listener->maxconn) {

And we're seeing this upon startup for processes 25746..25755 :

Thus as you can see that FDs are properly enabled and paused for the
unavailable ones.

willy@wtap:haproxy$ grep 4294967295 log | grep 25746
25746 write(4294967295, "25746: enabling fd 4\n", 21 
25746 write(4294967295, "25746: enabling fd 5\n", 21 
25746 write(4294967295, "25746: pausing fd 5\n", 20) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: enabling fd 6\n", 21) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: pausing fd 6\n", 20) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: enabling fd 7\n", 21 
25746 write(4294967295, "25746: pausing fd 7\n", 20 
willy@wtap:haproxy$ grep 4294967295 log | grep 25747
25747 write(4294967295, "25747: enabling fd 4\n", 21 
25747 write(4294967295, "25747: pausing fd 4\n", 20 
25747 write(4294967295, "25747: enabling fd 5\n", 21 
25747 write(4294967295, "25747: enabling fd 6\n", 21 
25747 write(4294967295, "25747: pausing fd 6\n", 20 
25747 write(4294967295, "25747: enabling fd 7\n", 21 
25747 write(4294967295, "25747: pausing fd 7\n", 20 
willy@wtap:haproxy$ grep 4294967295 log | grep 25748
25748 write(4294967295, "25748: enabling fd 4\n", 21 
25748 write(4294967295, "25748: pausing fd 4\n", 20 
25748 write(4294967295, "25748: enabling fd 5\n", 21 
25748 write(4294967295, "25748:

Re: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection

[Janusz Dziemidowicz]

2016-04-15 11:16 GMT+02:00 Pavlos Parissis :
> But on server side you need openssl 1.1.0[1] which is not ready yet and
> I think it requires changes on haproxy. Nginx has already some level of
> support[2] for openssl 1.1.0.

Sure, I didn't mean that it will work right now, but someday,
somewhere in the future;)

-- 
Janusz Dziemidowicz

Re: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection

[Janusz Dziemidowicz]

2016-04-14 17:39 GMT+02:00 David Martin :
> Here's a revised patch, it throws a fatal config error if
> SSL_CTX_set1_curves_list() fails.  The default echde option is used so
> current configurations should not be impacted.
>
> Sorry Janusz, forgot the list on my reply.

I believe that now it is wrong as SSL_CTX_set_ecdh_auto works
differently than this code implies.
>From what I was able to tell from OpenSSL code (always a pleasure
looking at) it works as follows:
- SSL_CTX_set_ecdh_auto turns on negotiation of curves, without this
no curves will be negotiated (and only one configured curve will be
used, "the old way")
- the list of curves that are considered during negotiation contain
all of the OpenSSL supported curves
- unless you also call SSL_CTX_set1_curves_list() and narrow it down
to the list you prefer

Right now you patch either calls SSL_CTX_set_ecdh_auto or
SSL_CTX_set1_curves_list, but not both. Unless I'm mistaken, this
kinda is not how it is supposed to be used.
Have you tested behavior of the server with any command line client?

I believe this should be something like:
#if new OpenSSL
   SSL_CTX_set_ecdh_auto(... 1)
   SSL_CTX_set1_curves_list() with user supplied ecdhe or
ECDHE_DEFAULT_CURVE by default
#elif ...
   SSL_CTX_set_tmp_ecdh() with user supplied ecdhe or
ECDHE_DEFAULT_CURVE by default
#endif

This way haproxy behaves exactly the same with default configuration
and any version of OpenSSL. User can configure multiple curves if
there is sufficiently new OpenSSL.

Changes to the documentation would also be nice in the patch :)

-- 
Janusz Dziemidowicz

Re: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection

[Pavlos Parissis]

On 15/04/2016 10:58 πμ, Janusz Dziemidowicz wrote:
> 2016-04-15 6:55 GMT+02:00 Willy Tarreau :
>>> Switching ECDHE curves can have performance impact, for example result
>>> of openssl speed on my laptop:
>>>  256 bit ecdh (nistp256)   0.0003s   2935.3
>>>  384 bit ecdh (nistp384)   0.0027s364.9
>>>  521 bit ecdh (nistp521)   0.0016s623.2
>>> The difference is so high for nistp256 because OpenSSL has heavily
>>> optimized implementation
>>> (https://www.imperialviolet.org/2010/12/04/ecc.html).
>>
>> Wow, and despite this you want to let the client force the server to
>> switch to 384 ? Looks like a hue DoS to me.
> 
> Just to be sure, I'm not the original author, I've just made some comments ;)
> Some people tend to use the strongest possible crypto, just for the
> sake of it. Usually on low traffic sites :)
> Anyway, Chrome 50 just pushes support for x25519. 

But on server side you need openssl 1.1.0[1] which is not ready yet and
I think it requires changes on haproxy. Nginx has already some level of
support[2] for openssl 1.1.0.

Cheers,
Pavlos

[1] https://github.com/openssl/openssl/issues/309
[2] https://trac.nginx.org/nginx/ticket/860



signature.asc
Description: OpenPGP digital signature

Re: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection

[Janusz Dziemidowicz]

2016-04-15 6:55 GMT+02:00 Willy Tarreau :
>> Switching ECDHE curves can have performance impact, for example result
>> of openssl speed on my laptop:
>>  256 bit ecdh (nistp256)   0.0003s   2935.3
>>  384 bit ecdh (nistp384)   0.0027s364.9
>>  521 bit ecdh (nistp521)   0.0016s623.2
>> The difference is so high for nistp256 because OpenSSL has heavily
>> optimized implementation
>> (https://www.imperialviolet.org/2010/12/04/ecc.html).
>
> Wow, and despite this you want to let the client force the server to
> switch to 384 ? Looks like a hue DoS to me.

Just to be sure, I'm not the original author, I've just made some comments ;)
Some people tend to use the strongest possible crypto, just for the
sake of it. Usually on low traffic sites :)
Anyway, Chrome 50 just pushes support for x25519. I believe this will
also have a very fast implementation, so ability to configure more
curves will probably be handy in near future.

-- 
Janusz Dziemidowicz