Re: TLS Tickets and CPU usage

[Willy Tarreau]

On Fri, Mar 25, 2016 at 03:43:31PM +0100, Nenad Merdanovic wrote:
> Hello Willy,
> 
> On 03/25/2016 03:29 PM, Nenad Merdanovic wrote:
> [..snip..]
> 
> Ah, just ignore this :) I've now realized what you meant.

:-)

> Sure, I'll
> rewrite the patch like that. To me it doesn't make much difference in
> readability and they do accomplish the same purpose, so we can do it as
> you prefer.

The difference for me is that in one case I have to open the C file
to check if "i" is signed or unsigned to estimate the range of the
resulting modulo while in the other case I know that it's only applied
to a positive value so it's positive by definition. In your case the
code was protected at the next line so it was OK (and confirmed by the
two testers), but once people pick a few lines there and reuse them
somewhere else it's a different story. Or if they are simply inspired
they might not get this important detail.

So it's not a matter of strict preference, just less ambiguity or
doubt when reading a patch or a small portion of code.

Also modulos are commonly used to ensure a number is between 0 and N-1
as often used in maths, but that's clearly not how computers use them,
instead they only take the reminder of the divide operation, and it's
common when reading such code to make the wrong assumption. How many of
us have seen apparently harmless but exploitable functions like this :

 char hex_digit(int v) {
 return hextab[v % 16];
 }

or even :

 char base_digit(int v, int base) {
 return base ? base36_tab[v % base] : 0;
 }

That's why I tend to be extra careful where there's a risk it could be
overlooked.

Cheers,
Willy

Re: TLS Tickets and CPU usage

[Nenad Merdanovic]

Hello Willy,

On 03/25/2016 03:29 PM, Nenad Merdanovic wrote:
[..snip..]

Ah, just ignore this :) I've now realized what you meant. Sure, I'll
rewrite the patch like that. To me it doesn't make much difference in
readability and they do accomplish the same purpose, so we can do it as
you prefer.

Regards,
Nenad

Re: TLS Tickets and CPU usage

[Nenad Merdanovic]

Hello Willy,

On 03/25/2016 01:37 PM, Willy Tarreau wrote:
> Hi Nenad,
> 
> On Fri, Mar 25, 2016 at 11:35:01AM +0100, Nenad Merdanovic wrote:
>> diff --git a/src/ssl_sock.c b/src/ssl_sock.c
>> index 1017388..767d6e9 100644
>> --- a/src/ssl_sock.c
>> +++ b/src/ssl_sock.c
>> @@ -5406,7 +5406,7 @@ static int bind_parse_tls_ticket_keys(char **args, int 
>> cur_arg, struct proxy *px
>>  fclose(f);
>>  
>>  /* Use penultimate key for encryption, handle when TLS_TICKETS_NO = 1 */
>> -i-=2;
>> +i = (i - 2) % TLS_TICKETS_NO;
>>  keys_ref->tls_ticket_enc_index = i < 0 ? 0 : i;
> 
> I'm still seeing an issue here which is that i is an integer so
> (i - 2) % TLS_TICKETS_NO will be negative for values of i between
> 0 and 1.
> 

Right, but this is covered with:
keys_ref->tls_ticket_enc_index = i < 0 ? 0 : i;

It can only happen in one case, if TLS_TICKET_NO = 1 (build time) and
there is only one key in the file (i = 1). There are explicit checks for
other cases:
#if (defined SSL_CTRL_SET_TLSEXT_TICKET_KEY_CB && TLS_TICKETS_NO > 0)

and

if (i < TLS_TICKETS_NO) {
if (err)
memprintf(err, "'%s' : please supply at least %d
keys in the tls-tickets-file", args[cur_arg+1], TLS_TICKETS_NO);
return ERR_ALERT | ERR_FATAL;
}

The ternary operator here covers the case where the result is negative
and sets the index to 0 (which is the only key we have).

> If this is intended, then maybe it would be better do fix it this way
> instead so that there's no ambiguity regarding the validity of the
> above operation :
> 
> - keys_ref->tls_ticket_enc_index = i < 0 ? 0 : i;
> + keys_ref->tls_ticket_enc_index = i < 0 ? 0 : i % TLS_TICKETS_NO;
> 

I am afraid this would do the unwanted, because i % TLS_TICKET_NO, as
you said, can be negative, and we'd have a negative index being used as
an array index:
head =
objt_listener(conn->target)->bind_conf->keys_ref->tls_ticket_enc_index;
memcpy(key_name, keys[head].name, 16);


> What do you think ?
> 
> Thanks,
> Willy
> 

I might've missed something, but I think the fix should be as-is, but
please do not commit it yet, I need to do some more testing.

Regards,
Nenad

Re: TLS Tickets and CPU usage

[Olivier Doucet]

2016-03-25 11:35 GMT+01:00 Nenad Merdanovic :

> Hey Olivier,
>
> Can you try the attached patch? I need to run some more tests, but I
> think this should fix it.
>

A summary of all tests performed :
WITHOUT PATCH:

With 0 ticket in file : HAProxy refuse to start " '/tmp/tls_ticket_keys' :
please supply at least 3 keys in the tls-tickets-file"
With 1 ticket in file : HAProxy refuses to start (same)
With 2 tickets in file : HAProxy refuses to start (same)
With 3 tickets in file : working as expected (reuse OK)
With 4 tickets in file : working as expected (reuse OK)
With 5 tickets in file : no more session reuse

WITH PATCH
"0001-BUG-MEDIUM-Fix-RFC5077-resumption-when-more-than-TLS.patch" :
With 0 ticket in file : HAProxy refuse to start " '/tmp/tls_ticket_keys' :
please supply at least 3 keys in the tls-tickets-file"
With 1 ticket in file : HAProxy refuses to start (same)
With 2 tickets in file : HAProxy refuses to start (same)
With 3 tickets in file : working as expected (reuse OK)
With 4 tickets in file : working as expected (reuse OK)
With 5 tickets in file : working as expected (reuse OK)
With 6 tickets in file : working as expected (reuse OK)


Many thanks to you all, and to Vincent Bernat for the tool rfc5077-client

Olivier




> Regards,
> Nenad
>
> On 3/24/2016 10:05 PM, Olivier Doucet wrote:
> > Hi again,
> >
> >
> > 2016-03-24 21:15 GMT+01:00 Lukas Tribus  > >:
> >
> > Hi Nenad,
> >
> >
> > >> Well, its not supposed to look like this, there is clearly
> something
> > >> wrong. Master key fluctuates between the requests with TLS tickets
> > >> and the reuse collumn shows failure.
> > >
> > > Looks like a haproxy bug, I think I can reproduce it.
> > >
> > > Can you try with EXACTLY 3 keys in /tmp/tls_ticket_keys?
> >
> >
> > Tried and now behaviour is like expected !
> > https://gist.github.com/anonymous/779fbc4f1cf8b23e9b1f
> >
> > And, I can confirm that now, CPU is not doubled \o/
> >
> >
> >
> >
> >
> > there seems to be a bug in the handling of the tls-ticket-keys file.
> >
> > When there are 5 or more ticket keys in the file, clients using TLS
> > tickets
> > can no longer resume the TLS session (and fallback to full
> negotiation):
> >
> > https://gist.github.com/anonymous/6ec7c863f497cfd849a4
> >
> >
> > Workaround would be to remove the oldest key from the file, so
> > that the number of keys in the file remains below 5.
> >
> > That's what I did : keep last 2 keys and add a new one.
> >
> >  Olivier
>

Re: TLS Tickets and CPU usage

[Willy Tarreau]

Hi Nenad,

On Fri, Mar 25, 2016 at 11:35:01AM +0100, Nenad Merdanovic wrote:
> diff --git a/src/ssl_sock.c b/src/ssl_sock.c
> index 1017388..767d6e9 100644
> --- a/src/ssl_sock.c
> +++ b/src/ssl_sock.c
> @@ -5406,7 +5406,7 @@ static int bind_parse_tls_ticket_keys(char **args, int 
> cur_arg, struct proxy *px
>   fclose(f);
>  
>   /* Use penultimate key for encryption, handle when TLS_TICKETS_NO = 1 */
> - i-=2;
> + i = (i - 2) % TLS_TICKETS_NO;
>   keys_ref->tls_ticket_enc_index = i < 0 ? 0 : i;

I'm still seeing an issue here which is that i is an integer so
(i - 2) % TLS_TICKETS_NO will be negative for values of i between
0 and 1.

If this is intended, then maybe it would be better do fix it this way
instead so that there's no ambiguity regarding the validity of the
above operation :

-   keys_ref->tls_ticket_enc_index = i < 0 ? 0 : i;
+   keys_ref->tls_ticket_enc_index = i < 0 ? 0 : i % TLS_TICKETS_NO;

What do you think ?

Thanks,
Willy

RE: TLS Tickets and CPU usage

[Lukas Tribus]

> Hey Olivier,
>
> Can you try the attached patch? I need to run some more tests, but I
> think this should fix it.

It definitely fixes the test case here.


thanks,

Lukas

  

Re: TLS Tickets and CPU usage

[Nenad Merdanovic]

Hey Olivier,

Can you try the attached patch? I need to run some more tests, but I
think this should fix it.

Regards,
Nenad

On 3/24/2016 10:05 PM, Olivier Doucet wrote:
> Hi again,
> 
> 
> 2016-03-24 21:15 GMT+01:00 Lukas Tribus  >:
> 
> Hi Nenad,
> 
> 
> >> Well, its not supposed to look like this, there is clearly something
> >> wrong. Master key fluctuates between the requests with TLS tickets
> >> and the reuse collumn shows failure.
> >
> > Looks like a haproxy bug, I think I can reproduce it.
> >
> > Can you try with EXACTLY 3 keys in /tmp/tls_ticket_keys?
> 
> 
> Tried and now behaviour is like expected !
> https://gist.github.com/anonymous/779fbc4f1cf8b23e9b1f
> 
> And, I can confirm that now, CPU is not doubled \o/
> 
> 
> 
>  
> 
> there seems to be a bug in the handling of the tls-ticket-keys file.
> 
> When there are 5 or more ticket keys in the file, clients using TLS
> tickets
> can no longer resume the TLS session (and fallback to full negotiation):
> 
> https://gist.github.com/anonymous/6ec7c863f497cfd849a4
> 
> 
> Workaround would be to remove the oldest key from the file, so
> that the number of keys in the file remains below 5.
> 
> That's what I did : keep last 2 keys and add a new one.
> 
>  Olivier
From 074792e61070c8fae16cd0b2fae4dfb6e6eeff7a Mon Sep 17 00:00:00 2001
From: Nenad Merdanovic 
Date: Fri, 25 Mar 2016 11:14:43 +0100
Subject: [PATCH] BUG/MEDIUM: Fix RFC5077 resumption when more than
 TLS_TICKETS_NO are present

Olivier Doucet reported the issue on the ML and tested that when using
more than TLS_TICKETS_NO keys in the file, the CPU usage is much higeher
than expected.

Lukas Tribus then provided a test case which showed that resumption doesn't
work at all in that case.

This fix needs to be backported to 1.6.

Signed-off-by: Nenad Merdanovic 
---
 src/ssl_sock.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 1017388..767d6e9 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -5406,7 +5406,7 @@ static int bind_parse_tls_ticket_keys(char **args, int 
cur_arg, struct proxy *px
fclose(f);
 
/* Use penultimate key for encryption, handle when TLS_TICKETS_NO = 1 */
-   i-=2;
+   i = (i - 2) % TLS_TICKETS_NO;
keys_ref->tls_ticket_enc_index = i < 0 ? 0 : i;
keys_ref->unique_id = -1;
conf->keys_ref = keys_ref;
-- 
2.7.0

Re: TLS Tickets and CPU usage

[Olivier Doucet]

Hi again,


2016-03-24 21:15 GMT+01:00 Lukas Tribus :

> Hi Nenad,
>
>
> >> Well, its not supposed to look like this, there is clearly something
> >> wrong. Master key fluctuates between the requests with TLS tickets
> >> and the reuse collumn shows failure.
> >
> > Looks like a haproxy bug, I think I can reproduce it.
> >
> > Can you try with EXACTLY 3 keys in /tmp/tls_ticket_keys?
>

Tried and now behaviour is like expected !
https://gist.github.com/anonymous/779fbc4f1cf8b23e9b1f

And, I can confirm that now, CPU is not doubled \o/





> there seems to be a bug in the handling of the tls-ticket-keys file.
>
> When there are 5 or more ticket keys in the file, clients using TLS tickets
> can no longer resume the TLS session (and fallback to full negotiation):
>
> https://gist.github.com/anonymous/6ec7c863f497cfd849a4
>
>
> Workaround would be to remove the oldest key from the file, so
> that the number of keys in the file remains below 5.
>
That's what I did : keep last 2 keys and add a new one.

 Olivier

Re: TLS Tickets and CPU usage

[Nenad Merdanovic]

Hey Lucas,

On 03/24/2016 09:15 PM, Lukas Tribus wrote:
> Hi Nenad,
> 
> 
>>> Well, its not supposed to look like this, there is clearly something
>>> wrong. Master key fluctuates between the requests with TLS tickets
>>> and the reuse collumn shows failure.
>>
>> Looks like a haproxy bug, I think I can reproduce it.
>>
>> Can you try with EXACTLY 3 keys in /tmp/tls_ticket_keys?
> 
> 
> there seems to be a bug in the handling of the tls-ticket-keys file.
> 
> When there are 5 or more ticket keys in the file, clients using TLS tickets
> can no longer resume the TLS session (and fallback to full negotiation):
> 
> https://gist.github.com/anonymous/6ec7c863f497cfd849a4
> 

Thanks a lot for the report. I think I have a fix, just need to validate it.

Regards,
Nenad

> 
> Workaround would be to remove the oldest key from the file, so
> that the number of keys in the file remains below 5.
> 
> 
> 
> cheers,
> 
> Lukas
> 
> 
> 

RE: TLS Tickets and CPU usage

[Lukas Tribus]

Hi Nenad,


>> Well, its not supposed to look like this, there is clearly something
>> wrong. Master key fluctuates between the requests with TLS tickets
>> and the reuse collumn shows failure.
>
> Looks like a haproxy bug, I think I can reproduce it.
>
> Can you try with EXACTLY 3 keys in /tmp/tls_ticket_keys?


there seems to be a bug in the handling of the tls-ticket-keys file.

When there are 5 or more ticket keys in the file, clients using TLS tickets
can no longer resume the TLS session (and fallback to full negotiation):

https://gist.github.com/anonymous/6ec7c863f497cfd849a4


Workaround would be to remove the oldest key from the file, so
that the number of keys in the file remains below 5.



cheers,

Lukas

  

RE: TLS Tickets and CPU usage

[Lukas Tribus]

Hi Oliver,


> 2016-03-24 17:12 GMT+01:00 Lukas Tribus  
> mailto:luky...@hotmail.com>>: 
> > If thats not it, and no old haproxy instances are present after the 
> > reload, could you compile Vincent's rfc5077-client from [1]: 
> > Output can be find here 
> > : https://gist.github.com/anonymous/6ec7c863f497cfd849a4 
> > (HTTP 500 error is normal, as you are using HEAD / HTTP/1.0 and our web 
> > servers require a Host header) 
>  
> Well, its not supposed to look like this, there is clearly something 
> wrong. Master key fluctuates between the requests with TLS tickets 
> and the reuse collumn shows failure. 


Looks like a haproxy bug, I think I can reproduce it.


Can you try with EXACTLY 3 keys in /tmp/tls_ticket_keys?

Then check with the rfc5077-client and if possible check CPU load in
production.


Thanks,

Lukas

  

Re: TLS Tickets and CPU usage

[Olivier Doucet]

2016-03-24 17:12 GMT+01:00 Lukas Tribus :

> > If thats not it, and no old haproxy instances are present after the
> > reload, could you compile Vincent's rfc5077-client from [1]:
> > Output can be find here
> > : https://gist.github.com/anonymous/6ec7c863f497cfd849a4
> > (HTTP 500 error is normal, as you are using HEAD / HTTP/1.0 and our web
> > servers require a Host header)
>
> Well, its not supposed to look like this, there is clearly something
> wrong. Master key fluctuates between the requests with TLS tickets
> and the reuse collumn shows failure.
>
> Are there any middleboxes between the server and the client? Can
> you try directly on the server so it doesn't leave the box (specifically
> it doesn't cross any firewalls or other SSL/TLS intercepting MITM).
>

I'm sure there is no firewall or MITM.
HAProxy is launched with nbproc 7, but the frontend I'm asking for is bind
to a single one.

Olivier

RE: TLS Tickets and CPU usage

[Lukas Tribus]

> If thats not it, and no old haproxy instances are present after the 
> reload, could you compile Vincent's rfc5077-client from [1]: 
> Output can be find here 
> : https://gist.github.com/anonymous/6ec7c863f497cfd849a4 
> (HTTP 500 error is normal, as you are using HEAD / HTTP/1.0 and our web 
> servers require a Host header) 

Well, its not supposed to look like this, there is clearly something
wrong. Master key fluctuates between the requests with TLS tickets
and the reuse collumn shows failure.

Are there any middleboxes between the server and the client? Can
you try directly on the server so it doesn't leave the box (specifically
it doesn't cross any firewalls or other SSL/TLS intercepting MITM).


Lukas

  

Re: TLS Tickets and CPU usage

[Olivier Doucet]

2016-03-24 12:57 GMT+01:00 Lukas Tribus :

> > Ok, when you say CPU usage double do you mean the CPU usage after
> > a reload/restart, or do you mean CPU usage in general (even after not
> > reloading haproxy)?
> > CPU is at 100% just after reload for more than 30s (was a few seconds
> > before) and then CPU usage stays doubled all the time.
>
> Ok, so it looks like resumption doesn't work at all with TLS tickets.
>
> Are you sure the haproxy reload works fine - no old haproxy instances
> run in the background serving obsolete TLS keys?
>
Yes, I'm sure.


> There have been some bugs with reloading haproxy, fixed in 1.6.4.
>
I recompiled HAProxy with latest version and OpenSSL 1.0.2g.

I activated tls-ticket and CPU usage doubled again.
I tried a reload then, and CPU stays stable.

So at least the reload problem with CPU at 100% seems resolved. But I do
not understand why using TLS-tickets is using so much more CPUs (I hoped it
would be "slightly" higher, not doubled).

BTW, servers are 2x Intel Xeon L5630  @ 2.13GHz and certificates issued are
all SHA256RSA.
I will use ECDSA certificates in the future, I was just waiting for
transparent support of ECDSA/RSA certificates in HAProxy (done in 1.7, just
waiting for the stable release on this).




> If thats not it, and no old haproxy instances are present after the
> reload, could you compile Vincent's rfc5077-client from [1]:
>
Output can be find here :
https://gist.github.com/anonymous/6ec7c863f497cfd849a4
(HTTP 500 error is normal, as you are using HEAD / HTTP/1.0 and our web
servers require a Host header)

Olivier

RE: TLS Tickets and CPU usage

[Lukas Tribus]

> Ok, when you say CPU usage double do you mean the CPU usage after 
> a reload/restart, or do you mean CPU usage in general (even after not 
> reloading haproxy)? 
> CPU is at 100% just after reload for more than 30s (was a few seconds 
> before) and then CPU usage stays doubled all the time. 

Ok, so it looks like resumption doesn't work at all with TLS tickets.

Are you sure the haproxy reload works fine - no old haproxy instances
run in the background serving obsolete TLS keys?

There have been some bugs with reloading haproxy, fixed in 1.6.4.


If thats not it, and no old haproxy instances are present after the
reload, could you compile Vincent's rfc5077-client from [1]:

git clone https://github.com/vincentbernat/rfc5077.git
cd rfc5077
make rfc5077-client


./rfc5077-client -4 


Make sure you have the dependencies installed from the
github page (mainly libssl-dev and pkg-config).



cheers,

Lukas


[1] https://github.com/vincentbernat/rfc5077



  

Re: TLS Tickets and CPU usage

[Olivier Doucet]

Hi Lukas,



2016-03-24 11:15 GMT+01:00 Lukas Tribus :

> > But CPU usage doubled ! I disabled it by adding again
> > "ssl-default-bind-options no-tls-tickets" and CPU usage returned to
> > normal.
>
> Ok, when you say CPU usage double do you mean the CPU usage after
> a reload/restart, or do you mean CPU usage in general (even after not
> reloading haproxy)?
>
CPU is at 100% just after reload for more than 30s (was a few seconds
before) and then CPU usage stays doubled all the time.



>
> > And /tmp/tls_ticket_keys generated with "openssl rand -base64 48"
> > called 3x + appended at each reload.
>
> By calling it 3 times you are basically destroying the old keys making
> sure that TLS tickets CANNOT be reused. You must only generate
> a new key ONCE per reload.
>

I misspoke . I generate 3 keys on haproxy first startup, then append only
one ticket at each reload.

Olivier

RE: TLS Tickets and CPU usage

[Lukas Tribus]

Hi Oliver,


> Hello guys, 
> 
> I'm having troubles with HAProxy 1.6.3 and TLS ticket, so let me 
> explain here my case. 
> 
> I'm running HAProxy 1.6.3 (since december) and all was running fine. 
> TLS ticket was explicitely disabled. The only downside of this setup is 
> that after each reload, I have a CPU spike for a few seconds. I thought 
> this was due to session renegociation (right ?) 
> 
> A few days ago, I decided to activate TLS-Ticket and use option 
> tls-ticket-keys on bind lines. My hope was to remove this CPU spike, as 
> session renegociation should be faster. 
> But CPU usage doubled ! I disabled it by adding again 
> "ssl-default-bind-options no-tls-tickets" and CPU usage returned to 
> normal. 

Ok, when you say CPU usage double do you mean the CPU usage after
a reload/restart, or do you mean CPU usage in general (even after not
reloading haproxy)?



> And /tmp/tls_ticket_keys generated with "openssl rand -base64 48" 
> called 3x + appended at each reload.

By calling it 3 times you are basically destroying the old keys making
sure that TLS tickets CANNOT be reused. You must only generate
a new key ONCE per reload.


Lukas

  

TLS Tickets and CPU usage

[Olivier Doucet]

Hello guys,

I'm having troubles with HAProxy 1.6.3 and TLS ticket, so let me explain
here my case.

I'm running HAProxy 1.6.3 (since december) and all was running fine. TLS
ticket was explicitely disabled. The only downside of this setup is that
after each reload, I have a CPU spike for a few seconds. I thought this was
due to session renegociation (right ?)

A few days ago, I decided to activate TLS-Ticket and use option
tls-ticket-keys on bind lines. My hope was to remove this CPU spike, as
session renegociation should be faster.
But CPU usage doubled ! I disabled it by adding again
"ssl-default-bind-options no-tls-tickets" and CPU usage returned to normal.

>From the doc, I read that activating TLS ticket may use "slightly" more
CPU, but I hoped that using tickets file could help in this case.
Apparently I'm wrong.

Any detailed explanation and feedback would be really useful here.

Snippet of my config (I know I'm using old syntax for listen/bind) :

global
tune.ssl.default-dh-param 2048
tune.ssl.lifetime 100800
tune.ssl.cachesize 100
#ssl-default-bind-options no-tls-tickets
ssl-default-bind-ciphers
ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384
:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-D
ES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!
DES:!MD5:!PSK:!RC4

listen :443
bind xxx:443 ssl crt /etc/ssl/ssl_xxx.pem no-sslv3 tls-ticket-keys
/tmp/tls_ticket_keys
server s107 xxx:80 check weight 5 fall 60

***
HAProxy version :

HA-Proxy version 1.6.3 2015/12/25
Copyright 2000-2015 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = native
  CC  = gcc
  CFLAGS  = -O2 -march=native -g -fno-strict-aliasing
-Wdeclaration-after-statement
  OPTIONS = USE_OPENSSL=1 USE_PCRE=1 USE_TFO=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built without compression support (neither USE_ZLIB nor USE_SLZ are set)
Compression algorithms supported : identity("identity")
Built with OpenSSL version : OpenSSL 1.0.2f  28 Jan 2016
Running on OpenSSL version : OpenSSL 1.0.2f  28 Jan 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.2 2007-06-19
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND


*
And /tmp/tls_ticket_keys generated with "openssl rand -base64 48" called 3x
+ appended at each reload.


Olivier