Re: v1.9.6+HEAD: segfault in h1_skip_chunk_crlf

[Willy Tarreau]

Hi Christopher,

On Thu, Apr 18, 2019 at 09:36:30PM +0200, Christopher Faulet wrote:
> > It seems related to the last commits from Christopher Faulet, maybe
> > around this commit:
> > http://git.haproxy.org/?p=haproxy-1.9.git;a=commit;h=0c2973662163ab2753a54e729ecdb09dd694c2dd
> > BUG/MINOR: mux-h1: Process input even if the input buffer is empty
> > 
> Hi,
> 
> You're right there is a bug in this commit. Here is a patch that should fix
> the issue.

Good catch but in my opinion we should instead fix the called function
(h1_skip_chunk_crlf). h1_skip_chunk_size() properly deals with the case
where start==stop, but h1_skip_chunk_crlf() says "If not enough data are
available, the function does not change anything and returns zero", which
is not true so others will be caught again. Thus your mux-h1 code was
correct. I'd rather simply do this to comply with the promise in the
comment :

diff --git a/include/common/h1.h b/include/common/h1.h
index f0f2039..4b73ed4 100644
--- a/include/common/h1.h
+++ b/include/common/h1.h
@@ -201,6 +201,9 @@ static inline int h1_skip_chunk_crlf(const struct buffer *bu
const char *ptr = b_peek(buf, start);
int bytes = 1;
 
+   if (stop <= start)
+   return 0;
+
/* NB: we'll check data availability at the end. It's not a
 * problem because whatever we match first will be checked
 * against the correct length.

Thanks,
Willy

Re: [PATCH] FEATURE/MEDIUM: enable travis-ci builds

[Илья Шипицин]

чт, 18 апр. 2019 г. в 21:51, Илья Шипицин :

>
>
> чт, 18 апр. 2019 г. в 21:45, Willy Tarreau :
>
>> Hi Ilya,
>>
>> On Wed, Apr 17, 2019 at 01:06:11PM +0500,  ??? wrote:
>> > btw, we can run lua tests
>> >
>> > https://travis-ci.com/chipitsine/haproxy-1/builds/108641032
>> >
>> > so... how do we want to run lua ? always enabled ? or two builds (with
>> and
>> > without lua) ?
>>
>> I'd say that we generally detect more problems by having lots of features
>> than few features. However it's true that build issues are often
>> discovered
>> when features are disabled precisely because developers tend to enable
>> lots
>> of them to cover a wide range of issues.
>>
>> Thus at first enabling it would possibly be better. Later if you figure
>> it's cheap to run multiple builds, it might make sense to have two builds,
>> or even try to randomly mix certain options. Another thing that regularly
>> breaks is openssl-derivatives like LibreSSL and BoringSSL. I'm just not
>> convinced there are enough users to warrant extra tests at the moment
>> though.
>>
>
> I planned to add various openssl versions like 1.0.1, 1.0.2, 1.0.0, 1.1.1,
> ...
> I think we can add libressl, boringssl as well.
>
>
>
>>
>> I've merged your patch in its current state hoping it will help, but
>> I still have zero idea about how to use it, so please send some doc
>> ASAP.
>>
>
> also travis-ci must be enabled on github project,
> please follow this guide
> https://docs.travis-ci.com/user/tutorial#to-get-started-with-travis-ci
>


I mean the following steps (after that builds will run on every commit)

-

Go to Travis-ci.com  and *Sign up with GitHub*
.
-

Accept the Authorization of Travis CI. You’ll be redirected to GitHub.
-

Click the green *Activate* button, and select the repositories you want to
use with Travis CI.


>
>
>
>>
>> Thanks,
>> Willy
>>
>

Re: v1.9.6+HEAD: segfault in h1_skip_chunk_crlf

[Christopher Faulet]

Le 18/04/2019 à 16:55, William Dauchy a écrit :

Hello,

We are triggering a segfault on the last HEAD of haproxy-1.9 tree, last
commit being
1e0fd266db3e503783ff623faabcb1dfe211cb89 BUG/MINOR: mworker: disable busy 
polling in the master process

backtrace:

Thread 1 (Thread 0x7f73aeffd700 (LWP 13044)):
#0  h1_skip_chunk_crlf (stop=0, start=0, buf=0x7f739802b708) at 
include/common/h1.h:208
208 if (*ptr == '\r') {
#1  h1_process_data (h1s=h1s@entry=0x7f739802a910, 
h1m=h1m@entry=0x7f739802a998, htx=0x7f739803a0f0, buf=buf@entry=0x7f739802b708, 
ofs=ofs@entry=0x7f73aefda9a8, max=max@entry=0, 
htxbuf=htxbuf@entry=0x7f73980313e8, reserve=reserve@entry=1024) at 
src/mux_h1.c:1204
#2  0x5623b1086dc3 in h1_process_input (flags=, 
buf=0x7f73980313e8, h1c=0x7f739802b6f0) at src/mux_h1.c:1391
#3  h1_rcv_buf (cs=, buf=0x7f73980313e8, count=, 
flags=) at src/mux_h1.c:2289
#4  0x5623b10b9c99 in si_cs_recv (cs=cs@entry=0x7f73980219f0) at 
src/stream_interface.c:1258
#5  0x5623b10ba160 in si_cs_io_cb (t=, ctx=, 
state=) at src/stream_interface.c:739
#6  0x5623b10ea30a in process_runnable_tasks () at src/task.c:390
#7  0x5623b106336f in run_poll_loop () at src/haproxy.c:2648
#8  run_thread_poll_loop (data=) at src/haproxy.c:2713
#9  0x7f73bbae6dd5 in start_thread () from /lib64/libpthread.so.0
#10 0x7f73ba81fead in clone () from /lib64/libc.so.6

It seems related to the last commits from Christopher Faulet, maybe
around this commit:
http://git.haproxy.org/?p=haproxy-1.9.git;a=commit;h=0c2973662163ab2753a54e729ecdb09dd694c2dd
BUG/MINOR: mux-h1: Process input even if the input buffer is empty


Hi,

You're right there is a bug in this commit. Here is a patch that should 
fix the issue.


Thanks,
--
Christopher
>From ee37367d428bdde70ce8d406b18a6701eafd535a Mon Sep 17 00:00:00 2001
From: Christopher Faulet 
Date: Thu, 18 Apr 2019 21:24:28 +0200
Subject: [PATCH] BUG/MEDIUM: mux-h1: Don't try to parse chunks if there is no
 data to read

H1 function used to parse the chunks CRLF must not be called when there is no
data in the buffer. The bug was introduced by the commit 91f77d599 ("BUG/MINOR:
mux-h1: Process input even if the input buffer is empty"). But, in fact, there
is no reason to try to parse chunks metadata if there is no data.

This patch must be backported to 1.9.
---
 src/mux_h1.c | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/src/mux_h1.c b/src/mux_h1.c
index 46bc33c90..529344805 100644
--- a/src/mux_h1.c
+++ b/src/mux_h1.c
@@ -1155,6 +1155,8 @@ static size_t h1_process_data(struct h1s *h1s, struct h1m *h1m, struct htx *htx,
 		  new_chunk:
 			/* te:chunked : parse chunks */
 			if (h1m->state == H1_MSG_CHUNK_CRLF) {
+if (!max)
+	goto end;
 ret = h1_skip_chunk_crlf(buf, *ofs, *ofs + max);
 if (ret <= 0)
 	goto end;
@@ -1168,6 +1170,8 @@ static size_t h1_process_data(struct h1s *h1s, struct h1m *h1m, struct htx *htx,
 			if (h1m->state == H1_MSG_CHUNK_SIZE) {
 unsigned int chksz;
 
+if (!max)
+	goto end;
 ret = h1_parse_chunk_size(buf, *ofs, *ofs + max, );
 if (ret <= 0)
 	goto end;
@@ -1220,6 +1224,8 @@ static size_t h1_process_data(struct h1s *h1s, struct h1m *h1m, struct htx *htx,
 if (h1s->flags & H1S_F_HAVE_I_TLR)
 	goto skip_tlr_parsing;
 
+if (!max)
+	goto end;
 ret = h1_measure_trailers(buf, *ofs, *ofs + max);
 if (ret > data_space)
 	ret = (htx_is_empty(htx) ? -1 : 0);
-- 
2.20.1

Re: [PR] IPv6: properly format an address coming from IPv6 socket as hex in lf_ip

[Willy Tarreau]

Hi Lukas,

On Thu, Apr 18, 2019 at 08:45:18PM +0200, Lukas Tribus wrote:
> Hi Willy,
> 
> 
> On Fri, 8 Mar 2019 at 14:23, PR Bot  wrote:
> >
> > Dear list!
> >
> > Author: Radek Zajic 
> > Number of patches: 1
> >
> > This is an automated relay of the Github pull request:
> >IPv6: properly format an address coming from IPv6 socket as hex in
> >lf_ip
> >
> > Patch title(s):
> >IPv6: properly format an address coming from IPv6 socket as hex string 
> > in lf_ip
> >
> > Link:
> >https://github.com/haproxy/haproxy/pull/59
> >
> > Edit locally:
> >wget https://github.com/haproxy/haproxy/pull/59.patch && vi 59.patch
> >
> > Apply locally:
> >curl https://github.com/haproxy/haproxy/pull/59.patch | git am -
> >
> > Description:
> >Dear `haproxy` maintainers,
> >
> >we've recently discovered an
> >issue with formatting IPv6 addresses as hex strings (in e.g.
> >`%{+X}ci`). This issue occurs e.g. when formatting `unique-id-format`
> >as recommended in the docs: `%{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid`.
> >This pull request proposes a change that properly formats addresses
> >from IPv4 sockets as 8-char hex-string and addresses from IPv6 sockets
> >as 32-char hex-string.
> > [...]
> 
> 
> For housekeeping purposes: I believe this patch fall through the
> cracks (I did not look at it though).

You're right. I thought I took it already but very likely I was too
busy to spend time figuring what the original issue was and redoing
it with a proper commit message :-/

Radek, please have a look at CONTRIBUTING and make sure that your
commit message explains what the problem was, what solution you
decided on and why it's supposedly the best. Also any hint about
the suitability for backporting to stable branches would be welcome.

Thanks,
Willy

Re: [PR] IPv6: properly format an address coming from IPv6 socket as hex in lf_ip

[Lukas Tribus]

Hi Willy,


On Fri, 8 Mar 2019 at 14:23, PR Bot  wrote:
>
> Dear list!
>
> Author: Radek Zajic 
> Number of patches: 1
>
> This is an automated relay of the Github pull request:
>IPv6: properly format an address coming from IPv6 socket as hex in
>lf_ip
>
> Patch title(s):
>IPv6: properly format an address coming from IPv6 socket as hex string in 
> lf_ip
>
> Link:
>https://github.com/haproxy/haproxy/pull/59
>
> Edit locally:
>wget https://github.com/haproxy/haproxy/pull/59.patch && vi 59.patch
>
> Apply locally:
>curl https://github.com/haproxy/haproxy/pull/59.patch | git am -
>
> Description:
>Dear `haproxy` maintainers,
>
>we've recently discovered an
>issue with formatting IPv6 addresses as hex strings (in e.g.
>`%{+X}ci`). This issue occurs e.g. when formatting `unique-id-format`
>as recommended in the docs: `%{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid`.
>This pull request proposes a change that properly formats addresses
>from IPv4 sockets as 8-char hex-string and addresses from IPv6 sockets
>as 32-char hex-string.
> [...]


For housekeeping purposes: I believe this patch fall through the
cracks (I did not look at it though).


lukas

Re: [PATCH] wurfl device detection build fixes and dummy library

[Willy Tarreau]

Hi Paul,

On Thu, Apr 18, 2019 at 02:46:17PM +0200, Paul Stephen Borile wrote:
> Hi All, Willy,
> 
> please find attached to this email the 6 patches that cover various areas
> of restyling of
> the WURFL device detection feature for HAProxy. All patches can be back
> ported to 1.9 if necessary.
> Last patch is a dummy WURFL library that can be used to build/run haproxy
> compiled with the USE_WURFL option to make easier checking for any build
> problem in the future.
> We'll try to do the same and make sure that the module does not break
> builds again as happened in the past.

Many thanks for taking care of this. I'll have a look at it ASAP.

Thanks,
Willy

Re: [PATCH] FEATURE/MEDIUM: enable travis-ci builds

[Илья Шипицин]

чт, 18 апр. 2019 г. в 21:45, Willy Tarreau :

> Hi Ilya,
>
> On Wed, Apr 17, 2019 at 01:06:11PM +0500,  ??? wrote:
> > btw, we can run lua tests
> >
> > https://travis-ci.com/chipitsine/haproxy-1/builds/108641032
> >
> > so... how do we want to run lua ? always enabled ? or two builds (with
> and
> > without lua) ?
>
> I'd say that we generally detect more problems by having lots of features
> than few features. However it's true that build issues are often discovered
> when features are disabled precisely because developers tend to enable lots
> of them to cover a wide range of issues.
>
> Thus at first enabling it would possibly be better. Later if you figure
> it's cheap to run multiple builds, it might make sense to have two builds,
> or even try to randomly mix certain options. Another thing that regularly
> breaks is openssl-derivatives like LibreSSL and BoringSSL. I'm just not
> convinced there are enough users to warrant extra tests at the moment
> though.
>

I planned to add various openssl versions like 1.0.1, 1.0.2, 1.0.0, 1.1.1,
...
I think we can add libressl, boringssl as well.



>
> I've merged your patch in its current state hoping it will help, but
> I still have zero idea about how to use it, so please send some doc
> ASAP.
>

also travis-ci must be enabled on github project,
please follow this guide
https://docs.travis-ci.com/user/tutorial#to-get-started-with-travis-ci


>
> Thanks,
> Willy
>

Re: [PATCH] FEATURE/MEDIUM: enable travis-ci builds

[Willy Tarreau]

Hi Ilya,

On Wed, Apr 17, 2019 at 01:06:11PM +0500,  ??? wrote:
> btw, we can run lua tests
> 
> https://travis-ci.com/chipitsine/haproxy-1/builds/108641032
> 
> so... how do we want to run lua ? always enabled ? or two builds (with and
> without lua) ?

I'd say that we generally detect more problems by having lots of features
than few features. However it's true that build issues are often discovered
when features are disabled precisely because developers tend to enable lots
of them to cover a wide range of issues.

Thus at first enabling it would possibly be better. Later if you figure
it's cheap to run multiple builds, it might make sense to have two builds,
or even try to randomly mix certain options. Another thing that regularly
breaks is openssl-derivatives like LibreSSL and BoringSSL. I'm just not
convinced there are enough users to warrant extra tests at the moment
though.

I've merged your patch in its current state hoping it will help, but
I still have zero idea about how to use it, so please send some doc
ASAP.

Thanks,
Willy

Re: Randomly high CPU usage

[Willy Tarreau]

Hello Marco,

On Thu, Apr 18, 2019 at 05:27:26PM +0200, Marco Corte wrote:
> Hello!
> 
> From time to time, about twice daily, and without any apparent reason,
> haproxy jumps from using about 15% CPU usage to 100% (relative to the single
> core it can use).
> The situation becomes normal again after about 15-20 minutes.
(...)

We've been fighting with these ones over the last few days and they
had multiple causes, some in the scheduler, some at other locations,
like streams not properly dealing with certain combinations of errors.
I've backported all these recent fixes to 1.9 today. If the problem
happens often enough for you to notice it, it would be useful to test
either the latest git or tomorrow's snapshot. Otherwise we'll emit a
new 1.9 soon to flush the pipe of these annoying issues.

> During one of these events, I was able to capture (see below)
>   sudo socat /var/run/haproxy.socket - <<< "show info"
> 
> If anyone is interested, I also have a quite long
>   sudo socat /var/run/haproxy.socket - <<< "show sess all"

Please keep this one until you can confirm the issue is gone, I may
have to ask it to you if it's still present.

> Unfortunately I was not fast enough to collect
>   sudo socat /var/run/haproxy.socket - <<< "show fd"
> during the event

Don't worry, these are all useful information already.

Thanks!
Willy

Randomly high CPU usage

[Marco Corte]

Hello!

From time to time, about twice daily, and without any apparent reason, 
haproxy jumps from using about 15% CPU usage to 100% (relative to the 
single core it can use).

The situation becomes normal again after about 15-20 minutes.

During one of these events, I was able to capture (see below)
  sudo socat /var/run/haproxy.socket - <<< "show info"

If anyone is interested, I also have a quite long
  sudo socat /var/run/haproxy.socket - <<< "show sess all"

Unfortunately I was not fast enough to collect
  sudo socat /var/run/haproxy.socket - <<< "show fd"
during the event


Any help/idea is appreciated.

Thank you

.marcoc


Name: HAProxy
Version: 1.9.6-1ppa1~bionic
Release_date: 2019/03/30
Nbthread: 1
Nbproc: 1
Process_num: 1
Pid: 1101
Uptime: 0d 9h08m22s
Uptime_sec: 32902
Memmax_MB: 0
PoolAlloc_MB: 12
PoolUsed_MB: 12
PoolFailed: 0
Ulimit-n: 50316
Maxsock: 50316
Maxconn: 2
Hard_maxconn: 2
CurrConns: 1726
CumConns: 605176
CumReq: 2797465
MaxSslConns: 0
CurrSslConns: 1743
CumSslConns: 721023
Maxpipes: 5000
PipesUsed: 0
PipesFree: 2
ConnRate: 16
ConnRateLimit: 0
MaxConnRate: 123
SessRate: 16
SessRateLimit: 0
MaxSessRate: 123
SslRate: 14
SslRateLimit: 0
MaxSslRate: 118
SslFrontendKeyRate: 6
SslFrontendMaxKeyRate: 95
SslFrontendSessionReuse_pct: 57
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 4
SslCacheLookups: 142428
SslCacheMisses: 119326
CompressBpsIn: 1656949
CompressBpsOut: 294846
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 2315
Run_queue: 2
Idle_pct: 85
node: pgli01
Stopping: 0
Jobs: 1816
Unstoppable Jobs: 0
Listeners: 88
ActivePeers: 1
ConnectedPeers: 1
DroppedLogs: 0
BusyPolling: 0

v1.9.6+HEAD: segfault in h1_skip_chunk_crlf

[William Dauchy]

Hello,

We are triggering a segfault on the last HEAD of haproxy-1.9 tree, last
commit being
1e0fd266db3e503783ff623faabcb1dfe211cb89 BUG/MINOR: mworker: disable busy 
polling in the master process

backtrace:

Thread 1 (Thread 0x7f73aeffd700 (LWP 13044)):
#0  h1_skip_chunk_crlf (stop=0, start=0, buf=0x7f739802b708) at 
include/common/h1.h:208
208 if (*ptr == '\r') {
#1  h1_process_data (h1s=h1s@entry=0x7f739802a910, 
h1m=h1m@entry=0x7f739802a998, htx=0x7f739803a0f0, buf=buf@entry=0x7f739802b708, 
ofs=ofs@entry=0x7f73aefda9a8, max=max@entry=0, 
htxbuf=htxbuf@entry=0x7f73980313e8, reserve=reserve@entry=1024) at 
src/mux_h1.c:1204
#2  0x5623b1086dc3 in h1_process_input (flags=, 
buf=0x7f73980313e8, h1c=0x7f739802b6f0) at src/mux_h1.c:1391
#3  h1_rcv_buf (cs=, buf=0x7f73980313e8, count=, 
flags=) at src/mux_h1.c:2289
#4  0x5623b10b9c99 in si_cs_recv (cs=cs@entry=0x7f73980219f0) at 
src/stream_interface.c:1258
#5  0x5623b10ba160 in si_cs_io_cb (t=, ctx=, 
state=) at src/stream_interface.c:739
#6  0x5623b10ea30a in process_runnable_tasks () at src/task.c:390
#7  0x5623b106336f in run_poll_loop () at src/haproxy.c:2648
#8  run_thread_poll_loop (data=) at src/haproxy.c:2713
#9  0x7f73bbae6dd5 in start_thread () from /lib64/libpthread.so.0
#10 0x7f73ba81fead in clone () from /lib64/libc.so.6

It seems related to the last commits from Christopher Faulet, maybe
around this commit:
http://git.haproxy.org/?p=haproxy-1.9.git;a=commit;h=0c2973662163ab2753a54e729ecdb09dd694c2dd
BUG/MINOR: mux-h1: Process input even if the input buffer is empty

Best,
-- 
William

Re: QAT intermittent healthcheck errors

[Emeric Brun]

On 4/18/19 11:06 AM, Emeric Brun wrote:
> Hi Marcin,
> 
> On 4/12/19 6:10 PM, Marcin Deranek wrote:
>> Hi Emeric,
>>
>> On 4/12/19 5:26 PM, Emeric Brun wrote:
>>
>>> Do you have ssl enabled on the server side?
>>
>> Yes, ssl is on frontend and backend with ssl checks enabled.
>>
>>> If it is the case could replace health check with a simple tcp check 
>>> (without ssl)?
>>
>> What I noticed before that if I (re)start HAProxy and reload immediately no 
>> stuck processes are present. If I wait before reloading stuck processes show 
>> up.
>> After disabling checks (I still keep ssl enabled for normal traffic) reloads 
>> work just fine (tried many time). Do you know how to enable TCP healthchecks 
>> while keeping SSL for non-healthcheck requests ?
> 
> I think you can do that this way:
> 
> Remove the option httchk (or prefix it by "no": "no option httchk " if it is 
> configured into the defaults section
> 
> and add the following 2 lines:
> 
> option tcp-check
> tcp-check connect
> 
> This shouldn't perform the handshake but just validate that the port is open. 
> The regular traffic will continue to use the ssl
> on server side.
> 
>  
>>> Regarding the show info/lsoff  it seems there is no more sessions on client 
>>> side but remaining ssl jobs (CurrSslConns) and I supsect the health checks 
>>> to miss a cleanup of their ssl sessions using the QAT. (this is just an 
>>> assumption)
>>
>> In general instance where I test QAT does not have any "real" client traffic 
>> except small amount of healtcheck requests per frontend which are internally 
>> handled by HAProxy itself. Still TLS handshake still needs to take place. 
>> There are many more backend healthchecks. Looks like your assumption was 
>> correct..
> 
> Good!, We continue to dig in that direction.
> 
> An other interesting trace would be to perform a "show sess" command on a 
> stucked process through the master cli.

And also the "show fd" 

R,
Emeric

[PATCH] wurfl device detection build fixes and dummy library

[Paul Stephen Borile]

Hi All, Willy,

please find attached to this email the 6 patches that cover various areas
of restyling of
the WURFL device detection feature for HAProxy. All patches can be back
ported to 1.9 if necessary.
Last patch is a dummy WURFL library that can be used to build/run haproxy
compiled with the USE_WURFL option to make easier checking for any build
problem in the future.
We'll try to do the same and make sure that the module does not break
builds again as happened in the past.

-Paul

-- 
Paul Stephen Borile
Director, WURFL Product Development - Italy
Scientiamobile Inc -  paul.bor...@scientiamobile.com +39 348 7108474
Milano Office : +39 02 620227260
skype: paulstephenborile
"Reducing complexity and size must be the goal in every step."  Niklaus
Wirth
From d04f6a5d94f59c043a3819e7626981c41c6c2779 Mon Sep 17 00:00:00 2001
From: paulborile 
Date: Thu, 18 Apr 2019 11:57:04 +0200
Subject: [PATCH 3/6] CLEANUP: wurfl: removed deprecated methods

last 2 major releases of libwurfl included a complete review of engine options with
the result of deprecating many features. The patch removes unecessary code and fixes
the documentation.
Can be backported on any version of haproxy.
---
 doc/WURFL-device-detection.txt |   9 +--
 doc/configuration.txt  |  37 ++--
 examples/wurfl-example.cfg |  10 +---
 src/wurfl.c| 106 +++--
 4 files changed, 15 insertions(+), 147 deletions(-)

diff --git a/doc/WURFL-device-detection.txt b/doc/WURFL-device-detection.txt
index c9686151..0d61db0a 100644
--- a/doc/WURFL-device-detection.txt
+++ b/doc/WURFL-device-detection.txt
@@ -23,25 +23,18 @@ These are the supported WURFL directives (see doc/configuration.txt) :
virtual capabilities, property names we plan to use in injected headers)
 - wurfl-information-list-separator  (character that will be
used to separate values in a response header, ',' by default).
-- wurfl-engine-mode  (Sets the WURFL engine target. You can choose
-   between "accuracy" and "performance","performance" by default)
 - wurfl-cache-size  (Sets the WURFL caching strategy)
 - wurfl-patch-file [] (Sets the paths to custom WURFL patch files)
 
 Sample configuration :
 
 global
-	wurfl-data-file /usr/share/wurfl/wurfl-eval.xml
+	wurfl-data-file /usr/share/wurfl/wurfl.zip
 
 	wurfl-information-list wurfl_id model_name
 
 	#wurfl-information-list-separator |
 
-	wurfl-engine-mode performance
-	#wurfl-engine-mode accuracy
-
-	## double LRU cache
-	wurfl-cache-size 10,3
 	## single LRU cache
 	#wurfl-cache-size 10
 	## no cache
diff --git a/doc/configuration.txt b/doc/configuration.txt
index ccf614fd..5b99da47 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -599,9 +599,7 @@ The following keywords are supported in the "global" section :
- wurfl-data-file
- wurfl-information-list
- wurfl-information-list-separator
-   - wurfl-engine-mode
- wurfl-cache-size
-   - wurfl-useragent-priority
 
  * Performance tuning
- max-spread-checks
@@ -1246,10 +1244,6 @@ wurfl-information-list []*
   - wurfl_api_version   Contains a string representing the currently
 used Libwurfl API version.
 
-  - wurfl_engine_target Contains a string representing the currently
-set WURFL Engine Target. Possible values are
-"HIGH_ACCURACY", "HIGH_PERFORMANCE", "INVALID".
-
   - wurfl_info  A string containing information on the parsed
 wurfl.xml and its full path.
 
@@ -1258,8 +1252,6 @@ wurfl-information-list []*
 
   - wurfl_normalized_useragent  The normalized useragent.
 
-  - wurfl_useragent_priorityThe user agent priority used by WURFL.
-
   Please note that this option is only available when haproxy has been compiled
   with USE_WURFL=1.
 
@@ -1277,36 +1269,15 @@ wurfl-patch-file []
   Please note that this option is only available when haproxy has been compiled
   with USE_WURFL=1.
 
-wurfl-engine-mode { accuracy | performance }
-  Sets the WURFL engine target. You can choose between 'accuracy' or
-  'performance' targets. In performance mode, desktop web browser detection is
-  done programmatically without referencing the WURFL data. As a result, most
-  desktop web browsers are returned as generic_web_browser WURFL ID for
-  performance. If either performance or accuracy are not defined, performance
-  mode is enabled by default.
-
-  Please note that this option is only available when haproxy has been compiled
-  with USE_WURFL=1.
-
-wurfl-cache-size [,]
-  Sets the WURFL caching strategy. Here  is the Useragent cache size, and
-   is the internal device cache size. There are three possibilities here :
+wurfl-cache-size 
+  Sets the WURFL Useragent cache size. For faster lookups, already processed user
+  agents are kept in a LRU cache :
   - "0" : no cache is used.
-  -  : the Single LRU cache is 

Re: QAT intermittent healthcheck errors

[Emeric Brun]

Hi Marcin,

On 4/12/19 6:10 PM, Marcin Deranek wrote:
> Hi Emeric,
> 
> On 4/12/19 5:26 PM, Emeric Brun wrote:
> 
>> Do you have ssl enabled on the server side?
> 
> Yes, ssl is on frontend and backend with ssl checks enabled.
> 
>> If it is the case could replace health check with a simple tcp check 
>> (without ssl)?
> 
> What I noticed before that if I (re)start HAProxy and reload immediately no 
> stuck processes are present. If I wait before reloading stuck processes show 
> up.
> After disabling checks (I still keep ssl enabled for normal traffic) reloads 
> work just fine (tried many time). Do you know how to enable TCP healthchecks 
> while keeping SSL for non-healthcheck requests ?

I think you can do that this way:

Remove the option httchk (or prefix it by "no": "no option httchk " if it is 
configured into the defaults section

and add the following 2 lines:

option tcp-check
tcp-check connect

This shouldn't perform the handshake but just validate that the port is open. 
The regular traffic will continue to use the ssl
on server side.

 
>> Regarding the show info/lsoff  it seems there is no more sessions on client 
>> side but remaining ssl jobs (CurrSslConns) and I supsect the health checks 
>> to miss a cleanup of their ssl sessions using the QAT. (this is just an 
>> assumption)
> 
> In general instance where I test QAT does not have any "real" client traffic 
> except small amount of healtcheck requests per frontend which are internally 
> handled by HAProxy itself. Still TLS handshake still needs to take place. 
> There are many more backend healthchecks. Looks like your assumption was 
> correct..

Good!, We continue to dig in that direction.

An other interesting trace would be to perform a "show sess" command on a 
stucked process through the master cli.

R,
Emeric

[PATCH] http-request do-resolve

[Baptiste]

Hi all, Willy,

Please find attached to this email the 4 patches for the http-request
do-resolve action I submitted a few months ago.
I integrated all feedback from Willy and also now support tcp-request
content do-resolve.

Baptiste
From e96ff49ee05dbdc15dc7582349e6314dcfccb20e Mon Sep 17 00:00:00 2001
From: Baptiste Assmann 
Date: Tue, 30 Jan 2018 08:10:20 +0100
Subject: [PATCH 3/5] MINOR: obj_type: new object type for struct stream

This patch creates a new obj_type for the struct stream in HAProxy.
---
 include/proto/obj_type.h | 13 +
 include/types/obj_type.h |  1 +
 include/types/stream.h   |  4 +++-
 3 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/include/proto/obj_type.h b/include/proto/obj_type.h
index 47273ca..19865bb 100644
--- a/include/proto/obj_type.h
+++ b/include/proto/obj_type.h
@@ -30,6 +30,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 
 static inline enum obj_type obj_type(enum obj_type *t)
@@ -158,6 +159,18 @@ static inline struct dns_srvrq *objt_dns_srvrq(enum obj_type *t)
 	return __objt_dns_srvrq(t);
 }
 
+static inline struct stream *__objt_stream(enum obj_type *t)
+{
+	return container_of(t, struct stream, obj_type);
+}
+
+static inline struct stream *objt_stream(enum obj_type *t)
+{
+	if (!t || *t != OBJ_TYPE_STREAM)
+		return NULL;
+	return __objt_stream(t);
+}
+
 static inline void *obj_base_ptr(enum obj_type *t)
 {
 	switch (obj_type(t)) {
diff --git a/include/types/obj_type.h b/include/types/obj_type.h
index e141d69..9410718 100644
--- a/include/types/obj_type.h
+++ b/include/types/obj_type.h
@@ -41,6 +41,7 @@ enum obj_type {
 	OBJ_TYPE_CONN, /* object is a struct connection */
 	OBJ_TYPE_SRVRQ,/* object is a struct dns_srvrq */
 	OBJ_TYPE_CS,   /* object is a struct conn_stream */
+	OBJ_TYPE_STREAM,   /* object is a struct stream */
 	OBJ_TYPE_ENTRIES   /* last one : number of entries */
 } __attribute__((packed)) ;
 
diff --git a/include/types/stream.h b/include/types/stream.h
index 93a39a3..b6a3e84 100644
--- a/include/types/stream.h
+++ b/include/types/stream.h
@@ -151,7 +151,9 @@ struct stream {
 		struct stktable *table;
 	} store[8]; /* tracked stickiness values to store */
 	int store_count;
-	/* 4 unused bytes here */
+
+	enum obj_type obj_type; /* object type == OBJ_TYPE_STREAM */
+	/* 3 unused bytes here */
 
 	struct stkctr stkctr[MAX_SESS_STKCTR];  /* content-aware stick counters */
 
-- 
2.7.4

From bd4bf0c60a8b78555c050b4ffbd399a239de8be6 Mon Sep 17 00:00:00 2001
From: Baptiste Assmann 
Date: Mon, 21 Jan 2019 08:34:50 +0100
Subject: [PATCH 4/5] MINOR: action: new '(http-request|tcp-request content)
 do-resolve' action

The 'do-resolve' action is an http-request or tcp-request content action
which allows to run DNS resolution at run time in HAProxy.
The name to be resolved can be picked up in the request sent by the
client and the result of the resolution is stored in a variable.
The time the resolution is being performed, the request is on pause.
If the resolution can't provide a suitable result, then the variable
will be empty. It's up to the admin to take decisions based on this
statement (return 503 to prevent loops).

Read carefully the documentation concerning this feature, to ensure your
setup is secure and safe to be used in production.

This patch creates a global counter to track various errors reported by
the action 'do-resolve'.
---
 doc/configuration.txt  |  57 ++
 include/proto/action.h |   3 +
 include/proto/dns.h|   4 +
 include/types/action.h |   7 ++
 include/types/stats.h  |   1 +
 include/types/stream.h |   9 ++
 src/action.c   |  34 ++
 src/dns.c  | 301 +
 src/proto_http.c   |   1 +
 src/stats.c|   3 +
 src/stream.c   |  11 ++
 11 files changed, 431 insertions(+)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index 357a67e..a36103a 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -4186,6 +4186,60 @@ http-request deny [deny_status ] [ { if | unless }  ]
   those that can be overridden by the "errorfile" directive.
   No further "http-request" rules are evaluated.
 
+http-request do-resolve(,,[ipv4,ipv6])  :
+
+  This action performs a DNS resolution of the output of  and stores
+  the result in the variable . It uses the DNS resolvers section
+  pointed by .
+  It is possible to choose a resolution preference using the optional
+  arguments 'ipv4' or 'ipv6'.
+  When performing the DNS resolution, the client side connection is on
+  pause waiting till the end of the resolution.
+  If an IP address can be found, it is stored into . If any kind of
+  error occurs, then  is not set.
+  One can use this action to discover a server IP address at run time and
+  based on information found in the request (IE a Host header).
+  If this action is used to find the server's IP address (using the
+  "set-dst" action),