Re: [ANNOUNCE] haproxy-1.9.6

[Aleksandar Lazic]

Am 09.04.2019 um 18:06 schrieb Emmanuel Hocdet:
> 
>> Le 9 avr. 2019 à 09:58, Aleksandar Lazic > > a écrit :
>>
>> Hi Manu.
>>
>> Am 05.04.2019 um 12:36 schrieb Emmanuel Hocdet:
>>> Hi Aleks,
>>>
>>> Thanks you to have integrate BoringSSL!
>>>
 Le 29 mars 2019 à 14:51, Aleksandar Lazic >>> 
 > a écrit :

 Am 29.03.2019 um 14:25 schrieb Willy Tarreau:
> Hi Aleks,
>
> On Fri, Mar 29, 2019 at 02:09:28PM +0100, Aleksandar Lazic wrote:
>> With openssl are 2 tests failed but I'm not sure because of the setup or 
>> a
>> bug.
>> https://gitlab.com/aleks001/haproxy19-centos/-/jobs/186769272
>
> Thank you for the quick feedback. I remember about the first one being
> caused by a mismatch in the exact computed response size due to headers
> encoding causing some very faint variations, though I have no idea why
> I don't see it here, since I should as well, I'll have to check my regtest
> script. For the second one, it looks related to the reactivation of the
> HEAD method in this test which was broken in previous vtest. But I'm
> seeing in your trace that you're taking it from the git repo so that
> can't be that. I need to dig as well.
>
>> With boringssl are 3 tests failed but I'm not sure because of the setup 
>> or a
>> bug.
>> https://gitlab.com/aleks001/haproxy-19-boringssl/-/jobs/186780822
>
> For this one I don't know, curl reports some unexpected EOFs. I don't
> see why it would fail only with boringssl. Did it use to work in the
> past ?

 No. The tests with Boringssl always failed in one or another way.

>>>
>>> It’s strange. After quick test, it works in my environnements.
>>> I need to comment "${no-htx} option http-use-htx »
>>> to test with varnishtest.
>>
>> I use `make reg-tests -- --use-htx` to test with htx.
>>
>> https://gitlab.com/aleks001/haproxy-19-boringssl/blob/master/Dockerfile#L65
>>
>> Do you tested with or without htx?
>>
> 
> I use varnishtest, i will switch to vtest.
> I do again the test with and without htx, with 1.9 and 2.0dev, and it’s ok.

Then is it a problem on the gitlab environment as I thought.

Thank you for your time and approval.

Best regards
Aleks

 https://gitlab.com/aleks001/haproxy-19-boringssl/-/jobs/157743825
 https://gitlab.com/aleks001/haproxy-19-boringssl/-/jobs/157730793

 I'm not sure if the docker setup on gitlab is the limitation or just a bug.
 Sorry to be so unspecific.

> Thanks,
> Willy

 Regards
 Aleks
>>>
>>> ++
>>> Manu
>>
>> Best regards
>> Aleks
> 

Re: The headers added by haproxy are randomly overwritten by incoming client data when http-use-htx

[Lukas Tribus]

Hello,

On Tue, 9 Apr 2019 at 16:21, Radu Carpa  wrote:
>
> Hello,
>
> We encounter a nasty bug when htx is enabled.
> Under certain conditions, the incoming client data can overwrite part of
> the buffer with data prepared for backend servers.

This is most likely:
https://github.com/haproxy/haproxy/issues/78

CC'ing Christopher.


cheers,
lukas

Re: [ANNOUNCE] haproxy-1.9.6

[Emmanuel Hocdet]

> Le 9 avr. 2019 à 09:58, Aleksandar Lazic  a écrit :
> 
> Hi Manu.
> 
> Am 05.04.2019 um 12:36 schrieb Emmanuel Hocdet:
>> Hi Aleks,
>> 
>> Thanks you to have integrate BoringSSL!
>> 
>>> Le 29 mars 2019 à 14:51, Aleksandar Lazic >> 
>>> >> a écrit :
>>> 
>>> Am 29.03.2019 um 14:25 schrieb Willy Tarreau:
 Hi Aleks,
 
 On Fri, Mar 29, 2019 at 02:09:28PM +0100, Aleksandar Lazic wrote:
> With openssl are 2 tests failed but I'm not sure because of the setup or 
> a bug.
> https://gitlab.com/aleks001/haproxy19-centos/-/jobs/186769272
 
 Thank you for the quick feedback. I remember about the first one being
 caused by a mismatch in the exact computed response size due to headers
 encoding causing some very faint variations, though I have no idea why
 I don't see it here, since I should as well, I'll have to check my regtest
 script. For the second one, it looks related to the reactivation of the
 HEAD method in this test which was broken in previous vtest. But I'm
 seeing in your trace that you're taking it from the git repo so that
 can't be that. I need to dig as well.
 
> With boringssl are 3 tests failed but I'm not sure because of the setup 
> or a
> bug.
> https://gitlab.com/aleks001/haproxy-19-boringssl/-/jobs/186780822
 
 For this one I don't know, curl reports some unexpected EOFs. I don't
 see why it would fail only with boringssl. Did it use to work in the
 past ?
>>> 
>>> No. The tests with Boringssl always failed in one or another way.
>>> 
>> 
>> It’s strange. After quick test, it works in my environnements.
>> I need to comment "${no-htx} option http-use-htx »
>> to test with varnishtest.
> 
> I use `make reg-tests -- --use-htx` to test with htx.
> 
> https://gitlab.com/aleks001/haproxy-19-boringssl/blob/master/Dockerfile#L65 
> 
> 
> Do you tested with or without htx?
> 

I use varnishtest, i will switch to vtest.
I do again the test with and without htx, with 1.9 and 2.0dev, and it’s ok.

>>> https://gitlab.com/aleks001/haproxy-19-boringssl/-/jobs/157743825 
>>> 
>>> https://gitlab.com/aleks001/haproxy-19-boringssl/-/jobs/157730793 
>>> 
>>> 
>>> I'm not sure if the docker setup on gitlab is the limitation or just a bug.
>>> Sorry to be so unspecific.
>>> 
 Thanks,
 Willy
>>> 
>>> Regards
>>> Aleks
>> 
>> ++
>> Manu
> 
> Best regards
> Aleks

Re: QAT intermittent healthcheck errors

[Emeric Brun]

Hi Marcin,

On 4/9/19 3:07 PM, Marcin Deranek wrote:
> Hi Emeric,
> 
> I have followed all instructions and I got to the point where HAProxy starts 
> and does the job using QAT (backend healthchecks work and I frontend can 
> provide content over HTTPS). The problems starts when HAProxy gets reloaded. 
> With our current configuration on reload old HAProxy processes do not exit, 
> so after reload you end up with 2 generations of HAProxy processes: before 
> reload and after reload. I tried to find out what are conditions in which 
> HAProxy processes get "stuck" and I was not able to replicate it 
> consistently. In one case it was related to amount of backend servers with 
> 'ssl' on their line, but trying to add 'ssl' to some other servers in other 
> place had no effect. Interestingly in some cases for example with simple 
> configuration (1 frontend + 1 backend) HAProxy produced errors on reload (see 
> attachment) - in those cases processes rarely got "stuck" even though errors 
> were present.
> /dev/qat_adf_ctl is group writable for the group HAProxy runs on. Any help to 
> get this fixed / resolved would be welcome.
> Regards,
> 
> Marcin Deranek

I've checked the errors.txt and all the messages were written by the engine and 
are not part of the haproxy code. I can only do supposition for now but I think 
we face a first error due to a limitation of the amount of processes trying to 
access the engine: the reload will double the number of processes trying to 
attach the engine. Perhaps this issue can be bypassed tweaking the qat 
configuration file (some advise, from intel would be wellcome).

For the old stucked processes: I think the grow of processes also triggers 
errors on already attached ones in the qat engine but currently I ignore the 
way this errors are/should be raised to the application, it appears that they 
are currently not handled and that's why processes would be stuck (sessions may 
appear still valid for haproxy so the old process continues to wait for their 
end). We expected they were raised by the openssl API but it appears to not be 
the case. We have to check if we miss to handle an error  polling events on the 
file descriptor used to communicate with engine.


So we have to dig deeper and any help from Intel's guy or Qat aware devs will 
be appreciate.

Emeric

The headers added by haproxy are randomly overwritten by incoming client data when http-use-htx

[Radu Carpa]

Hello,

We encounter a nasty bug when htx is enabled.
Under certain conditions, the incoming client data can overwrite part of 
the buffer with data prepared for backend servers.
I was able to reproduce the issue by using the attached script 
('request.py') to generate http requests.
I did a wireshark capture of the traffic originating from haproxy toward 
the backend server.
In random cases, I observe a corrupted x-forwarded-for header in the 
requests sent to the backend server.

For example :

==
POST / HTTP/1.1
content-length: 14700
accept-encoding: gzip, deflate
accept: */*
user-agent: python-requests/2.6.0 CPython/2.7.5 Linux
host: a.exemple.com
223: 4

abcdefgh< the rest of the request body>

==

I wasn't able to reproduce the issue by doing requests to a haproxy 
located on localhost.
I believe there are timing requirements to reproduce it:
- Haproxy should start processing part of the HTTP request before 
receiving the rest of the body.
- Moreover, the rest of the request body should arrive after this 
partial processing (but before sending the first request to backend 
servers).
Having some network latency between the client and haproxy helps 
reproduce the issue. In my case: ~7ms.

The issue only occurs for requests with content-length>1448.
Moreover, If content-length == 1450, only the first 2 byte of the 
x-forwarded-for header name will be corrupted with the last 2 bytes of 
the request.

I was able to reproduce the issue with all haproxy 1.9.* release 
versions, and a fairly minimal haproxy.cfg (see attached).
The bug still persists with a freshly compiled git version (head commit: 
0b4b27efde653a4b7bbafb56df3796bffa4722ae).

Please tell me if you need more details.

Sincerely,

Radu


#!/usr/bin/env python
import requests
import time
import string

data = b''
for i in range(1,50):
for j in list(string.ascii_letters) + [str(x) for x in range(0,10)]:
data+=bytes(j*i, encoding='ascii')
data = data[:14700]
print(data)

for i in range(0, 200): 
try: 
requests.post("http://10.236.100.19;, data=data, headers={'host': 'a.exemple.com'})
except Exception as e:
print(e)
print(i)
global
 user haproxy
 group haproxy
 nbproc 1
 nbthread 1

defaults
 mode http
 option forwardfor
 option http-use-htx

frontend fe_main 
bind *:80 name http
use_backend be_main 

backend be_main
server srv0 10.236.72.23:31044 weight 10

v1.9.6 socket unresponsive with high cpu usage

[William Dauchy]

Hello,

Probably a useless report as I don't have a lot information to provide,
but we faced an issue where the unix socket was unresponsive, with the
processes using all cpu (1600% with 16 nbthreads)

I only have the backtrace of the main process but lost the backtrace of
all threads (nbthread 16). I was also unable to get a response from the
socket.

(gdb) bt
#0  0x5636716d7fbe in fwrr_set_server_status_up (srv=0x5636928e6700) at 
src/lb_fwrr.c:112
#1  0x56367162f8e7 in srv_update_status (s=0x5636928e6700) at 
src/server.c:4884
#2  0x56367162f4f2 in server_recalc_eweight (sv=sv@entry=0x5636928e6700, 
must_update=must_update@entry=1) at src/server.c:1310
#3  0x56367163845d in server_warmup (t=0x56369c969d40, 
context=0x5636928e6700, state=) at src/checks.c:1492
#4  0x5636716cd802 in process_runnable_tasks () at src/task.c:437
#5  0x563671646ebf in run_poll_loop () at src/haproxy.c:2645
#6  run_thread_poll_loop (data=data@entry=0x563671fb5d20) at src/haproxy.c:2710
#7  0x5636715be2f5 in main (argc=, argv=0x7fffdaa3b7f8) at 
src/haproxy.c:3346

note: this process is based v1.9.6 until
0b4b27efde653a4b7bbafb56df3796bffa4722ae on top of it.

I will try to grab more information about it if we are seeing the issue
again.

-- 
William

Re: QAT intermittent healthcheck errors

[Marcin Deranek]

Hi Emeric,

I have followed all instructions and I got to the point where HAProxy 
starts and does the job using QAT (backend healthchecks work and I 
frontend can provide content over HTTPS). The problems starts when 
HAProxy gets reloaded. With our current configuration on reload old 
HAProxy processes do not exit, so after reload you end up with 2 
generations of HAProxy processes: before reload and after reload. I 
tried to find out what are conditions in which HAProxy processes get 
"stuck" and I was not able to replicate it consistently. In one case it 
was related to amount of backend servers with 'ssl' on their line, but 
trying to add 'ssl' to some other servers in other place had no effect. 
Interestingly in some cases for example with simple configuration (1 
frontend + 1 backend) HAProxy produced errors on reload (see attachment) 
- in those cases processes rarely got "stuck" even though errors were 
present.
/dev/qat_adf_ctl is group writable for the group HAProxy runs on. Any 
help to get this fixed / resolved would be welcome.

Regards,

Marcin Deranek

On 3/13/19 12:04 PM, Emeric Brun wrote:

Hi Marcin,

On 3/11/19 4:27 PM, Marcin Deranek wrote:

On 3/11/19 11:51 AM, Emeric Brun wrote:


Mode async is enabled on both sides, server and frontend side.

But on server side, haproxy is using session resuming, so there is a new key 
computation (full handshake with RSA/DSA computation) only every 5 minutes 
(openssl default value).

You can force to recompute each time setting "no-ssl-reuse" on server line, but 
it will add a heavy load for ssl computation on the server.


Indeed, setting no-ssl-reuse makes use of QAT for healthchecks.
Looks like finally we are ready for QAT testing.
Thank you Emeric.
Regards,

Marcin Deranek




I've just re-check and i think you should also enable the 'PKEY_CRYPTO' algo to 
the engine

ssl-engine qat algo RSA,DSA,EC,DH,PKEY_CRYPTO

It will enable rhe offloading of the TLS1-PRF you can see there:

# /opt/booking-openssl/bin/openssl engine -c qat
(qat) Reference implementation of QAT crypto engine
  [RSA, DSA, DH, AES-128-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, 
AES-256-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA256, TLS1-PRF]

R,
Emeric

2019-04-09T14:22:45.523342+02:00 externallb hapee-lb[60816]: [NOTICE] 
098/142244 (60816) : New worker #1 (61249) forked
2019-04-09T14:22:45.523368+02:00 externallb hapee-lb[60816]: [NOTICE] 
098/142244 (60816) : New worker #2 (61250) forked
2019-04-09T14:22:45.523393+02:00 externallb hapee-lb[60816]: [NOTICE] 
098/142244 (60816) : New worker #3 (61251) forked
2019-04-09T14:22:45.523418+02:00 externallb hapee-lb[60816]: [NOTICE] 
098/142244 (60816) : New worker #4 (61252) forked
2019-04-09T14:22:45.523444+02:00 externallb hapee-lb[60816]: [NOTICE] 
098/142244 (60816) : New worker #5 (61253) forked
2019-04-09T14:22:45.523469+02:00 externallb hapee-lb[60816]: [NOTICE] 
098/142244 (60816) : New worker #6 (61255) forked
2019-04-09T14:22:45.523493+02:00 externallb hapee-lb[60816]: [NOTICE] 
098/142244 (60816) : New worker #7 (61258) forked
2019-04-09T14:22:45.523518+02:00 externallb hapee-lb[60816]: [NOTICE] 
098/142244 (60816) : New worker #8 (61259) forked
2019-04-09T14:22:45.523548+02:00 externallb hapee-lb[60816]: [NOTICE] 
098/142244 (60816) : New worker #9 (61261) forked
2019-04-09T14:22:45.523596+02:00 externallb hapee-lb[60816]: [error] 
cpaCyStopInstance() - : Can not get instance info
2019-04-09T14:22:45.523623+02:00 externallb hapee-lb[60816]: [error] 
SalCtrl_GetEnabledServices() - : Failed to get enabled services from ADF
2019-04-09T14:22:45.523649+02:00 externallb hapee-lb[60816]: [error] 
SalCtrl_ServiceEventHandler() - : Failed to get enabled services
2019-04-09T14:22:45.523674+02:00 externallb hapee-lb[60816]: [error] 
SalCtrl_GetEnabledServices() - : Failed to get enabled services from ADF
2019-04-09T14:22:45.523699+02:00 externallb hapee-lb[60816]: [error] 
SalCtrl_ServiceEventHandler() - : Failed to get enabled services
2019-04-09T14:22:45.523724+02:00 externallb hapee-lb[60816]: [error] 
SalCtrl_GetEnabledServices() - : Failed to get enabled services from ADF
2019-04-09T14:22:45.523749+02:00 externallb hapee-lb[60816]: [error] 
SalCtrl_ServiceEventHandler() - : Failed to get enabled services
2019-04-09T14:22:45.523774+02:00 externallb hapee-lb[60816]: [error] 
SalCtrl_GetEnabledServices() - : Failed to get enabled services from ADF
2019-04-09T14:22:45.523799+02:00 externallb hapee-lb[60816]: [error] 
SalCtrl_ServiceEventHandler() - : Failed to get enabled services
2019-04-09T14:22:45.523823+02:00 externallb hapee-lb[60816]: [error] 
SalCtrl_GetEnabledServices() - : Failed to get enabled services from ADF
2019-04-09T14:22:45.523848+02:00 externallb hapee-lb[60816]: [error] 
SalCtrl_ServiceEventHandler() - : Failed to get enabled services
2019-04-09T14:22:45.523874+02:00 externallb hapee-lb[60816]: [error] 
SalCtrl_GetEnabledServices() - : Failed to get enabled services from ADF
2019-04-09T14:22:45.523899+02:00 

Purchase order

[Mrs Wang]

Hello Dear,

We have interest in your products.
We will be glad if you can send us catalog and best prices to help us make a 
quick order.
Your early reply will be appreciated.

Thank you,
E-Mail: chiwan...@outlook.com
Business Manager
Mrs Wang

Re: haproxy1.9, SPOA: too many open files

[Christopher Faulet]

Le 09/04/2019 à 07:43, Kevin Zhu a écrit :
Use haproxy-1.9 and 2.0, SPOA will occure error "too many open file" 
when benchmark testing, spoa_example have this error too, even enable 
the async and pipelining.

But haproxy 1.8 have no this kind error.



Hi Kevin,

I need more information before investigating. Could you provide more 
information on the way you perform your benchmark. I also need to take a 
look on your HAProxy and spoe configurations. And finally, let me known 
how you start the agent.


Regards,
--
Christopher Faulet

Re: [ANNOUNCE] haproxy-1.9.6

[Aleksandar Lazic]

Hi Manu.

Am 05.04.2019 um 12:36 schrieb Emmanuel Hocdet:
> Hi Aleks,
> 
> Thanks you to have integrate BoringSSL!
> 
>> Le 29 mars 2019 à 14:51, Aleksandar Lazic > > a écrit :
>>
>> Am 29.03.2019 um 14:25 schrieb Willy Tarreau:
>>> Hi Aleks,
>>>
>>> On Fri, Mar 29, 2019 at 02:09:28PM +0100, Aleksandar Lazic wrote:
 With openssl are 2 tests failed but I'm not sure because of the setup or a 
 bug.
 https://gitlab.com/aleks001/haproxy19-centos/-/jobs/186769272
>>>
>>> Thank you for the quick feedback. I remember about the first one being
>>> caused by a mismatch in the exact computed response size due to headers
>>> encoding causing some very faint variations, though I have no idea why
>>> I don't see it here, since I should as well, I'll have to check my regtest
>>> script. For the second one, it looks related to the reactivation of the
>>> HEAD method in this test which was broken in previous vtest. But I'm
>>> seeing in your trace that you're taking it from the git repo so that
>>> can't be that. I need to dig as well.
>>>
 With boringssl are 3 tests failed but I'm not sure because of the setup or 
 a
 bug.
 https://gitlab.com/aleks001/haproxy-19-boringssl/-/jobs/186780822
>>>
>>> For this one I don't know, curl reports some unexpected EOFs. I don't
>>> see why it would fail only with boringssl. Did it use to work in the
>>> past ?
>>
>> No. The tests with Boringssl always failed in one or another way.
>>
> 
> It’s strange. After quick test, it works in my environnements.
> I need to comment "${no-htx} option http-use-htx »
> to test with varnishtest.

I use `make reg-tests -- --use-htx` to test with htx.

https://gitlab.com/aleks001/haproxy-19-boringssl/blob/master/Dockerfile#L65

Do you tested with or without htx?

>> https://gitlab.com/aleks001/haproxy-19-boringssl/-/jobs/157743825
>> https://gitlab.com/aleks001/haproxy-19-boringssl/-/jobs/157730793
>>
>> I'm not sure if the docker setup on gitlab is the limitation or just a bug.
>> Sorry to be so unspecific.
>>
>>> Thanks,
>>> Willy
>>
>> Regards
>> Aleks
> 
> ++
> Manu

Best regards
Aleks