Re: QAT intermittent healthcheck errors

2019-05-13 Thread Marcin Deranek
Hi Emeric, On 5/13/19 11:06 AM, Emeric Brun wrote: Just to known that I'm waiting for the feedback of intel's team and I will receive QAT 1.7 compliant hardware soon to make some tests here. Thank you for an update. Regards, Marcin Deranek

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-13 Thread Emeric Brun
Hi Marcin, > > Thank you Marcin, It shows that haproxy is waiting for an event on all those > fds because a crypto jobs were launched on the engine > and we can't free the session until the end of this job (it would result in a > segfault). > > So the processes are stucked, unable to free the

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-07 Thread Emeric Brun
On 5/7/19 3:35 PM, Marcin Deranek wrote: > Hi Emeric, > > On 5/7/19 1:53 PM, Emeric Brun wrote: >> On 5/7/19 1:24 PM, Marcin Deranek wrote: >>> Hi Emeric, >>> >>> On 5/7/19 11:44 AM, Emeric Brun wrote: Hi Marcin,>> As I use HAProxy 1.8 I had to adjust the patch (see attachment for e

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-07 Thread Marcin Deranek
Hi Emeric, On 5/7/19 1:53 PM, Emeric Brun wrote: On 5/7/19 1:24 PM, Marcin Deranek wrote: Hi Emeric, On 5/7/19 11:44 AM, Emeric Brun wrote: Hi Marcin,>> As I use HAProxy 1.8 I had to adjust the patch (see attachment for end result). Unfortunately after applying the patch there is no chan

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-07 Thread Emeric Brun
On 5/7/19 1:24 PM, Marcin Deranek wrote: > Hi Emeric, > > On 5/7/19 11:44 AM, Emeric Brun wrote: >> Hi Marcin,>> As I use HAProxy 1.8 I had to adjust the patch (see >> attachment for end result). Unfortunately after applying the patch there is >> no change in behavior: we still leak /dev/usd

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-07 Thread Marcin Deranek
Hi Emeric, On 5/7/19 11:44 AM, Emeric Brun wrote: Hi Marcin,>> As I use HAProxy 1.8 I had to adjust the patch (see attachment for end result). Unfortunately after applying the patch there is no change in behavior: we still leak /dev/usdm_drv descriptors and have "stuck" HAProxy instances a

Re: QAT intermittent healthcheck errors

2019-05-07 Thread Marcin Deranek
On 5/7/19 11:44 AM, Emeric Brun wrote: Could you perform a test recompiling the usdm_drv and the engine with this patch, it applies on QAT 1.7 but I've no hardware to test this version here. It should fix the fd leak. Will do and report back. Marcin Deranek

Re: QAT intermittent healthcheck errors

2019-05-07 Thread Emeric Brun
Hi Marcin,>> As I use HAProxy 1.8 I had to adjust the patch (see attachment for end result). Unfortunately after applying the patch there is no change in behavior: we still leak /dev/usdm_drv descriptors and have "stuck" HAProxy instances after reload.. >>> Regards, >> >> Could you perform

Re: QAT intermittent healthcheck errors

2019-05-06 Thread Emeric Brun
Hi Marcin, On 5/6/19 3:31 PM, Emeric Brun wrote: > Hi Marcin, > > On 5/6/19 3:15 PM, Marcin Deranek wrote: >> Hi Emeric, >> >> On 5/3/19 5:54 PM, Emeric Brun wrote: >>> Hi Marcin, >>> >>> On 5/3/19 4:56 PM, Marcin Deranek wrote: Hi Emeric, On 5/3/19 4:50 PM, Emeric Brun wrote:

Re: QAT intermittent healthcheck errors

2019-05-06 Thread Emeric Brun
Hi Marcin, On 5/6/19 3:15 PM, Marcin Deranek wrote: > Hi Emeric, > > On 5/3/19 5:54 PM, Emeric Brun wrote: >> Hi Marcin, >> >> On 5/3/19 4:56 PM, Marcin Deranek wrote: >>> Hi Emeric, >>> >>> On 5/3/19 4:50 PM, Emeric Brun wrote: >>> I've a testing platform here but I don't use the usdm_drv b

Re: QAT intermittent healthcheck errors

2019-05-06 Thread Marcin Deranek
Hi Emeric, On 5/3/19 5:54 PM, Emeric Brun wrote: Hi Marcin, On 5/3/19 4:56 PM, Marcin Deranek wrote: Hi Emeric, On 5/3/19 4:50 PM, Emeric Brun wrote: I've a testing platform here but I don't use the usdm_drv but the qat_contig_mem and I don't reproduce this issue (I'm using QAT 1.5, as the

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-03 Thread Emeric Brun
Hi Marcin, On 5/3/19 4:56 PM, Marcin Deranek wrote: > Hi Emeric, > > On 5/3/19 4:50 PM, Emeric Brun wrote: > >> I've a testing platform here but I don't use the usdm_drv but the >> qat_contig_mem and I don't reproduce this issue (I'm using QAT 1.5, as the >> doc says to use with my chip) . >

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-03 Thread Marcin Deranek
Hi Emeric, On 5/3/19 4:50 PM, Emeric Brun wrote: I've a testing platform here but I don't use the usdm_drv but the qat_contig_mem and I don't reproduce this issue (I'm using QAT 1.5, as the doc says to use with my chip) . I see. I use qat 1.7 and qat-engine 0.5.40. Anyway, could you re-co

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-03 Thread Emeric Brun
Hi Marcin, Good so we progress! I've a testing platform here but I don't use the usdm_drv but the qat_contig_mem and I don't reproduce this issue (I'm using QAT 1.5, as the doc says to use with my chip) . Anyway, could you re-compile a haproxy's binary if I provide you a testing patch? The i

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-03 Thread Marcin Deranek
Hi Emeric, It looks like on every reload master leaks /dev/usdm_drv device: # systemctl restart haproxy.service # ls -la /proc/$(cat haproxy.pid)/fd|fgrep dev lr-x-- 1 root root 64 May 3 15:40 0 -> /dev/null lrwx-- 1 root root 64 May 3 15:40 7 -> /dev/usdm_drv # systemctl reload hapro

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-03 Thread Emeric Brun
Hi Marcin, On 4/29/19 6:41 PM, Marcin Deranek wrote: > Hi Emeric, > > On 4/29/19 3:42 PM, Emeric Brun wrote: >> Hi Marcin, >> >>> I've also a contact at intel who told me to try this option on the qat engine: > --disable-qat_auto_engine_init_on_fork/--enable-qat_auto_engine_in

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-29 Thread Marcin Deranek
Hi Emeric, On 4/29/19 3:42 PM, Emeric Brun wrote: Hi Marcin, I've also a contact at intel who told me to try this option on the qat engine: --disable-qat_auto_engine_init_on_fork/--enable-qat_auto_engine_init_on_fork Disable/Enable the engine from being initialized automatically foll

Re: QAT intermittent healthcheck errors

2019-04-29 Thread Emeric Brun
Hi Marcin, > >> I've also a contact at intel who told me to try this option on the qat >> engine: >> >>> --disable-qat_auto_engine_init_on_fork/--enable-qat_auto_engine_init_on_fork >>> Disable/Enable the engine from being initialized automatically >>> following a >>> fork operation.

Re: QAT intermittent healthcheck errors

2019-04-29 Thread Marcin Deranek
Hi Emeric, On 4/29/19 2:47 PM, Emeric Brun wrote: Hi Marcin, On 4/19/19 3:26 PM, Marcin Deranek wrote: Hi Emeric, On 4/18/19 4:35 PM, Emeric Brun wrote: An other interesting trace would be to perform a "show sess" command on a stucked process through the master cli. And also the "show fd"

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-29 Thread Emeric Brun
Hi Marcin, On 4/19/19 3:26 PM, Marcin Deranek wrote: > Hi Emeric, > > On 4/18/19 4:35 PM, Emeric Brun wrote: >>> An other interesting trace would be to perform a "show sess" command on a >>> stucked process through the master cli. >> >> And also the "show fd" > > Here it is: > > show proc > # 

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-19 Thread Marcin Deranek
Hi Emeric, On 4/18/19 4:35 PM, Emeric Brun wrote: An other interesting trace would be to perform a "show sess" command on a stucked process through the master cli. And also the "show fd" Here it is: show proc # 13409 master 0 1

Re: QAT intermittent healthcheck errors

2019-04-19 Thread Marcin Deranek
On 4/18/19 11:06 AM, Emeric Brun wrote: I think you can do that this way: Remove the option httchk (or prefix it by "no": "no option httchk " if it is configured into the defaults section and add the following 2 lines: option tcp-check tcp-check connect This shouldn't perform the handshake b

Re: QAT intermittent healthcheck errors

2019-04-18 Thread Emeric Brun
On 4/18/19 11:06 AM, Emeric Brun wrote: > Hi Marcin, > > On 4/12/19 6:10 PM, Marcin Deranek wrote: >> Hi Emeric, >> >> On 4/12/19 5:26 PM, Emeric Brun wrote: >> >>> Do you have ssl enabled on the server side? >> >> Yes, ssl is on frontend and backend with ssl checks enabled. >> >>> If it is the ca

Re: QAT intermittent healthcheck errors

2019-04-18 Thread Emeric Brun
Hi Marcin, On 4/12/19 6:10 PM, Marcin Deranek wrote: > Hi Emeric, > > On 4/12/19 5:26 PM, Emeric Brun wrote: > >> Do you have ssl enabled on the server side? > > Yes, ssl is on frontend and backend with ssl checks enabled. > >> If it is the case could replace health check with a simple tcp che

Re: QAT intermittent healthcheck errors

2019-04-12 Thread Marcin Deranek
Hi Emeric, On 4/12/19 5:26 PM, Emeric Brun wrote: Do you have ssl enabled on the server side? Yes, ssl is on frontend and backend with ssl checks enabled. If it is the case could replace health check with a simple tcp check (without ssl)? What I noticed before that if I (re)start HAProxy

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-12 Thread Emeric Brun
Hi Marcin, Do you have ssl enabled on the server side? If it is the case could replace health check with a simple tcp check (without ssl)? Regarding the show info/lsoff it seems there is no more sessions on client side but remaining ssl jobs (CurrSslConns) and I supsect the health checks to m

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-12 Thread Marcin Deranek
Hi Emeric, On 4/10/19 2:20 PM, Emeric Brun wrote: On 4/10/19 1:02 PM, Marcin Deranek wrote: Hi Emeric, Our process limit in QAT configuration is quite high (128) and I was able to run 100+ openssl processes without a problem. According to Joel from Intel problem is in cleanup code - presuma

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-10 Thread Emeric Brun
Hi Marcin, > You can also use the 'master CLI' using '-S' and you could check if it > remains sessions on those older processes (doc is available in management.txt) Here the doc: https://cbonte.github.io/haproxy-dconv/1.9/management.html#9.4 Emeric

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-10 Thread Emeric Brun
Hi Marcin, On 4/10/19 1:02 PM, Marcin Deranek wrote: > Hi Emeric, > > Our process limit in QAT configuration is quite high (128) and I was able to > run 100+ openssl processes without a problem. According to Joel from Intel > problem is in cleanup code - presumably when HAProxy exits and frees

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-10 Thread Marcin Deranek
Hi Emeric, Our process limit in QAT configuration is quite high (128) and I was able to run 100+ openssl processes without a problem. According to Joel from Intel problem is in cleanup code - presumably when HAProxy exits and frees up QAT resources. Will try to see if I can get more debug inf

Re: QAT intermittent healthcheck errors

2019-04-09 Thread Emeric Brun
Hi Marcin, On 4/9/19 3:07 PM, Marcin Deranek wrote: > Hi Emeric, > > I have followed all instructions and I got to the point where HAProxy starts > and does the job using QAT (backend healthchecks work and I frontend can > provide content over HTTPS). The problems starts when HAProxy gets reloa

Re: QAT intermittent healthcheck errors

2019-04-09 Thread Marcin Deranek
Hi Emeric, I have followed all instructions and I got to the point where HAProxy starts and does the job using QAT (backend healthchecks work and I frontend can provide content over HTTPS). The problems starts when HAProxy gets reloaded. With our current configuration on reload old HAProxy pr

Re: [External] Re: QAT intermittent healthcheck errors

2019-03-13 Thread Emeric Brun
Hi Marcin, On 3/11/19 4:27 PM, Marcin Deranek wrote: > On 3/11/19 11:51 AM, Emeric Brun wrote: > >> Mode async is enabled on both sides, server and frontend side. >> >> But on server side, haproxy is using session resuming, so there is a new key >> computation (full handshake with RSA/DSA comput

Re: [External] Re: QAT intermittent healthcheck errors

2019-03-11 Thread Marcin Deranek
Hi Emeric, On 3/11/19 2:48 PM, Emeric Brun wrote: Once again, you could add the "no-ssl-reuse" statement if you want to check if QAT offloads the backend side, but it is clearly not an optimal option for production because it will generate an heavy load on your servers and force them to recom

Re: [External] Re: QAT intermittent healthcheck errors

2019-03-11 Thread Marcin Deranek
On 3/11/19 11:51 AM, Emeric Brun wrote: Mode async is enabled on both sides, server and frontend side. But on server side, haproxy is using session resuming, so there is a new key computation (full handshake with RSA/DSA computation) only every 5 minutes (openssl default value). You can forc

Re: QAT intermittent healthcheck errors

2019-03-11 Thread Emeric Brun
On 3/11/19 11:51 AM, Emeric Brun wrote: > On 3/11/19 11:06 AM, Marcin Deranek wrote: >> Hi Emeric, >> >> On 3/8/19 11:24 AM, Emeric Brun wrote: >>> Are you sure that servers won't use ECDSA certificates? Do you check that >>> conn are successful forcing 'ECDHE-RSA-AES256-GCM-SHA384' >> >> Backend

Re: QAT intermittent healthcheck errors

2019-03-11 Thread Emeric Brun
On 3/11/19 11:06 AM, Marcin Deranek wrote: > Hi Emeric, > > On 3/8/19 11:24 AM, Emeric Brun wrote: >> Are you sure that servers won't use ECDSA certificates? Do you check that >> conn are successful forcing 'ECDHE-RSA-AES256-GCM-SHA384' > > Backend servers only support TLS 1.2 and RSA certificat

Re: QAT intermittent healthcheck errors

2019-03-11 Thread Marcin Deranek
Hi Emeric, On 3/8/19 11:24 AM, Emeric Brun wrote: Are you sure that servers won't use ECDSA certificates? Do you check that conn are successful forcing 'ECDHE-RSA-AES256-GCM-SHA384' Backend servers only support TLS 1.2 and RSA certificates. Could you check algo supported by QAT doing this ?

Re: [External] Re: QAT intermittent healthcheck errors

2019-03-11 Thread Marcin Deranek
Hi Emeric, On 3/8/19 4:43 PM, Emeric Brun wrote: I've just realized that if your server are TLSv1.3 ssl-default-server-ciphers won't force anything (see ssl-default-server-ciphersuites documentation) Backend servers are 'only' TLS 1.2, so it should have desired effect. Will test suggested co

Re: QAT intermittent healthcheck errors

2019-03-08 Thread Emeric Brun
Hi Marcin, On 3/7/19 6:43 PM, Marcin Deranek wrote: > Hi, > > On 3/6/19 6:36 PM, Emeric Brun wrote: >> According to the documentation: >> >> ssl-mode-async >>    Adds SSL_MODE_ASYNC mode to the SSL context. This enables asynchronous TLS >>    I/O operations if asynchronous capable SSL engines are

Re: QAT intermittent healthcheck errors

2019-03-08 Thread Emeric Brun
Hi Marcin, On 3/7/19 6:43 PM, Marcin Deranek wrote: > Hi, > > On 3/6/19 6:36 PM, Emeric Brun wrote: >> According to the documentation: >> >> ssl-mode-async >>    Adds SSL_MODE_ASYNC mode to the SSL context. This enables asynchronous TLS >>    I/O operations if asynchronous capable SSL engines a

Re: QAT intermittent healthcheck errors

2019-03-07 Thread Marcin Deranek
Hi, On 3/6/19 6:36 PM, Emeric Brun wrote: According to the documentation: ssl-mode-async Adds SSL_MODE_ASYNC mode to the SSL context. This enables asynchronous TLS I/O operations if asynchronous capable SSL engines are used. The current implementation supports a maximum of 32 engines.

Re: QAT intermittent healthcheck errors

2019-03-06 Thread Marcin Deranek
Hi, On 3/6/19 6:36 PM, Emeric Brun wrote: To avoid this issue, you should ensure to use QAT only for the asymmetric computing algorithm (such as RSA DSA ECDSA). and not for ciphering ones (AES and everything else ...) The ssl engine statement allow you to filter such algos: ssl-engine [algo

Re: QAT intermittent healthcheck errors

2019-03-06 Thread Emeric Brun
Hi Marcin, On 3/6/19 3:23 PM, Marcin Deranek wrote: > Hi, > > In a process of evaluating performance of Intel Quick Assist Technology in > conjunction with HAProxy software I acquired Intel C62x Chipset card for > testing. I configured QAT engine in the following manner: > > * /etc/qat/c6xx_de

QAT intermittent healthcheck errors

2019-03-06 Thread Marcin Deranek
Hi, In a process of evaluating performance of Intel Quick Assist Technology in conjunction with HAProxy software I acquired Intel C62x Chipset card for testing. I configured QAT engine in the following manner: * /etc/qat/c6xx_dev[012].conf [GENERAL] ServicesEnabled = cy ConfigVersion = 2 CyN