vtest update
Hello ML, We have recently modified some reg tests which require an update for vtest, especially these ones: reg-tests/peers/s_basic_sync.vtc (added) reg-tests/peers/s_tls_basic_sync.vtc (added) reg-tests/http-messaging/h0.vtc(modified) So, if some reg tests fail do not hesitate to update your vtest program. Regards, Fred.
Re: QAT intermittent healthcheck errors
Hi, On 3/6/19 6:36 PM, Emeric Brun wrote: To avoid this issue, you should ensure to use QAT only for the asymmetric computing algorithm (such as RSA DSA ECDSA). and not for ciphering ones (AES and everything else ...) The ssl engine statement allow you to filter such algos: ssl-engine [algo ] I'm pretty sure I tried this, but I will try to re-test again with eg. RSA specified and see if that makes any difference. Regards, Marcin Deranek
Re: QAT intermittent healthcheck errors
Hi Marcin, On 3/6/19 3:23 PM, Marcin Deranek wrote: > Hi, > > In a process of evaluating performance of Intel Quick Assist Technology in > conjunction with HAProxy software I acquired Intel C62x Chipset card for > testing. I configured QAT engine in the following manner: > > * /etc/qat/c6xx_dev[012].conf > > [GENERAL] > ServicesEnabled = cy > ConfigVersion = 2 > CyNumConcurrentSymRequests = 512 > CyNumConcurrentAsymRequests = 64 > statsGeneral = 1 > statsDh = 1 > statsDrbg = 1 > statsDsa = 1 > statsEcc = 1 > statsKeyGen = 1 > statsDc = 1 > statsLn = 1 > statsPrime = 1 > statsRsa = 1 > statsSym = 1 > KptEnabled = 0 > StorageEnabled = 0 > PkeServiceDisabled = 0 > DcIntermediateBufferSizeInKB = 64 > > [KERNEL] > NumberCyInstances = 0 > NumberDcInstances = 0 > > [SHIM] > NumberCyInstances = 1 > NumberDcInstances = 0 > NumProcesses = 16 > LimitDevAccess = 0 > > Cy0Name = "UserCY0" > Cy0IsPolled = 1 > Cy0CoreAffinity = 0 > > OpenSSL produces good results without warnings / errors: > > * No QAT involved > > $ openssl speed -elapsed rsa2048 > You have chosen to measure elapsed time instead of user CPU time. > Doing 2048 bits private rsa's for 10s: 10858 2048 bits private RSA's in 10.00s > Doing 2048 bits public rsa's for 10s: 361207 2048 bits public RSA's in 10.00s > OpenSSL 1.1.1a FIPS 20 Nov 2018 > built on: Tue Jan 22 20:43:41 2019 UTC > options:bn(64,64) md2(char) rc4(16x,int) des(int) aes(partial) idea(int) > blowfish(ptr) > compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -g -pipe > -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong > --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic > -Wa,--noexecstack -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC > -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT > -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM > -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM > -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM > -DPOLY1305_ASM -DZLIB -DNDEBUG -DPURIFY -DDEVRANDOM="\"/dev/urandom\"" > -DSYSTEM_CIPHERS_FILE="/opt/openssl/etc/crypto-policies/back-ends/openssl.config" > sign verify sign/s verify/s > rsa 2048 bits 0.000921s 0.28s 1085.8 36120.7 > > * QAT enabled > > $ openssl speed -elapsed -engine qat -async_jobs 32 rsa2048 > engine "qat" set. > You have chosen to measure elapsed time instead of user CPU time. > Doing 2048 bits private rsa's for 10s: 205425 2048 bits private RSA's in > 10.00s > Doing 2048 bits public rsa's for 10s: 2150270 2048 bits public RSA's in 10.00s > OpenSSL 1.1.1a FIPS 20 Nov 2018 > built on: Tue Jan 22 20:43:41 2019 UTC > options:bn(64,64) md2(char) rc4(16x,int) des(int) aes(partial) idea(int) > blowfish(ptr) > compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -g -pipe > -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong > --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic > -Wa,--noexecstack -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC > -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT > -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM > -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM > -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM > -DPOLY1305_ASM -DZLIB -DNDEBUG -DPURIFY -DDEVRANDOM="\"/dev/urandom\"" > -DSYSTEM_CIPHERS_FILE="/opt/openssl/etc/crypto-policies/back-ends/openssl.config" > sign verify sign/s verify/s > rsa 2048 bits 0.49s 0.05s 20542.5 215027.0 > > So far so good. Unfortunately HAProxy 1.8 iwth QAT engine enabled > periodically fail with SSL checks of backend servers. The simplest > configuration I could get to reproduce it: > > * /etc/haproxy/haproxy.cfg > > global > user lbengine > group lbengine > daemon > ssl-mode-async > ssl-engine qat > ssl-server-verify none > stats socket /run/lb_engine/process-1.sock user lbengine group > lbengine mode 660 level admin expose-fd listeners process 1 > > defaults > mode http > timeout check 5s > timeout connect 4s > > backend pool_all > default-server inter 5s > > server server1 ip1:443 check ssl > server server2 ip2:443 check ssl > ... > server serverN ipN:443 check ssl > > Without QAT enabled everything works just fine - healthchecks do not flap. > With QAT engine enabled random server healtchecks flap: they fail and then > shortly after they recover eg. > > 2019-03-06T15:06:22+01:00 localhost hapee-lb[1832]: Server pool_all/server1 > is DOWN, reason: Layer6 timeout, check duration: 4000ms. 110 active and 0 > backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. > 2019-03-06T15:06:32+01:00 localhost hapee-lb[1832]: Server pool_all/server1 > is UP, reason: Layer6 check passed, check duration: 13ms. 117 active and 0 > backup servers
QAT intermittent healthcheck errors
Hi, In a process of evaluating performance of Intel Quick Assist Technology in conjunction with HAProxy software I acquired Intel C62x Chipset card for testing. I configured QAT engine in the following manner: * /etc/qat/c6xx_dev[012].conf [GENERAL] ServicesEnabled = cy ConfigVersion = 2 CyNumConcurrentSymRequests = 512 CyNumConcurrentAsymRequests = 64 statsGeneral = 1 statsDh = 1 statsDrbg = 1 statsDsa = 1 statsEcc = 1 statsKeyGen = 1 statsDc = 1 statsLn = 1 statsPrime = 1 statsRsa = 1 statsSym = 1 KptEnabled = 0 StorageEnabled = 0 PkeServiceDisabled = 0 DcIntermediateBufferSizeInKB = 64 [KERNEL] NumberCyInstances = 0 NumberDcInstances = 0 [SHIM] NumberCyInstances = 1 NumberDcInstances = 0 NumProcesses = 16 LimitDevAccess = 0 Cy0Name = "UserCY0" Cy0IsPolled = 1 Cy0CoreAffinity = 0 OpenSSL produces good results without warnings / errors: * No QAT involved $ openssl speed -elapsed rsa2048 You have chosen to measure elapsed time instead of user CPU time. Doing 2048 bits private rsa's for 10s: 10858 2048 bits private RSA's in 10.00s Doing 2048 bits public rsa's for 10s: 361207 2048 bits public RSA's in 10.00s OpenSSL 1.1.1a FIPS 20 Nov 2018 built on: Tue Jan 22 20:43:41 2019 UTC options:bn(64,64) md2(char) rc4(16x,int) des(int) aes(partial) idea(int) blowfish(ptr) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wa,--noexecstack -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG -DPURIFY -DDEVRANDOM="\"/dev/urandom\"" -DSYSTEM_CIPHERS_FILE="/opt/openssl/etc/crypto-policies/back-ends/openssl.config" signverifysign/s verify/s rsa 2048 bits 0.000921s 0.28s 1085.8 36120.7 * QAT enabled $ openssl speed -elapsed -engine qat -async_jobs 32 rsa2048 engine "qat" set. You have chosen to measure elapsed time instead of user CPU time. Doing 2048 bits private rsa's for 10s: 205425 2048 bits private RSA's in 10.00s Doing 2048 bits public rsa's for 10s: 2150270 2048 bits public RSA's in 10.00s OpenSSL 1.1.1a FIPS 20 Nov 2018 built on: Tue Jan 22 20:43:41 2019 UTC options:bn(64,64) md2(char) rc4(16x,int) des(int) aes(partial) idea(int) blowfish(ptr) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wa,--noexecstack -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG -DPURIFY -DDEVRANDOM="\"/dev/urandom\"" -DSYSTEM_CIPHERS_FILE="/opt/openssl/etc/crypto-policies/back-ends/openssl.config" signverifysign/s verify/s rsa 2048 bits 0.49s 0.05s 20542.5 215027.0 So far so good. Unfortunately HAProxy 1.8 iwth QAT engine enabled periodically fail with SSL checks of backend servers. The simplest configuration I could get to reproduce it: * /etc/haproxy/haproxy.cfg global user lbengine group lbengine daemon ssl-mode-async ssl-engine qat ssl-server-verify none stats socket /run/lb_engine/process-1.sock user lbengine group lbengine mode 660 level admin expose-fd listeners process 1 defaults mode http timeout check 5s timeout connect 4s backend pool_all default-server inter 5s server server1 ip1:443 check ssl server server2 ip2:443 check ssl ... server serverN ipN:443 check ssl Without QAT enabled everything works just fine - healthchecks do not flap. With QAT engine enabled random server healtchecks flap: they fail and then shortly after they recover eg. 2019-03-06T15:06:22+01:00 localhost hapee-lb[1832]: Server pool_all/server1 is DOWN, reason: Layer6 timeout, check duration: 4000ms. 110 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 2019-03-06T15:06:32+01:00 localhost hapee-lb[1832]: Server pool_all/server1 is UP, reason: Layer6 check passed, check duration: 13ms. 117 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Increasing check frequency (lowering check interval) makes the problem occur more frequently. Anybody has a clue why this is happening ? Has anybody seen such behavior ? Regards, Marcin Deranek
Re: read async auth date from file
Hi Jeff, If the file is only stored in the FS and you can't put its content into HAProxy's memory at run time (using a map as explained by Jarno), then you may want to use SPOE. So the blocking files IO will be done in a process running outside of HAProxy. You have an SPOA (agent) example in HAProxy's source code, written in C. If you want an SPOA in an other language, I would say "stay tuned" :) Baptiste On Sun, Mar 3, 2019 at 9:20 AM Jeff wrote: > I need to add an authorization header for a target server, e.g. >http-request add-header Authorization Bearer\ MYTOKENDATA > > where MYTOKENDATA is read from a file for each proxy message. > (MYTOKENDATA is written asynchronously to the file by another > process.) > > How to do this in HAProxy? > > thanks, > Jeff > >
Re: read async auth date from file
Hi, On Sun, Mar 03, Jeff wrote: > I need to add an authorization header for a target server, e.g. >http-request add-header Authorization Bearer\ MYTOKENDATA > > where MYTOKENDATA is read from a file for each proxy message. Does this mean that each http request needs to read the MYTOKENDATA from file (file read access for each and every request) ? > (MYTOKENDATA is written asynchronously to the file by another > process.) > > How to do this in HAProxy? Just a few ideas from top of my head: - use maps / cli to update map values (https://www.haproxy.com/blog/introduction-to-haproxy-maps/) - lua could probably do this, but AFAIK doing file io will block rest of the haproxy, so it might be better to read the MYTOKENDATA from redis or memcache (or something similar). Is your use case something like described here: https://www.haproxy.com/blog/using-haproxy-as-an-api-gateway-part-2-authentication/ -Jarno -- Jarno Huuskonen
