Re: haproxy 2.2-dev8-7867525 - 100% cpu usage on 1 core after config 'reload'
Hi Christopher, Op 29-5-2020 om 09:00 schreef Christopher Faulet: Le 29/05/2020 à 00:45, PiBa-NL a écrit : Hi List, I noticed a issue with 2.2-dev8-release and with 2.2-dev8-7867525 the issue is still there that when a reload is 'requested' it fails to stop the old worker.. Hi Pieter, I was able to reproduce the bug. Thanks for the reproducer. I've fixed it. It should be ok now. Thanks for the quick fix! It works for me. Regards, PiBa-NL (Pieter)
haproxy 2.2-dev8-7867525 - 100% cpu usage on 1 core after config 'reload'
Hi List, I noticed a issue with 2.2-dev8-release and with 2.2-dev8-7867525 the issue is still there that when a reload is 'requested' it fails to stop the old worker.. The old worker shuts down most of its threads, but 1 thread starts running at 100% cpu usage of a core. Not sure yet 'when' the issue was introduced exactly.. Ive skiped quite a few dev releases and didnt have time to disect it to a specific version/commit yet. Ill try and do that during the weekend i noone does it earlier ;).. Normally dont use -W but am 'manually' restarting haproxy with -sf parameters.. but this seemed like the easier reproduction.. Also i 'think' i noticed once that dispite the -W parameter and logging output that a worker was spawned that there was only 1 process running, but couldnt reproduce that one sofar again... Also i havnt tried to see if and how i can connect through the master to the old worker process yet... perhaps also something i can try later.. I 'suspect' it has something to do with the healthchecks though... (and their refactoring as i think happened.?.) Anyhow perhaps this is already enough for someone to take a closer look.? If more info is needed ill try and provide :). Regards, PiBa-NL (Pieter) *Reproduction (works 99% of the time..):* haproxy -W -f /var/etc/haproxy-2020/haproxy.cfg kill -s USR2 17683 *haproxy.cfg* frontend www bind 127.0.0.1:81 mode http backend testVPS_ipv4 mode http retries 3 option httpchk OPTIONS /Test HTTP/1.1\r\nHost:\ test.test.nl server vps2a 192.168.30.10:80 id 10109 check inter 15000 backend O365mailrelay mode tcp option smtpchk HELO no option log-health-checks server-template O365smtp 2 test.mail.protection.outlook.com:25 id 122 check inter 1 *haproxy -vv* HA-Proxy version 2.2-dev8-7867525 2020/05/28 - https://haproxy.org/ Status: development branch - not safe for use in production. Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open Running on: FreeBSD 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017 r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -pipe -g -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-null-dereference -Wno-unused-label -Wno-unused-parameter -Wno-sign-compare -Wno-ignored-qualifiers -Wno-unused-command-line-argument -Wno-missing-field-initializers -Wno-address-of-packed-member -DFREEBSD_PORTS -DFREEBSD_PORTS OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_STATIC_PCRE=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ACCEPT4=1 USE_ZLIB=1 Feature list : -EPOLL +KQUEUE -NETFILTER +PCRE +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -BACKTRACE +STATIC_PCRE -STATIC_PCRE2 +TPROXY -LINUX_TPROXY -LINUX_SPLICE +LIBCRYPT -CRYPT_H +GETADDRINFO +OPENSSL +LUA -FUTEX +ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY -TFO -NS -DL -RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER -PRCTL -THREAD_DUMP -EVPORTS Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_THREADS=64, default=16). Built with OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017 Running on OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 Built with Lua version : Lua 5.3.4 Built with clang compiler version 4.0.0 (tags/RELEASE_400/final 297347) Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY Built with PCRE version : 8.40 2017-01-11 Running on PCRE version : 8.40 2017-01-11 PCRE library supports JIT : yes Encrypted password support via crypt(3): yes Built with zlib version : 1.2.11 Running on zlib version : 1.2.11 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Available polling systems : kqueue : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use kqueue. Available multiplexer protocols : (protocols marked as cannot be specified using 'proto' keyword) h2 : mode=HTTP side=FE|BE mux=H2 fcgi : mode=HTTP side=BE mux=FCGI : mode=HTTP side=FE|BE mux=H1 : mode=TCP side=FE|BE mux=PASS Available services : none Available filters : [SPOE] spoe [CACHE] cache [FCGI] fcgi-app [TRACE] trace [COMP] compression
Re: disabling test if ipv6 not supported ?
Hi Ilya, Op 21-5-2020 om 04:57 schreef Илья Шипицин: Hello, seems, freebsd images on cirrus-ci run with no ipv6 support https://cirrus-ci.com/task/6613883307687936 It fails on srv3 configuration, any ideay why doesn't it complain about srv2 as that also seems to me it uses IPv6..? any idea how we can skip such tests ? I think the test or code should get fixed.. not skipped because it fails. Note that this specific test recently got this extra 'addr ::1' server check parameter on srv3. Perhaps that that syntax is written/parsed wrongly? Cheers, Ilya Shipitcin Regards, PiBa-NL (Pieter)
Re: server-state application failed for server 'x/y', invalid srv_admin_state value '32'
Hi Baptiste, Op 6-4-2020 om 11:43 schreef Baptiste: Hi Piba, my answers inline. Using 2.2-dev5-c3500c3, I've got both a server and a servertemplate/server that are marked 'down' due to dns not replying with (enough) records. That by itself is alright.. (and likely has been like that for a while so i don't think its a regression.) You're right, this has always been like that. For the 'regression part' i was thinking about the warnings below which where 'likely' like that before 2.2-dev5 as well, it wasn't about marking servers as down which is totally expected & desired ;) i seem to read from your response here like you thought i thought otherwise...(damn that gets hard to understand, sorry..) And as you have confirmed its already causing the warnings in 1.8 as well.. So not a regression in 2.2-dev5 itself. But when i perform a 'seemless reload' with a serverstates file it causes the warnings below for both server and template.: [WARNING] 095/150909 (74796) : server-state application failed for server 'x/y', invalid srv_admin_state value '32' [WARNING] 095/150909 (74796) : server-state application failed for server 'x2/z3', invalid srv_admin_state value '32' Is there a way to get rid of these warnings, and if 32 is a invalid value, how did it get into the state file at all? I can confirm this is not supposed to happen! And I could reproduce this behavior since HAProxy 1.8. Not sure if its a bug or a feature request, but i do think it should be changed :). Can it be added to some todo list? Thanks. This is a bug from my point of view. I'll check this. Could you please open a github issue and tag me in there? Done: https://github.com/haproxy/haproxy/issues/576 Baptiste Thanks and regards, PiBa-NL (Pieter)
server-state application failed for server 'x/y', invalid srv_admin_state value '32'
Hi List, Using 2.2-dev5-c3500c3, I've got both a server and a servertemplate/server that are marked 'down' due to dns not replying with (enough) records. That by itself is alright.. (and likely has been like that for a while so i don't think its a regression.) But when i perform a 'seemless reload' with a serverstates file it causes the warnings below for both server and template.: [WARNING] 095/150909 (74796) : server-state application failed for server 'x/y', invalid srv_admin_state value '32' [WARNING] 095/150909 (74796) : server-state application failed for server 'x2/z3', invalid srv_admin_state value '32' Is there a way to get rid of these warnings, and if 32 is a invalid value, how did it get into the state file at all? ## Severely cut down config snippet..: backend x server y AppSrv:8084 id 161 check inter 1 weight 1 resolvers globalresolvers backend x2 server-template z 3 smtp.company.tld:25 id 167 check inter 1 weight 10 resolvers globalresolvers One could argue that my backend x should have a better dns name configured, if it doesn't exists i apparently messed up something.. For the second x2 backend though isn't it 'normal' to have a template sizing account for 'future growth' of the cluster? And as such always have some extra template-servers available that are in 'MAINT / resolution' state? As such when 2 servers of x2 are up, and the 3rd is in resolution state, it shouldn't warn on a restart imho as its to be expected for most setups.?. Not sure if its a bug or a feature request, but i do think it should be changed :). Can it be added to some todo list? Thanks. Thanks and regards, PiBa-NL (Pieter)
Re: [PATCHES] dns related
Hi Baptiste, Op 26-3-2020 om 12:46 schreef William Lallemand: On Wed, Mar 25, 2020 at 11:15:37AM +0100, Baptiste wrote: Hi there, A couple of patches here to cleanup and fix some bugs introduced by 13a9232ebc63fdf357ffcf4fa7a1a5e77a1eac2b. Baptiste Thanks Baptiste, merged. Thanks for this one. Question though, are you still working on making a non-existing server template go into 'resolution' state? See below/attached picture with some details.. (or see https://www.mail-archive.com/haproxy@formilux.org/msg36373.html ) Regards, PiBa-NL (Pieter)
Re: commit 493d9dc makes a SVN-checkout stall..
Hi Olivier, Willy, Just to confirm, as expected it (c3500c3) indeed works for me :). Thanks for the quick fix! Regards, PiBa-NL (Pieter) Op 25-3-2020 om 17:16 schreef Willy Tarreau: On Wed, Mar 25, 2020 at 05:08:03PM +0100, Olivier Houchard wrote: That is... interesting, not sure I reached such an outstanding result. Oh I stopped trying to guess long ago :-) This is now fixed, sorry about that ! Confirmed, much better now, thanks! Willy
commit 493d9dc makes a SVN-checkout stall..
Hi List, Willy, Today i thought lets give v2.2-dev5 a try for my production environment ;). Soon it turned out to cause SVN-Checkout to stall/disconnect for a repository we run locally in a Collab-SVN server. I tracked it down to this commit: 493d9dc (MEDIUM: mux-h1: do not blindly wake up the tasklet at end of request anymore) causing the problem for the first time. Is there something tricky there that can be suspected to cause the issue.? Perhaps a patch i can try? While 'dissecting' the issue i deleted the whole directory each time and performed a new svn-checkout several times. It doesn't always stall at the exact same point but usually after checking out around +- 20 files something between 0.5 and 2 MB. , the commit before that one allows me to checkout 500+MB through haproxy without issue.. A wireshark seems to show that haproxy is sending several of RST,ACK packets for a 4 different connections to the svn-server at the same milisecond after it was quiet for 2 seconds.. The whole issue happens in a timeframe of start of checkout till when it stalls within 15 seconds. The 'nokqueue' i usually try on my FreeBSD machine doesn't change anything. Hope you have an idea where to look. Providing captures/logs is a bit difficult without some careful scrubbing.. Regards, PiBa-NL (Pieter) ### Complete config (that still reproduces the issue.. things cant get much simpler than this..): frontend InternalSites.8.6-merged bind 192.168.8.67:80 mode http use_backend APP01-JIRA-SVN_ipvANY backend APP01-JIRA-SVN_ipvANY mode http server svn 192.168.104.20:8080 ### uname -a FreeBSD freebsd11 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017 r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 ### haproxy -vv HA-Proxy version 2.2-dev5-3e128fe 2020/03/24 - https://haproxy.org/ Status: development branch - not safe for use in production. Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -pipe -g -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-null-dereference -Wno-unused-label -Wno-unused-parameter -Wno-sign-compare -Wno-ignored-qualifiers -Wno-unused-command-line-argument -Wno-missing-field-initializers -Wno-address-of-packed-member -DFREEBSD_PORTS -DFREEBSD_PORTS OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_STATIC_PCRE=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ACCEPT4=1 USE_ZLIB=1 Feature list : -EPOLL +KQUEUE -NETFILTER +PCRE +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -BACKTRACE +STATIC_PCRE -STATIC_PCRE2 +TPROXY -LINUX_TPROXY -LINUX_SPLICE +LIBCRYPT -CRYPT_H +GETADDRINFO +OPENSSL +LUA -FUTEX +ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY -TFO -NS -DL -RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER -PRCTL -THREAD_DUMP -EVPORTS Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_THREADS=64, default=16). Built with OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017 Running on OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 Built with Lua version : Lua 5.3.4 Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY Built with PCRE version : 8.40 2017-01-11 Running on PCRE version : 8.40 2017-01-11 PCRE library supports JIT : yes Encrypted password support via crypt(3): yes Built with zlib version : 1.2.11 Running on zlib version : 1.2.11 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Available polling systems : kqueue : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use kqueue. Available multiplexer protocols : (protocols marked as cannot be specified using 'proto' keyword) h2 : mode=HTTP side=FE|BE mux=H2 fcgi : mode=HTTP side=BE mux=FCGI : mode=HTTP side=FE|BE mux=H1 : mode=TCP side=FE|BE mux=PASS Available services : none Available filters : [SPOE] spoe [CACHE] cache [FCGI] fcgi-app [TRACE] trace [COMP] compression
Re: dns fails to process response / hold valid? (since commit 2.2-dev0-13a9232)
Hi Baptiste, Op 19-2-2020 om 13:06 schreef Baptiste: Hi, I found a couple of bugs in that part of the code. Can you please try the attached patch? (0001 is useless but I share it too in case of) Works for me, thanks! It will allow parsing of additional records for SRV queries only and when done, will silently ignore any record which are not A or . @maint team, please don't apply the patch yet, I want to test it much more before. When the final patch is ready ill be happy to give it a try as well. Baptiste On a side note. With config below i would expect 2 servers with status 'MAINT(resolving)'. Using this configuration in Unbound (4 server IP's defined.): server: local-data: "_https._tcp.pkg.test.tld 3600 IN SRV 0 100 80 srv1.test.tld" local-data: "_https._tcp.pkg.test.tld 3600 IN SRV 0 100 80 srv2.test.tld" local-data: "srv1.test.tld 3600 IN A 192.168.0.51" local-data: "srv2.test.tld 3600 IN A 192.168.0.52" local-data: "srvX.test.tld 3600 IN A 192.168.0.53" local-data: "srvX.test.tld 3600 IN A 192.168.0.54" And this in a HAProxy backend: server-template PB_SRVrecords 3 ipv4@_https._tcp.pkg.test.tld:77 id 10110 check inter 18 resolvers globalresolvers resolve-prefer ipv4 server-template PB_multipleA 3 i...@srvx.test.tld:78 id 10111 check inter 18 resolvers globalresolvers resolve-prefer ipv4Results in 6 servers, but 1 is This results in 6 servers of which 1 server has 'MAINT(resolution)' status and 1 has an IP of 0.0.0.0 but shows as 'DOWN'. I would have expected 2 servers with status MAINT.? (p.s. none of the IP's actually exist on my network so that the other servers are also shown as down is correct..) PB_ipv4,PB_SRVrecords1,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,124,124,,1,10102,10110,,0,,2,0,,0,L4CON,,74995,0,0,0,0,0,0,0,0,-1,,,0,0,0,0Layer4 connection problem,,2,3,0192.168.0.51:80,,http0,0,0,,,0,,0,0,0,0,0, PB_ipv4,PB_SRVrecords2,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,94,94,,1,10102,2,,0,,2,0,,0,L4CON,,75029,0,0,0,0,0,0,0,0,-1,,,0,0,0,0Layer4 connection problem,,2,3,0192.168.0.52:80,,http0,0,0,,,0,,0,0,0,0,0, PB_ipv4,PB_SRVrecords3,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,64,64,,1,10102,3,,0,,2,0,,0,L4CON,,75039,0,0,0,0,0,0,0,0,-1,,,0,0,0,0Layer4 connection problem,,2,3,00.0.0.0:77,,http0,0,0,,,0,,0,0,0,0,0, PB_ipv4,PB_multipleA1,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,2,34,34,,1,10102,10111,,0,,2,0,,0,L4CON,,75002,0,0,0,0,0,0,0,0,-1,,,0,0,0,0Layer4 connection problem,,2,3,0192.168.0.53:78,,http0,0,0,,,0,,0,0,0,0,0, PB_ipv4,PB_multipleA2,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,2,4,4,,1,10102,5,,0,,2,0,,0,L4CON,,75014,0,0,0,0,0,0,0,0,-1,,,0,0,0,0Layer4 connection problem,,2,3,0192.168.0.54:78,,http0,0,0,,,0,,0,0,0,0,0, PB_ipv4,PB_multipleA3,0,0,0,0,,0,0,0,,0,,0,0,0,0,MAINT (resolution),1,1,0,0,1,199,199,,1,10102,6,,0,,2,0,,00,0,0,0,0,0,0,0,-1,,,0,0,0,00.0.0.0:78,,http0,0,0,,,0,,0,0,0,0,0, If additional info is desired, please let me know :). On Tue, Feb 18, 2020 at 2:03 PM Baptiste <mailto:bed...@gmail.com>> wrote: Hi guys, Thx Tim for investigating. I'll check the PCAP and see why such behavior happens. Baptiste On Tue, Feb 18, 2020 at 12:09 AM Tim Düsterhus mailto:t...@bastelstu.be>> wrote: Pieter, Am 09.02.20 um 15:35 schrieb PiBa-NL: > Before commit '2.2-dev0-13a9232, released 2020/01/22 (use additional > records from SRV responses)' i get seemingly proper working resolving of > server a name. > After this commit all responses are counted as 'invalid' in the socket > stats. I can confirm the issue with the provided configuration. The 'if (len == 0) {' check in line 1045 of the commit causes HAProxy to consider the responses 'invalid': Thanks for confirming :). https://github.com/haproxy/haproxy/commit/13a9232ebc63fdf357ffcf4fa7a1a5e77a1eac2b#diff-b2ddf457bc423779995466f7d8b9d147R1045-R1048 Best regards Tim Düsterhus Regards, PiBa-NL (Pieter)
Re: dns fails to process response / hold valid? (since commit 2.2-dev0-13a9232)
Hi List, Hereby a little bump. Can someone take a look? (maybe the pcap attachment didn't fly well through spam filters. (or the email formatting..)?) (or because i (wrongly?) chose to include Baptiste specifically in my addressing (he committed the original patch that caused the change in behaviour)..) Anyhow the current '2.2-dev2-a71667c, released 2020/02/17' is still affected. If someone was already planning to, please don't feel 'pushed' by this mail. i'm just trying to make sure this doesn't fall through the cracks :). Regards, PiBa-NL (Pieter) Op 9-2-2020 om 15:35 schreef PiBa-NL: Hi List, Baptiste, After updating haproxy i found that the DNS resolver is no longer working for me. Also i wonder about the exact effect that 'hold valid' should have. I pointed haproxy to a 'Unbound 1.9.4' dns server that does the recursive resolving of the dns request made by haproxy. Before commit '2.2-dev0-13a9232, released 2020/01/22 (use additional records from SRV responses)' i get seemingly proper working resolving of server a name. After this commit all responses are counted as 'invalid' in the socket stats. Attached also a pcap of the dns traffic. Which shows a short capture of a single attempt where 3 retries for both A and records show up. There is a additional record of type 'OPT' is present in the response.. But the exact same keeps repeating every 5 seconds. As for 'hold valid' (tested with the commit before this one) it seems that the stats page of haproxy shows the server in 'resolution' status way before the 3 minute 'hold valid' has passed when i simply disconnect the network of the server running the Unbound-DNS server. Though i guess that is less important that dns working at all in the first place.. If any additional information is needed please let me know :). Can you/someone take a look? Thanks in advance. p.s. i think i read something about a 'vtest' that can test the haproxy DNS functionality, if you have a example that does this i would be happy to provide a vtest with a reproduction of the issue though i guess it will be kinda 'slow' if it needs to test for hold valid timings.. Regards, PiBa-NL (Pieter) haproxy config: resolvers globalresolvers nameserver pfs_routerbox 192.168.0.18:53 resolve_retries 3 timeout retry 200 hold valid 3m hold nx 10s hold other 15s hold refused 20s hold timeout 25s hold obsolete 30s timeout resolve 5s frontend nu_nl bind 192.168.0.19:433 name 192.168.0.19:433 ssl crt-list /var/etc/haproxy/nu_nl.crt_list mode http log global option http-keep-alive timeout client 3 use_backend nu.nl_ipvANY backend nu.nl_ipvANY mode http id 2113 log global timeout connect 3 timeout server 3 retries 3 option httpchk GET / HTTP/1.0\r\nHost:\ nu.nl\r\nAccept:\ */* server nu_nl nu.nl:443 id 2114 ssl check inter 1 verify none resolvers globalresolvers check-sni nu.nl resolve-prefer ipv4 haproxy_socket.sh show resolvers Resolvers section globalresolvers nameserver pfs_routerbox: sent: 216 snd_error: 0 valid: 0 update: 0 cname: 0 cname_error: 0 any_err: 108 nx: 0 timeout: 0 refused: 0 other: 0 invalid: 108 too_big: 0 truncated: 0 outdated: 0 haproxy -vv HA-Proxy version 2.2-dev0-13a9232 2020/01/22 - https://haproxy.org/ Status: development branch - not safe for use in production. Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -pipe -g -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-null-dereference -Wno-unused-label -Wno-unused-parameter -Wno-sign-compare -Wno-ignored-qualifiers -Wno-unused-command-line-argument -Wno-missing-field-initializers -Wno-address-of-packed-member -DFREEBSD_PORTS -DFREEBSD_PORTS OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_REGPARM=1 USE_STATIC_PCRE=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ACCEPT4=1 USE_ZLIB=1 Feature list : -EPOLL +KQUEUE -MY_EPOLL -MY_SPLICE -NETFILTER +PCRE +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY -LINUX_TPROXY -LINUX_SPLICE +LIBCRYPT -CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA -FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY -TFO -NS -DL -RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER -PRCTL -THREAD_DUMP -EVPORTS Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with mult
dns fails to process response / hold valid? (since commit 2.2-dev0-13a9232)
Hi List, Baptiste, After updating haproxy i found that the DNS resolver is no longer working for me. Also i wonder about the exact effect that 'hold valid' should have. I pointed haproxy to a 'Unbound 1.9.4' dns server that does the recursive resolving of the dns request made by haproxy. Before commit '2.2-dev0-13a9232, released 2020/01/22 (use additional records from SRV responses)' i get seemingly proper working resolving of server a name. After this commit all responses are counted as 'invalid' in the socket stats. Attached also a pcap of the dns traffic. Which shows a short capture of a single attempt where 3 retries for both A and records show up. There is a additional record of type 'OPT' is present in the response.. But the exact same keeps repeating every 5 seconds. As for 'hold valid' (tested with the commit before this one) it seems that the stats page of haproxy shows the server in 'resolution' status way before the 3 minute 'hold valid' has passed when i simply disconnect the network of the server running the Unbound-DNS server. Though i guess that is less important that dns working at all in the first place.. If any additional information is needed please let me know :). Can you/someone take a look? Thanks in advance. p.s. i think i read something about a 'vtest' that can test the haproxy DNS functionality, if you have a example that does this i would be happy to provide a vtest with a reproduction of the issue though i guess it will be kinda 'slow' if it needs to test for hold valid timings.. Regards, PiBa-NL (Pieter) haproxy config: resolvers globalresolvers nameserver pfs_routerbox 192.168.0.18:53 resolve_retries 3 timeout retry 200 hold valid 3m hold nx 10s hold other 15s hold refused 20s hold timeout 25s hold obsolete 30s timeout resolve 5s frontend nu_nl bind 192.168.0.19:433 name 192.168.0.19:433 ssl crt-list /var/etc/haproxy/nu_nl.crt_list mode http log global option http-keep-alive timeout client 3 use_backend nu.nl_ipvANY backend nu.nl_ipvANY mode http id 2113 log global timeout connect 3 timeout server 3 retries 3 option httpchk GET / HTTP/1.0\r\nHost:\ nu.nl\r\nAccept:\ */* server nu_nl nu.nl:443 id 2114 ssl check inter 1 verify none resolvers globalresolvers check-sni nu.nl resolve-prefer ipv4 haproxy_socket.sh show resolvers Resolvers section globalresolvers nameserver pfs_routerbox: sent: 216 snd_error: 0 valid: 0 update: 0 cname: 0 cname_error: 0 any_err: 108 nx: 0 timeout: 0 refused: 0 other: 0 invalid: 108 too_big: 0 truncated: 0 outdated: 0 haproxy -vv HA-Proxy version 2.2-dev0-13a9232 2020/01/22 - https://haproxy.org/ Status: development branch - not safe for use in production. Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -pipe -g -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-null-dereference -Wno-unused-label -Wno-unused-parameter -Wno-sign-compare -Wno-ignored-qualifiers -Wno-unused-command-line-argument -Wno-missing-field-initializers -Wno-address-of-packed-member -DFREEBSD_PORTS -DFREEBSD_PORTS OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_REGPARM=1 USE_STATIC_PCRE=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ACCEPT4=1 USE_ZLIB=1 Feature list : -EPOLL +KQUEUE -MY_EPOLL -MY_SPLICE -NETFILTER +PCRE +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY -LINUX_TPROXY -LINUX_SPLICE +LIBCRYPT -CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA -FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY -TFO -NS -DL -RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER -PRCTL -THREAD_DUMP -EVPORTS Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_THREADS=64, default=2). Built with OpenSSL version : OpenSSL 1.1.1a-freebsd 20 Nov 2018 Running on OpenSSL version : OpenSSL 1.1.1a-freebsd 20 Nov 2018 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 Built with Lua version : Lua 5.3.5 Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY Built with PCRE version : 8.43 2019-02-23 Running on PCRE version : 8.43 2019-02-23 PCRE library supports JIT : yes Encrypted password support via crypt(3): yes Built with zlib version : 1.2.11 Running on zlib versio
mcli vtest broken by commit.?. MEDIUM: connections: Get ride of the xprt_done callback.
Hi Olivier, Just to let you know, seems this commit has broken a few regtests: http://git.haproxy.org/?p=haproxy.git;a=commit;h=477902bd2e8c1e978ad43d22dba1f28525bb797a https://api.cirrus-ci.com/v1/task/5885732300521472/logs/main.log Testing with haproxy version: 2.2-dev1 #top TEST reg-tests/mcli/mcli_show_info.vtc TIMED OUT (kill -9) #top TEST reg-tests/mcli/mcli_show_info.vtc FAILED (10.044) signal=9 #top TEST reg-tests/mcli/mcli_start_progs.vtc TIMED OUT (kill -9) #top TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (10.019) signal=9 Can reproduce it on my own FreeBSD machine as well, the testcase just sits and waits.. until the vtest-timeout strikes. Do you need more info? If so what can i provide.? Regards, Pieter
Re: freebsd ci is broken - commit 08fa16e - curl download stalls in reg-tests/compression/lua_validation.vtc
Hi Olivier, Willy, Ilya, Thanks! I confirm 2.2-dev0-ac81474 fixes this issue for me. And cirrus-ci also shows 'all green' again :). Running the same test with 16vCPU and kqueue enabled its 'all okay': 0 tests failed, 0 tests skipped, 200 tests passed. Op 15-1-2020 om 19:20 schreef Olivier Houchard: Hi guys, On Tue, Jan 14, 2020 at 09:45:34PM +0100, Willy Tarreau wrote: Hi guys, On Tue, Jan 14, 2020 at 08:02:51PM +0100, PiBa-NL wrote: Below a part of the output that the test generates for me. The first curl request seems to succeed, but the second one runs into a timeout.. When compiled with the commit before 08fa16e <https://github.com/haproxy/haproxy/commit/08fa16e397ffb1c6511b98ade2a3bfff9435e521> Ah, and unsurprizingly I'm the author :-/ I'm wondering why it only affects FreeBSD (very likely kqueue in fact, I suppose it works if you start with -dk). Maybe something subtle escaped me in the poller after the previous changes. Should i update to a newer FreeBSD version, or is it likely unrelated, and in need of some developer attention.. Do you (Willy or anyone), need more information from my side? Or is there a patch i can try to validate? I don't think I need more info for now and your version has nothing to do with this (until proven otherwise). I apparently really broke something there. I think I have a FreeBSD VM somewhere, in the worst case I'll ask Olivier for some help :-) To give you a quick update, we investigating that, and I'm still not really sure why it only affects FreeBSD, but we fully understood the problem, and it should be fixed by now. Regards, Olivier Regards, PiBa-NL (Pieter)
Re: freebsd ci is broken - commit 08fa16e - curl download stalls in reg-tests/compression/lua_validation.vtc
Hi Ilya, Willy, Op 14-1-2020 om 21:40 schreef Илья Шипицин: PiBa, how many CPU cores are you running ? it turned out that I run tests on very low vm, which only has 1 core. and tests pass. cirrus-ci as far as I remember do have many cores. I was running with 16 cores.. can you find single core vm ? Well, i reconfigured the VM to have 1 core, but same issue seems to show up, though not on every time the test is run, and actually a bit less often.. Below some additional testresults with different kqueue / vCPU settings.. *VM with 1 vCPU* Running: ./vtest/VTest-master/vtest -Dno-htx=no -l -k -b 50M -t 5 -n 20 ./work/haproxy-08fa16e/reg-tests/compression/lua_validation.vtc Results in: 4 tests failed, 0 tests skipped, 16 tests passed Adding "nokqueue" in the vtc file i get: 8 tests failed, 0 tests skipped, 12 tests passed 4 tests failed, 0 tests skipped, 16 tests passed So its a bit random, but the 'nokqueue' directive does not seem to affect results much.. *With 16 vCPU* Without nokqueue: 16 tests failed, 0 tests skipped, 4 tests passed With nokqueue (using poll): 17 tests failed, 0 tests skipped, 3 tests passed The failure rate seems certainly higher with many cores.. * Using commit 0eae632 it works OK* Just to be sure i re-tested on 16 cores with 2.2-dev0-0eae632 but that does nicely pass: 0 tests failed, 0 tests skipped, 20 tests passed Regards, PiBa-NL (Pieter)
Re: freebsd ci is broken - commit 08fa16e - curl download stalls in reg-tests/compression/lua_validation.vtc
Hi Ilya, Thanks! Op 14-1-2020 om 07:48 schreef Илья Шипицин: Hello, since https://github.com/haproxy/haproxy/commit/08fa16e397ffb1c6511b98ade2a3bfff9435e521 freebsd CI is red: https://cirrus-ci.com/task/5960933184897024 I'd say "it is something with CI itself", when I run the same tests locally on freebsd, it is green. Sadly i do get the same problem on my test server (version info below its version 11.1 is a bit outdated, but hasn't failed my before...). PiBa ? thanks, Ilya Shipitcin Below a part of the output that the test generates for me. The first curl request seems to succeed, but the second one runs into a timeout.. When compiled with the commit before 08fa16e <https://github.com/haproxy/haproxy/commit/08fa16e397ffb1c6511b98ade2a3bfff9435e521> it does not show that behaviour.. Current latest(24c928c) commit is still affected.. top shell_out| % Total % Received % Xferd Average Speed Time Time Time Current top shell_out| Dload Upload Total Spent Left Speed top shell_out|\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 418k 0 418k 0 0 1908k 0 --:--:-- --:--:-- --:--:-- 1908k top shell_out| % Total % Received % Xferd Average Speed Time Time Time Current top shell_out| Dload Upload Total Spent Left Speed top shell_out|\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 141k 0 141k 0 0 284k 0 --:--:-- --:--:-- --:--:-- 284k\r100 343k 0 343k 0 0 156k 0 --:--:-- 0:00:02 --:--:-- 156k\r100 343k 0 343k 0 0 105k 0 --:--:-- 0:00:03 --:--:-- 105k\r100 343k 0 343k 0 0 81274 0 --:--:-- 0:00:04 --:--:-- 81274\r100 343k 0 343k 0 0 65228 0 --:--:-- 0:00:05 --:--:-- 65240\r100 343k 0 343k 0 0 54481 0 --:--:-- 0:00:06 --:--:-- 34743\r100 343k 0 343k 0 0 46768 0 --:--:-- 0:00:07 --:--:-- 0\r100 343k 0 343k 0 0 40968 0 --:--:-- 0:00:08 --:--:-- 0\r100 343k 0 343k 0 0 36452 0 --:--:-- 0:00:09 --:--:-- 0\r100 343k 0 343k 0 0 32830 0 --:--:-- 0:00:10 --:--:-- 0\r100 343k 0 343k 0 0 29865 0 --:--:-- 0:00:11 --:--:-- 0\r100 343k 0 343k 0 0 27395 0 --:--:-- 0:00:12 --:--:-- 0\r100 343k 0 343k 0 0 25297 0 --:--:-- 0:00:13 --:--:-- 0\r100 343k 0 343k 0 0 23500 0 --:--:-- 0:00:14 --:--:-- 0\r100 343k 0 343k 0 0 23431 0 --:--:-- 0:00:15 --:--:-- 0 top shell_out|curl: (28) Operation timed out after 15002 milliseconds with 351514 bytes received top shell_out|Expecting checksum 4d9c62aa5370b8d5f84f17ec2e78f483 top shell_out|Received checksum: da2d120aedfd693eeba9cf1e578897a8 top shell_status = 0x0001 top shell_exit not as expected: got 0x0001 wanted 0x * top RESETTING after ./work/haproxy-08fa16e/reg-tests/compression/lua_validation.vtc Should i update to a newer FreeBSD version, or is it likely unrelated, and in need of some developer attention.. Do you (Willy or anyone), need more information from my side? Or is there a patch i can try to validate? Regards, PiBa-NL (Pieter) Yes im running a somewhat outdated OS here: FreeBSD freebsd11 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017 r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 Version used: haproxy -vv HA-Proxy version 2.2-dev0-08fa16e 2020/01/08 - https://haproxy.org/ Status: development branch - not safe for use in production. Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -pipe -g -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-null-dereference -Wno-unused-label -Wno-unused-parameter -Wno-sign-compare -Wno-ignored-qualifiers -Wno-unused-command-line-argument -Wno-missing-field-initializers -Wno-address-of-packed-member -DFREEBSD_PORTS -DFREEBSD_PORTS OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_REGPARM=1 USE_STATIC_PCRE=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ACCEPT4=1 USE_ZLIB=1 Feature list : -EPOLL +KQUEUE -MY_EPOLL -MY_SPLICE -NETFILTER +PCRE +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY -LINUX_TPROXY -LINUX_SPLICE +LIBCRYPT -CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA -FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY -TFO -NS -DL -RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER -PRCTL -THREAD_DUMP -EVPORTS Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built wi
Re: commit 246c024 - breaks loading crt-list with .ocsp files present
Op 15-10-2019 om 13:52 schreef William Lallemand: I pushed the fix. Thanks Fix confirmed. Thank you.
Re: freebsd builds are broken for few days - 30ee1ef, proxy_protocol_random_fail.vtc fails because scheme and host are now present in the syslog output.
Hi Christopher, It seems you fixed/changed the issue i noticed below a few minutes ago in commit 452e578 :) , thanks. One question remaining about this on my side is if it is expected that some platforms will use 'normalized' URI and others platforms just the regular / ? Regards, PiBa-NL (Pieter) Op 14-10-2019 om 21:22 schreef PiBa-NL: Hi Ilya, Willy, Op 13-10-2019 om 19:30 schreef Илья Шипицин: https://cirrus-ci.com/github/haproxy/haproxy I'll bisect if noone else knows what's going on @IIlya, thanks for checking my favorite platform, FreeBSD ;). @Willy, this 30ee1ef <http://git.haproxy.org/?p=haproxy.git;a=commit;h=30ee1ef> (MEDIUM: h2: use the normalized URI encoding for absolute form requests) commit 'broke' the expect value of the vtest, i don't know why other platforms don't see the same change in syslog output though.. Anyhow this is the output i get when running the /reg-tests/connection/proxy_protocol_random_fail.vtc Slog_1 0.033 syslog|<134>Oct 14 19:56:34 haproxy[78982]: ::1:47040 [14/Oct/2019:19:56:34.391] ssl-offload-http~ ssl-offload-http/http 0/0/0/0/0 503 222 - - 1/1/0/0/0 0/0 "POST https://[::1]:47037/1 HTTP/2.0" ** Slog_1 0.033 === expect ~ "ssl-offload-http/http .* \"POST /[1-8] HTTP/(2\\.0... Slog_1 0.033 EXPECT FAILED ~ "ssl-offload-http/http .* "POST /[1-8] HTTP/(2\.0|1\.1)"" If i change the vtc vtest file from: expect ~ "ssl-offload-http/http .* \"POST /[1-8] HTTP/(2\\.0|1\\.1)\"" To: expect ~ "ssl-offload-http/http .* \"POST https://[[]::1]:[0-9]{1,5}/[1-8] HTTP/(2\\.0|1\\.1)\"" or: expect ~ "ssl-offload-http/http .* \"POST https://[[]${h1_ssl_addr}]:${h1_ssl_port}/[1-8] HTTP/(2\\.0|1\\.1)\"" Then the test succeeds for me... but now the question is, should or shouldn't the scheme and host be present in the syslog output on all platforms.? Or should the regex contain a (optional?) check for this extra part? (Also note that even with these added variables in my second regext attempt its still using accolades around the IPv6 address.. not sure if all machines would use ipv6 for their localhost connection..) Regards, PiBa-NL (Pieter)
commit 246c024 - breaks loading crt-list with .ocsp files present
Hi William, I'm having an issue with the latest master code 2.1-dev2-4a66013. It does compile but doesn't want to load my crt-list with .ocsp files present for the certificates mentioned. The commit that broke this is: 246c024 # haproxy -v HA-Proxy version 2.1-dev2-4a66013 2019/10/14 - https://haproxy.org/ # haproxy -f ./PB-TEST/ultimo_testcase/xxx/haproxy.cfg -d [ALERT] 286/223026 (39111) : parsing [./PB-TEST/ultimo_testcase/xxx/haproxy.cfg:61] : 'bind 0.0.0.0:443' : 'crt-list' : error processing line 1 in file '/usr/ports-pb_haproxy-devel/PB-TEST/ultimo_testcase/xxx/rtrcld.xxx.crt_list' : (null) [ALERT] 286/223026 (39111) : Error(s) found in configuration file : ./PB-TEST/ultimo_testcase/xxx/haproxy.cfg [ALERT] 286/223026 (39111) : Fatal errors found in configuration. Content of the crt-list file, but removing the alpn stuff doesn't help...: /usr/ports-pb_haproxy-devel/PB-TEST/ultimo_testcase/xxx/rtrcld.xxx.pem [ alpn h2,http/1.1] /usr/ports-pb_haproxy-devel/PB-TEST/ultimo_testcase/xxx/rtrcld.xxx/rtrcld.xxx_5ab0da70ab0cc.pem [ alpn h2,http/1.1] The last line is an empty one.. but it already complains about line 1... which seems valid and the .pem file exists.. exact same config loads alright commits before this one: 246c024. I do have a 'filled' .ocsp file present. But no matter if its outdated, empty or correct the error above stays. When the .ocsp is absent it complains about line 2 of the cert-list.. Which has its own .ocsp as well.. Can you take a look? Thanks in advance. Regards, PiBa-NL (Pieter)
Re: freebsd builds are broken for few days - 30ee1ef, proxy_protocol_random_fail.vtc fails because scheme and host are now present in the syslog output.
Hi Ilya, Willy, Op 13-10-2019 om 19:30 schreef Илья Шипицин: https://cirrus-ci.com/github/haproxy/haproxy I'll bisect if noone else knows what's going on @IIlya, thanks for checking my favorite platform, FreeBSD ;). @Willy, this 30ee1ef <http://git.haproxy.org/?p=haproxy.git;a=commit;h=30ee1ef> (MEDIUM: h2: use the normalized URI encoding for absolute form requests) commit 'broke' the expect value of the vtest, i don't know why other platforms don't see the same change in syslog output though.. Anyhow this is the output i get when running the /reg-tests/connection/proxy_protocol_random_fail.vtc Slog_1 0.033 syslog|<134>Oct 14 19:56:34 haproxy[78982]: ::1:47040 [14/Oct/2019:19:56:34.391] ssl-offload-http~ ssl-offload-http/http 0/0/0/0/0 503 222 - - 1/1/0/0/0 0/0 "POST https://[::1]:47037/1 HTTP/2.0" ** Slog_1 0.033 === expect ~ "ssl-offload-http/http .* \"POST /[1-8] HTTP/(2\\.0... Slog_1 0.033 EXPECT FAILED ~ "ssl-offload-http/http .* "POST /[1-8] HTTP/(2\.0|1\.1)"" If i change the vtc vtest file from: expect ~ "ssl-offload-http/http .* \"POST /[1-8] HTTP/(2\\.0|1\\.1)\"" To: expect ~ "ssl-offload-http/http .* \"POST https://[[]::1]:[0-9]{1,5}/[1-8] HTTP/(2\\.0|1\\.1)\"" or: expect ~ "ssl-offload-http/http .* \"POST https://[[]${h1_ssl_addr}]:${h1_ssl_port}/[1-8] HTTP/(2\\.0|1\\.1)\"" Then the test succeeds for me... but now the question is, should or shouldn't the scheme and host be present in the syslog output on all platforms.? Or should the regex contain a (optional?) check for this extra part? (Also note that even with these added variables in my second regext attempt its still using accolades around the IPv6 address.. not sure if all machines would use ipv6 for their localhost connection..) Regards, PiBa-NL (Pieter)
Re: haproxy -v doesn't show commit used when building from 2.0 repository?
Hi Willy, Op 1-8-2019 om 6:21 schreef Willy Tarreau: Hi Pieter, On Wed, Jul 31, 2019 at 10:56:54PM +0200, PiBa-NL wrote: Hi List, I have build haproxy 2.0.3-0ff395c from sources however after running a 'haproxy -v' it shows up as: 'HA-Proxy version 2.0.3 2019/07/23 - https://haproxy.org/' this isn't really correct imho as its a version based on code committed on date 7/30. And i kinda expected the commit-id to be part of the version shown? I know what's happening, I always forget to do it with each new major release. We're using Git attributes to automatically patch files "SUBVERS" and "VERDATE" when creating the archive : $ cat info/attributes SUBVERS export-subst VERDATE export-subst And this is something I forget to re-create with each new repository, I've fixed it now. It will be OK with new snapshots starting tomorrow. Thanks! Willy Works for me, building latest commit in 2.0 repository now haproxy -v shows: "HA-Proxy version 2.0.3-7343c71 2019/08/01 - https://haproxy.org/"; for me. Thanks for your quick fix&reply :). Regards, PiBa-NL (Pieter)
haproxy -v doesn't show commit used when building from 2.0 repository?
Hi List, I have build haproxy 2.0.3-0ff395c from sources however after running a 'haproxy -v' it shows up as: 'HA-Proxy version 2.0.3 2019/07/23 - https://haproxy.org/' this isn't really correct imho as its a version based on code committed on date 7/30. And i kinda expected the commit-id to be part of the version shown? Did i do something wrong? I thought the commit should automatically become part of the version. Though its very well possible ive broken the local freebsd makefile im using.. When building from master repository it seems to work fine though. If its caused by the contents of the repository, can it be changed? I find it really useful to see which commit a certain compiled haproxy binary was based upon. Thanks in advance . Regards, PiBa-NL (Pieter)
haproxy -v doesn't show commit used when building from 2.0 repository?
Hi List, I have build haproxy 2.0.3-0ff395c from sources however after running a 'haproxy -v' it shows up as: 'HA-Proxy version 2.0.3 2019/07/23 - https://haproxy.org/' this isn't really correct imho as its a version based on code committed on date 7/30. And i kinda expected the commit-id to be part of the version shown? Did i do something wrong? I thought the commit should automatically become part of the version. Though its very well possible ive broken the local freebsd makefile im using.. When building from master repository it seems to work fine though. If its caused by the contents of the repository, can it be changed? I find it really useful to see which commit a certain compiled haproxy binary was based upon. Thanks in advance :). Regards, PiBa-NL (Pieter)
Re: slow healthchecks after dev6+ with added commit "6ec902a MINOR: threads: serialize threads initialization"
Hi Willy, Op 11-6-2019 om 11:37 schreef Willy Tarreau: On Tue, Jun 11, 2019 at 09:06:46AM +0200, Willy Tarreau wrote: I'd like you to give it a try in your environment to confirm whether or not it does improve things. If so, I'll clean it up and merge it. I'm also interested in any reproducer you could have, given that the made up test case I did above doesn't even show anything alarming. No need to waste your time anymore, I now found how to reproduce it with this config : global stats socket /tmp/sock1 mode 666 level admin nbthread 64 backend stopme timeout server 1s option tcp-check tcp-check send "debug dev exit\n" server cli unix@/tmp/sock1 check The I run it in loops bound to different CPU counts : $ time for i in {1..20}; do taskset -c 0,1,2,3 ./haproxy -db -f slow-init.cfg >/dev/null 2>&1 done With a single CPU, it can take up to 10 seconds to run the loop on commits e186161 and e4d7c9d while it takes 0.18 second with the patch. With 4 CPUs like above, it takes 1.5s with e186161, 2.3s with e4d7c9d and 0.16 second with the patch. The tests I had run consisted in starting hundreds of thousands of listeners to amplify the impact of the start time, but in the end it was diluting the extra time in an already very long time. Running it in loops like above is quite close to what regtests do and explains why I couldn't spot the difference (e.g. a few hundreds of ms at worst among tens of seconds). Thus I'm merging the patch now (cleaned up already and tested as well without threads). Let's hope it's the last time :-) Thanks, Willy Seems i kept you busy for another day.. But the result is there, it looks 100% fixed to me :). Running without nbthread, and as such using 16 threads of the VM i'm using, i now get this: 2.0-dev7-ca3551f 2019/06/11 ** h1 0.732 WAIT4 pid=80796 status=0x0002 (user 0.055515 sys 0.039653) ** h2 0.846 WAIT4 pid=80799 status=0x0002 (user 0.039039 sys 0.039039) # top TEST ./test/tls_health_checks-org.vtc passed (0.848) Also with repeating the testcase 1000 times while running 10 of them in parallel only 1 of them failed with a timeout: S1 0.280 syslog|<134>Jun 11 22:24:48 haproxy[88306]: ::1:63856 [11/Jun/2019:22:24:48.074] fe1/1: Timeout during SSL handshake I think together with the really short timeouts in the testcase itself this is an excellent result. I'm considering this one fully fixed, thanks again. Regards, PiBa-NL (Pieter)
Re: slow healthchecks after dev6+ with added commit "6ec902a MINOR: threads: serialize threads initialization"
Hi Willy, Op 10-6-2019 om 16:14 schreef Willy Tarreau: Hi Pieter, On Mon, Jun 10, 2019 at 04:06:13PM +0200, PiBa-NL wrote: Things certainly look better again now regarding this issue. Ah cool! Running the test repeatedly, and manually looking over the results its pretty much as good as it was before. There seems to be a 1 ms increase in the check-duration, but maybe this is because of the moved initialization which on startup delays the first test a millisecond or something? It should not. At this point I think it can be anything including measurement noise or even thread assigment on startup! Below some test results that are based on manual observation and some in my head filtering of the console output.. (mistakes included ;) ) repeat 10 ./vt -v ./work/haproxy-*/reg-tests/checks/tls_health_checks.vtc | grep Layer7 | grep OK | grep WARNING Commit-ID , min-max time for +-95% check durations , comment e4d7c9d , 6 - 9 ms , all tests pass (1 tests out of +- a hundred showed 29ms , none below 6ms and almost half of them show 7ms) Great! 6ec902a , 11 - 150 ms , of the 12 tests that passed That's quite a difference indeed. e186161 , 5 - 8 ms , all tests pass (1 test used 15 ms, more than half the tests show 5ms check duration the majority of the remainder show 6ms) OK! I'm not sure if this deserves further investigation at the moment, i think it does not. Thanks for spending your weekend on this :) that wasn't my intention. Oh don't worry, you know I'm a low-level guy, just give me a problem to solve with a few bits available only and I can spend countless hours on it! Others entertain themselves playing games, for me this is a game :-) Thanks a lot for testing, at least we know there isn't another strange thing hidden behind. Cheers, Willy After a bit more fiddling i noticed that the new startup method seems more CPU intensive. Also it can be seen the vtest does take a bit longer to pass 1.3sec v.s. 0.8sec even though the health-check durations themselves are short as expected. Also its using quite a bit more 'user' cpu. I was wondering if this is a consequence of the new init sequence, or perhaps some improvement is still needed there.? I noticed this after trying to run multiple tests simultaneously again they interfered more with each-other then they used to.. 2.0-dev6-e4d7c9d 2019/06/10 ** h1 1.279 WAIT4 pid=63820 status=0x0002 (user 4.484293 sys 0.054781) ** h2 1.394 WAIT4 pid=63823 status=0x0002 (user 4.637692 sys 0.015588) # top TEST ./test/tls_health_checks-org.vtc passed (1.395) Before the latest changes it used less 'user': 2.0-dev6-e186161 2019/06/07 ** h1 0.783 WAIT4 pid=65811 status=0x0002 (user 1.077052 sys 0.031218) ** h2 0.897 WAIT4 pid=65814 status=0x0002 (user 0.341360 sys 0.037928) # top TEST ./test/tls_health_checks-org.vtc passed (0.899) And with 'nbthread 1' the user cpu usage is even more dramatically lower with the same test.. 2.0-dev6-e4d7c9d 2019/06/10 ** h1 0.684 WAIT4 pid=67990 status=0x0002 (user 0.015203 sys 0.015203) ** h2 0.791 WAIT4 pid=67993 status=0x0002 (user 0.013551 sys 0.009034) # top TEST ./test/tls_health_checks-org.vtc passed (0.793) 2.0-dev6-e186161 2019/06/07 ** h1 0.682 WAIT4 pid=65854 status=0x0002 (user 0.007158 sys 0.021474) ** h2 0.790 WAIT4 pid=65857 status=0x0002 (user 0.007180 sys 0.014361) If a single threaded haproxy process can run with 0.015 user-cpu-usage, i would not have expected it to required 4.4 on a 16 core cpu for the same startup&actions. Where it should be easier to spawn a second thread with the already parsed config instead of more expensive.?. Even if it parses the config once in each thread separately it doesn't make sense to me. So i thought also to try with 'nbthread 8' and that still seems to be 'alright' as seen below.. so i guess with the default of nbthread 16 the h1 and h2 get into some conflict fighting over the available cores.??. And haproxy by default will use all cores since 2.0-dev3 so i guess it might cause some undesirable effects in the field once it gets released and isn't the only process running on a machine, and even if it is the only intensive process, i wonder what other VM's might think about it on the same hypervisor, though i know VM's always give 'virtual performance' ;) .. Running with nbthread 8, still relatively low user usage & test time: 2.0-dev6-e4d7c9d 2019/06/10 ** h1 0.713 WAIT4 pid=68467 status=0x0002 (user 0.197443 sys 0.022781) ** h2 0.824 WAIT4 pid=68470 status=0x0002 (user 0.184567 sys 0.026366) # top TEST ./test/tls_health_checks-org.vtc passed (0.825) Hope you can make sense of some of this. Sorry for not noticing earlier, i guess i was to focused at only the health-check-duration. Or maybe its just me interpreting the numbers wrongly, that's surely also an option. Regards, PiBa-NL (Pieter)
Re: slow healthchecks after dev6+ with added commit "6ec902a MINOR: threads: serialize threads initialization"
Hi Willy, Op 10-6-2019 om 11:09 schreef Willy Tarreau: Hi Pieter, On Sat, Jun 08, 2019 at 06:07:09AM +0200, Willy Tarreau wrote: Hi Pieter, On Fri, Jun 07, 2019 at 11:32:18PM +0200, PiBa-NL wrote: Hi Willy, After the commit "6ec902a MINOR: threads: serialize threads initialization" however i have failing / slow health checks in the tls_health_checks.vtc test. Before that the Layer7-OK takes 5ms after this commit the healthcheck takes up to 75ms or even more.. It causes the 20ms connect/server timeouts to also to fail the test fairly often but not always.. This is very strange, as the modification only involves threads startup. Hmmm actually I'm starting to think about a possibility I need to verify. I suspect it may happen that a thread manages to finish its initialization before others request synchronzation, thus belives it's alone and starts. I'm going to have a deeper look at this problem with this in mind. I didn't notice the failed check here but I'll hammer it a bit more. Sorry for the long silence, it was harder than I thought. So I never managed to reproduce this typical issue, even by adding random delays here and there, but I managed to see that some threads were starting the event loop before others were done initializing, which will obviously result in issues such as missed events that could result in what you observed. I initially thought I could easily add a synchronization step using the current two bit fields (and spent my whole week-end writing parallel algorithms and revisiting all our locking mechanism just because of this). After numerous failed attempts, I later figured that I needed to represent more than 4 states per thread and that 2 bits are not enough. Bah... at least I had fun time... Thus I added a new field and a simple function to allow the code to start in synchronous steps. We now initialize one thread at a time, then once they are all initialized we enable the listeners, and once they are enabled, we start the pollers in all threads. It is pretty obvious from the traces that it now does the right thing. However since I couldn't reproduce the health check issue you were facing, I'm interested in knowing if it's still present with the latest master, as it could also uncover another issue. Thanks! Willy Things certainly look better again now regarding this issue. Running the test repeatedly, and manually looking over the results its pretty much as good as it was before. There seems to be a 1 ms increase in the check-duration, but maybe this is because of the moved initialization which on startup delays the first test a millisecond or something? Below some test results that are based on manual observation and some in my head filtering of the console output.. (mistakes included ;) ) repeat 10 ./vt -v ./work/haproxy-*/reg-tests/checks/tls_health_checks.vtc | grep Layer7 | grep OK | grep WARNING Commit-ID , min-max time for +-95% check durations , comment e4d7c9d , 6 - 9 ms , all tests pass (1 tests out of +- a hundred showed 29ms , none below 6ms and almost half of them show 7ms) 6ec902a , 11 - 150 ms , of the 12 tests that passed e186161 , 5 - 8 ms , all tests pass (1 test used 15 ms, more than half the tests show 5ms check duration the majority of the remainder show 6ms) I'm not sure if this deserves further investigation at the moment, i think it does not. Thanks for spending your weekend on this :) that wasn't my intention. Regards, PiBa-NL (Pieter)
slow healthchecks after dev6+ with added commit "6ec902a MINOR: threads: serialize threads initialization"
Hi Willy, After the commit "6ec902a MINOR: threads: serialize threads initialization" however i have failing / slow health checks in the tls_health_checks.vtc test. Before that the Layer7-OK takes 5ms after this commit the healthcheck takes up to 75ms or even more.. It causes the 20ms connect/server timeouts to also to fail the test fairly often but not always.. Seems like something isn't quite right there. Can you check? Regards, PiBa-NL (Pieter) Log can be seen below (p.s. i added milliseconds output also to the vtest log.. ): *** h2 0.299 debug|[WARNING] 157/231456 (78988) : Health check for server be2/srv1 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 149ms, status: 1/1 UP. ## With HTX * top 0.000 TEST ./work/haproxy-6ec902a/reg-tests/checks/tls_health_checks.vtc starting top 0.000 extmacro def pwd=/usr/ports/net/haproxy-devel top 0.000 extmacro def no-htx= top 0.000 extmacro def localhost=127.0.0.1 top 0.000 extmacro def bad_backend=127.0.0.1 19775 top 0.000 extmacro def bad_ip=192.0.2.255 top 0.000 macro def testdir=/usr/ports/net/haproxy-devel/./work/haproxy-6ec902a/reg-tests/checks top 0.000 macro def tmpdir=/tmp/vtc.78981.41db79fd ** top 0.000 === varnishtest "Health-check test over TLS/SSL" * top 0.000 VTEST Health-check test over TLS/SSL ** top 0.000 === feature ignore_unknown_macro ** top 0.000 === server s1 { ** s1 0.000 Starting server s1 0.000 macro def s1_addr=127.0.0.1 s1 0.000 macro def s1_port=19776 s1 0.000 macro def s1_sock=127.0.0.1 19776 * s1 0.000 Listen on 127.0.0.1 19776 ** top 0.001 === server s2 { ** s2 0.001 Starting server s2 0.001 macro def s2_addr=127.0.0.1 s2 0.001 macro def s2_port=19777 s2 0.001 macro def s2_sock=127.0.0.1 19777 * s2 0.001 Listen on 127.0.0.1 19777 ** top 0.002 === syslog S1 -level notice { ** S1 0.002 Starting syslog server S1 0.002 macro def S1_addr=127.0.0.1 S1 0.002 macro def S1_port=14641 S1 0.002 macro def S1_sock=127.0.0.1 14641 * S1 0.002 Bound on 127.0.0.1 14641 ** s2 0.002 Started on 127.0.0.1 19777 (1 iterations) ** s1 0.002 Started on 127.0.0.1 19776 (1 iterations) ** top 0.002 === haproxy h1 -conf { ** S1 0.002 Started on 127.0.0.1 14641 (level: 5) ** S1 0.002 === recv h1 0.007 macro def h1_cli_sock=::1 19778 h1 0.007 macro def h1_cli_addr=::1 h1 0.007 macro def h1_cli_port=19778 h1 0.007 setenv(cli, 8) h1 0.007 macro def h1_fe1_sock=::1 19779 h1 0.007 macro def h1_fe1_addr=::1 h1 0.007 macro def h1_fe1_port=19779 h1 0.007 setenv(fe1, 9) h1 0.007 macro def h1_fe2_sock=::1 19780 h1 0.007 macro def h1_fe2_addr=::1 h1 0.007 macro def h1_fe2_port=19780 h1 0.007 setenv(fe2, 10) ** h1 0.007 haproxy_start h1 0.007 opt_worker 0 opt_daemon 0 opt_check_mode 0 h1 0.007 argv|exec "haproxy" -d -f "/tmp/vtc.78981.41db79fd/h1/cfg" h1 0.007 conf| global h1 0.007 conf|\tstats socket "/tmp/vtc.78981.41db79fd/h1/stats.sock" level admin mode 600 h1 0.007 conf| stats socket "fd@${cli}" level admin h1 0.007 conf| h1 0.007 conf| global h1 0.007 conf| tune.ssl.default-dh-param 2048 h1 0.007 conf| h1 0.007 conf| defaults h1 0.007 conf| mode http h1 0.007 conf| timeout client 20 h1 0.007 conf| timeout server 20 h1 0.007 conf| timeout connect 20 h1 0.007 conf| h1 0.007 conf| backend be1 h1 0.007 conf| server srv1 127.0.0.1:19776 h1 0.007 conf| h1 0.007 conf| backend be2 h1 0.007 conf| server srv2 127.0.0.1:19777 h1 0.007 conf| h1 0.007 conf| frontend fe1 h1 0.007 conf| option httplog h1 0.007 conf| log 127.0.0.1:14641 len 2048 local0 debug err h1 0.007 conf| bind "fd@${fe1}" ssl crt /usr/ports/net/haproxy-devel/./work/haproxy-6ec902a/reg-tests/checks/common.pem h1 0.007 conf| use_backend be1 h1 0.007 conf| h1 0.007 conf| frontend fe2 h1 0.007 conf| option tcplog h1 0.007 conf| bind "fd@${fe2}" ssl crt /usr/ports/net/haproxy-devel/./work/haproxy-6ec902a/reg-tests/checks/common.pem h1 0.007 conf| use_backend be2 h1 0.007 XXX 12 @637 *** h1 0.008 PID: 78985 h1 0.008 macro def h1_pid=78985 h1 0.008 macro def h1_name=/tmp/vtc.78981.41db79fd/h1 ** top 0.008 === syslog S2 -level notice { ** S2 0.008 Starting syslog server S2 0.008 macro def S2_addr=127.0.0.1 S2 0.008 macro def S2_port=35409 S
Re: haproxy 2.0-dev5-a689c3d - A bogus STREAM [0x805547500] is spinning at 100000 calls per second and refuses to die, aborting now!
Hi Willy, Op 7-6-2019 om 9:03 schreef Willy Tarreau: Hi again Pieter, On Tue, Jun 04, 2019 at 04:59:06PM +0200, Willy Tarreau wrote: Whatever the values, no single stream should be woken up 100k times per second or it definitely indicates a bug (spinning loop that leads to reports of 100% CPU)! I'll see if I can get something out of this. So just for the record, this is expected to be fixed in dev6 (it's the major change there). I'm interested in your feedback on this one, of course! Willy The stream does not spin anymore with dev6 so that seems to work alright. Thanks. Regards, PiBa-NL (Pieter)
Re: 2.0-dev5-ea8dd94 - conn_fd_handler() - dumps core - Program terminated with signal 11, Segmentation fault.
Hi Olivier, Op 6-6-2019 om 18:20 schreef Olivier Houchard: Hi Pieter, On Wed, Jun 05, 2019 at 09:00:22PM +0200, PiBa-NL wrote: Hi Olivier, It seems this commit ea8dd94 broke something for my FreeBSD11 system. Before that commit (almost) all vtest's succeed. After it several cause core-dumps. (and keep doing that including the current HEAD: 03abf2d ) Can you take a look at the issue? Below in this mail are the following: - gdb# bt full of one of the crashed tests.. - summary of failed tests Regards, PiBa-NL (Pieter) Indeed, there were a few issues. I know pushed enough patches so that I only fail one reg test, which also failed before the offending commit (reg-tests/compression/basic.vtc). Can you confirm it's doing better for you too ? Thanks ! Olivier Looks better for me :). Testing with haproxy version: 2.0-dev5-7b3a79f 0 tests failed, 0 tests skipped, 36 tests passed This includes the /compression/basic.vtc for me. p.s. This result doesn't "always" happen. But at least it seems 'just as good' as before ea8dd94. For example i still see this on my tests: 1 tests failed, 0 tests skipped, 35 tests passed ## Gathering results ## ## Test case: ./work/haproxy-7b3a79f/reg-tests/http-rules/converters_ipmask_concat_strcmp_field_word.vtc ## ## test results in: "/tmp/haregtests-2019-06-06_20-22-18.5Z9PR6/vtc.4579.056b0e93" c1 0.3 EXPECT resp.status (504) == "200" failed But then another test run of the same binary again says '36 passed'.. so it seems some tests are rather timing sensitive, or maybe a other variable doesn't play nice.. Anyhow the core-dump as reported is fixed. Ill try and find why the testresults are a bit inconsistent when running them repeatedly.. Anyhow ill send a new mail for that if i find something conclusive :). Thanks, PiBa-NL (Pieter)
2.0-dev5-ea8dd94 - conn_fd_handler() - dumps core - Program terminated with signal 11, Segmentation fault.
Hi Olivier, It seems this commit ea8dd94 broke something for my FreeBSD11 system. Before that commit (almost) all vtest's succeed. After it several cause core-dumps. (and keep doing that including the current HEAD: 03abf2d ) Can you take a look at the issue? Below in this mail are the following: - gdb# bt full of one of the crashed tests.. - summary of failed tests Regards, PiBa-NL (Pieter) gdb --core /tmp/haregtests-2019-06-05_20-40-20.7ZSvbo/vtc.65353.510907b0/h1/haproxy.core ./work/haproxy-ea8dd94/haproxy GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `/usr/ports/net/haproxy-devel/work/haproxy-ea8dd94/haproxy -d -f /tmp/haregtests-'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libcrypt.so.5...done. Loaded symbols for /lib/libcrypt.so.5 Reading symbols from /lib/libz.so.6...done. Loaded symbols for /lib/libz.so.6 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /usr/lib/libssl.so.8...done. Loaded symbols for /usr/lib/libssl.so.8 Reading symbols from /lib/libcrypto.so.8...done. Loaded symbols for /lib/libcrypto.so.8 Reading symbols from /usr/local/lib/liblua-5.3.so...done. Loaded symbols for /usr/local/lib/liblua-5.3.so Reading symbols from /lib/libm.so.5...done. Loaded symbols for /lib/libm.so.5 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 conn_fd_handler (fd=55) at src/connection.c:201 201 conn->mux->wake && conn->mux->wake(conn) < 0) (gdb) bt full #0 conn_fd_handler (fd=55) at src/connection.c:201 conn = (struct connection *) 0x8027e7000 flags = 0 io_available = 1 #1 0x005fe3f2 in fdlist_process_cached_events (fdlist=0xa66ac0) at src/fd.c:452 fd = 55 old_fd = 55 e = 39 locked = 0 #2 0x005fdefc in fd_process_cached_events () at src/fd.c:470 No locals. #3 0x0051c15d in run_poll_loop () at src/haproxy.c:2553 next = 0 wake = 0 #4 0x00519ba7 in run_thread_poll_loop (data=0x0) at src/haproxy.c:2607 ptaf = (struct per_thread_alloc_fct *) 0x94f158 ptif = (struct per_thread_init_fct *) 0x94f168 ptdf = (struct per_thread_deinit_fct *) 0x7fffe640 ptff = (struct per_thread_free_fct *) 0x610596 #5 0x0051620d in main (argc=4, argv=0x7fffe648) at src/haproxy.c:3286 blocked_sig = {__bits = 0x7fffe398} old_sig = {__bits = 0x7fffe388} i = 16 err = 0 retry = 200 limit = {rlim_cur = 234908, rlim_max = 234909} errmsg = 0x7fffe550 "" pidfd = -1 Current language: auto; currently minimal ## Starting vtest ## Testing with haproxy version: 2.0-dev5-ea8dd94 # top TEST reg-tests/lua/txn_get_priv.vtc FAILED (0.308) exit=2 # top TEST reg-tests/ssl/wrong_ctx_storage.vtc FAILED (0.308) exit=2 # top TEST reg-tests/compression/lua_validation.vtc FAILED (0.433) exit=2 # top TEST reg-tests/checks/tls_health_checks.vtc TIMED OUT (kill -9) # top TEST reg-tests/checks/tls_health_checks.vtc FAILED (20.153) signal=9 # top TEST reg-tests/peers/tls_basic_sync_wo_stkt_backend.vtc TIMED OUT (kill -9) # top TEST reg-tests/peers/tls_basic_sync_wo_stkt_backend.vtc FAILED (20.137) signal=9 # top TEST reg-tests/peers/tls_basic_sync.vtc TIMED OUT (kill -9) # top TEST reg-tests/peers/tls_basic_sync.vtc FAILED (20.095) signal=9 # top TEST reg-tests/connection/proxy_protocol_random_fail.vtc TIMED OUT (kill -9) # top TEST reg-tests/connection/proxy_protocol_random_fail.vtc FAILED (20.195) signal=9 7 tests failed, 0 tests skipped, 29 tests passed ## Gathering results ## ## Test case: reg-tests/compression/lua_validation.vtc ## ## test results in: "/tmp/haregtests-2019-06-05_20-40-20.7ZSvbo/vtc.65353.510907b0" top 0.4 shell_exit not as expected: got 0x0001 wanted 0x h1 0.4 Bad exit status: 0x008b exit 0x0 signal 11 core 128 ## Test case: reg-tests/peers/tls_basic_sync.vtc ## ## test results in: "/tmp/haregtests-2019-06-05_20-40-20.7ZSvbo/vtc.65353.5e9ca234" ## Test case: reg-tests/connection/proxy_protocol_random_fail.vtc ## ## test results in: "/tmp/haregtests-2019-06-05_20-40-20.7ZSvbo/vtc.65353.19403d36"
Re: How to allow Client Requests at a given rate
choServer.cpp:117] > current rate : 2488 It looks like me its +- exactly the configured 10 requests that got allowed above in that minute summing up the rate numbers listed above. >>> until almost 60 no http request are received to back ends >> this time gap varies with every run ... >>> after 60 secs rate limits are applied properly >>>> E0422 11:00:07.690192 18653 EchoServer.cpp:117] > current rate : 1 E0422 11:00:10.411736 18653 EchoServer.cpp:117] > current rate : 1 E0422 11:00:11.412317 18653 EchoServer.cpp:117] > current rate : 1679 E0422 11:00:12.412369 18653 EchoServer.cpp:117] > current rate : 1667 E0422 11:00:13.451706 18653 EchoServer.cpp:117] > current rate : 1668 E0422 11:00:14.453778 18653 EchoServer.cpp:117] > current rate : 1668 E0422 11:00:15.457597 18653 EchoServer.cpp:117] > current rate : 1645 E0422 11:00:16.458938 18653 EchoServer.cpp:117] > current rate : 1762 E0422 11:00:17.470010 18653 EchoServer.cpp:117] > current rate : 1598 Can I get some info on the issue, is this know issue or am I missing some config for rate limiting to be applied properly ? Thanks in advance, Badari I wonder if instead of allowing 10 requests per minute you would like 1666 requests to be allowed per second.? Which should effectively be similar besides that 'bursts' of requests will be blocked sooner.. To do this use 1s instead of 1m for the 'http_req_rate(1m)'. and put the 1666 as a limit in the map file... Still you might see a burst of 1000 requests in the first millisecond, and only 666 allowed in the other 999 milliseconds (theoretically.?.). But also its probably not really relevant on which ms a request is allowed or blocked. you could argue that allowing 2 requests per millisecond would achieve almost the desired benchmark result. But then if there is nothing to do, and a few 10 users send a request at the same millisecond you might block 8... while the server has actually little to do... and though managing this on a millisecond level is likely ridiculous its just to make it a bit more clear that a short 'burst' of requests isn't necessarily bad and that requests arn't always expected to come in at all the same speed.. So depending on expected runtime of a request and when the server will start to have trouble the current 10/minute might be perfectly fine.. or make it a 1 per 10 seconds.?. So to sum things up.. the limiting is working, and its allowing 10 request in the first minute, just as specified. So in that regard its working correctly already.. Regards, PiBa-NL (Pieter)
Re: DNS Resolver Issues
Hi Daniel, Baptiste, @Daniel, can you remove the 'addr loadbalancer-internal.xxx.yyy' from the server check? It seems to me that that name is not being resolved by the 'resolvers'. And even if it would it would be kinda redundant as it is in the example as it is the same as the servername.?. Not sure how far below scenarios are all explained by this though.. @Baptiste, is it intentional that a wrong 'addr' dns name makes haproxy fail to start despite having the supposedly never failing 'default-server init-addr last,libc,none' ? Is it possibly a good feature request to support re-resolving a dns name for the addr setting as well ? Regards, PiBa-NL (Pieter) Op 21-3-2019 om 20:37 schreef Daniel Schneller: Hi! Thanks for the response. I had looked at the "hold" directives, but since they all seem to have reasonable defaults, I did not touch them. I specified 10s explictly, but it did not make a difference. I did some more tests, however, and it seems to have more to do with the number of responses for the initial(?) DNS queries. Hopefully these three tables make sense and don't get mangled in the mail. The "templated" proxy is defined via "server-template" with 3 "slots". The "regular" one just as "server". Test 1: Start out with both "valid" and "broken" DNS entries. Then comment out/add back one at a time as described in (1)-(5). Each time after changing /etc/hosts, restart dnsmasq and check haproxy via hatop. Haproxy started fresh once dnsmasq was set up to (1). | state state /etc/hosts | regular templated |- (1) BRK| UP/L7OK DOWN/L4TOUT VALID | MAINT/resolution | UP/L7OK | (2) BRK| DOWN/L4TOUT DOWN/L4TOUT #VALID | MAINT/resolution | MAINT/resolution | (3) #BRK | UP/L7OK UP/L7OK VALID | MAINT/resolution | MAINT/resolution | (4) BRK| UP/L7OK UP/L7OK VALID | DOWN/L4TOUT | MAINT/resolution | (5) BRK| DOWN/L4TOUT DOWN/L4TOUT #VALID | MAINT/resolution | MAINT/resolution This all looks normal and as expected. As soon as the "VALID" DNS entry is present, the UP state follows within a few seconds. Test 2: Start out "valid only" (1) and proceed as described in (2)-(5), again restarting dnsmasq each time, and haproxy reloaded after dnsmasq was set up to (1). | state state /etc/hosts | regular templated | (1) #BRK | UP/L7OK MAINT/resolution VALID | MAINT/resolution | UP/L7OK | (2) BRK| UP/L7OK DOWN/L4TOUT VALID | MAINT/resolution | UP/L7OK | (3) #BRK | UP/L7OK MAINT/resolution VALID | MAINT/resolution | UP/L7OK | (4) BRK| UP/L7OK DOWN/L4TOUT VALID | MAINT/resolution | UP/L7OK | (5) BRK| DOWN/L4TOUT DOWN/L4TOUT #VALID | MAINT/resolution | MAINT/resolution Everything good here, too. Adding the broken DNS entry does not bring the proxies down until only the broken one is left. Test 3: Start out "broken only" (1). Again, same as before, haproxy restarted once dnsmasq was initialized to (1). | state state /etc/hosts | regular templated | (1) BRK
Re: haproxy reverse proxy to https streaming backend
Hi Thomas, Op 15-3-2019 om 15:24 schreef Thomas Schmiedl: Hello Pieter, thanks for your help, it works well now. The regex solution was my only idea, because I'm not a developer. I know, the haproxy workaround isn't the best solution, but nobody would fix the xupnpd2 hls-handling. Maybe you could help me again. I see, that the playlist has 2 "states" ("header" tags). #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-TARGETDURATION:50 #EXT-X-DISCONTINUITY and #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE: #EXT-X-TARGETDURATION:2 The result from the lua-script (header tags) should always be: #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE: #EXT-X-TARGETDURATION:2 Something like this might do the trick? Just a 'fixed' header as the result with only the 'found' media sequence number inserted ?: local data = [=[ #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:01234 #EXT-X-TARGETDURATION:50 #EXT-X-DISCONTINUITY ]=] local mediaseq_dummy,mediasequence,mediaseq_eol = string.match(data, "(#EXT[-]X[-]MEDIA[-]SEQUENCE:)(%d+)(\n)") local result = [=[ #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:]=]..mediasequence..[=[ #EXT-X-TARGETDURATION:2 ]=] print("Result:\n"..result) Result: #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:01234 #EXT-X-TARGETDURATION:2 Thanks, Thomas Am 15.03.2019 um 00:27 schrieb PiBa-NL: Hi Thomas, Op 14-3-2019 om 20:28 schreef Thomas Schmiedl: Hello, I never got a reply from the original author of xupnpd2 to fix the hls-handling, so I created a lua-script (thanks to Thierry Fournier), but it's too slow for the router cpu. Could someone rewrite the script to a lua-c-module? I don't think making this exact code a lua-c-module would solve the issue, lua is not a 'slow' language. But I do wonder if regex is the right tool for data manipulation.. Regards, Thomas test.cfg: global lua-load /var/media/ftp/playlist.lua frontend main mode http bind *:8080 acl is_index_m3u8 path -m end /index.m3u8 http-request use-service lua.playlist if is_index_m3u8 default_backend forward backend forward mode http server gjirafa puma.gjirafa.com:443 ssl verify none playlist.lua: core.register_service("playlist", "http", function(applet) local tcp = core.tcp() tcp:connect_ssl("51.75.52.73", 443) tcp:send("GET ".. applet.path .." HTTP/1.1\r\nConnection: Close\r\nHost: puma.gjirafa.com\r\n\r\n") local body = tcp:receive("*a") local result = string.match(body,"^.*(#EXTM3U.-)#EXTINF") result = result .. string.match(body,"(...%d+.ts%d+.ts%d+.ts)[\r\n|0]*$") I think a 'easier' regex might already improve performance, can you try this one for example ?: result = result .. string.match(body,"(#EXTINF:%d+[/.]%d+,\n%d+[/.]ts.#EXTINF:%d[/.]%d%d%d,.%d+[/.]ts.#EXTINF:%d+[/.]%d+,\n%d+[/.]ts)[\r\n|0]*$") With my test using 'https://rextester.com/l/lua_online_compiler' and a little sample m3u8 it seemed to work faster anyhow. applet:set_status(200) applet:add_header("Content-Type", "application/x-mpegURL") applet:add_header("content-length", string.len(result)) applet:add_header("Connection", "close") applet:start_response() applet:send(result) end) Am 19.02.2019 um 21:31 schrieb Thomas Schmiedl: Am 19.02.2019 um 05:29 schrieb Willy Tarreau: Hello Thomas, On Sun, Feb 17, 2019 at 05:55:29PM +0100, Thomas Schmiedl wrote: Hello Bruno, I think the problem is the parsing of the .m3u8-playlist in xupnpd2. The first entry to the .ts-file is 4 hours behind the actual time. But I have no c++ experience to change the code. For me if it works but not correctly like this, it clearly indicates there is a (possibly minor) incompatibility between the client and the server. It just happens that if your client doesn't support https, it was never tested against this server and very likely needs to be adapted to work correctly. Is it possible in haproxy to manipulate the playlist file (server response), that only the last .ts-entries will be available and returned to xupnpd2? No, haproxy doesn't manipulate contents. Not only it's completely out of the scope of a load balancing proxy, but it would also encourage some users to try to work around some of their deployment issues in the ugliest possible way, causing even more trouble (and frankly, on *every* infrastructure where you find such horrible tricks deployed, the admins implore you to help them because they're in big trouble and are stuck with no option left to fix the issues they've created). If it's only a matter of modifying one file on the
Re: haproxy reverse proxy to https streaming backend
Hi Thomas, Op 14-3-2019 om 20:28 schreef Thomas Schmiedl: Hello, I never got a reply from the original author of xupnpd2 to fix the hls-handling, so I created a lua-script (thanks to Thierry Fournier), but it's too slow for the router cpu. Could someone rewrite the script to a lua-c-module? I don't think making this exact code a lua-c-module would solve the issue, lua is not a 'slow' language. But I do wonder if regex is the right tool for data manipulation.. Regards, Thomas test.cfg: global lua-load /var/media/ftp/playlist.lua frontend main mode http bind *:8080 acl is_index_m3u8 path -m end /index.m3u8 http-request use-service lua.playlist if is_index_m3u8 default_backend forward backend forward mode http server gjirafa puma.gjirafa.com:443 ssl verify none playlist.lua: core.register_service("playlist", "http", function(applet) local tcp = core.tcp() tcp:connect_ssl("51.75.52.73", 443) tcp:send("GET ".. applet.path .." HTTP/1.1\r\nConnection: Close\r\nHost: puma.gjirafa.com\r\n\r\n") local body = tcp:receive("*a") local result = string.match(body,"^.*(#EXTM3U.-)#EXTINF") result = result .. string.match(body,"(...%d+.ts%d+.ts%d+.ts)[\r\n|0]*$") I think a 'easier' regex might already improve performance, can you try this one for example ?: result = result .. string.match(body,"(#EXTINF:%d+[/.]%d+,\n%d+[/.]ts.#EXTINF:%d[/.]%d%d%d,.%d+[/.]ts.#EXTINF:%d+[/.]%d+,\n%d+[/.]ts)[\r\n|0]*$") With my test using 'https://rextester.com/l/lua_online_compiler' and a little sample m3u8 it seemed to work faster anyhow. applet:set_status(200) applet:add_header("Content-Type", "application/x-mpegURL") applet:add_header("content-length", string.len(result)) applet:add_header("Connection", "close") applet:start_response() applet:send(result) end) Am 19.02.2019 um 21:31 schrieb Thomas Schmiedl: Am 19.02.2019 um 05:29 schrieb Willy Tarreau: Hello Thomas, On Sun, Feb 17, 2019 at 05:55:29PM +0100, Thomas Schmiedl wrote: Hello Bruno, I think the problem is the parsing of the .m3u8-playlist in xupnpd2. The first entry to the .ts-file is 4 hours behind the actual time. But I have no c++ experience to change the code. For me if it works but not correctly like this, it clearly indicates there is a (possibly minor) incompatibility between the client and the server. It just happens that if your client doesn't support https, it was never tested against this server and very likely needs to be adapted to work correctly. Is it possible in haproxy to manipulate the playlist file (server response), that only the last .ts-entries will be available and returned to xupnpd2? No, haproxy doesn't manipulate contents. Not only it's completely out of the scope of a load balancing proxy, but it would also encourage some users to try to work around some of their deployment issues in the ugliest possible way, causing even more trouble (and frankly, on *every* infrastructure where you find such horrible tricks deployed, the admins implore you to help them because they're in big trouble and are stuck with no option left to fix the issues they've created). If it's only a matter of modifying one file on the fly, you may manage to do it using Lua : instead of forwarding the request to the server, you send it to a Lua function, which itself makes the request to the server, buffers the response, rewrites it, then sends it back to the client. You must just make sure to only send there the requests for the playlist file and nothing else. Could someone send me such a lua-script example and how to include in haproxy. Thanks I personally think this is ugly compared to trying to fix the faulty client. Maybe you can report your issue to the author(s) and share your config to help them reproduce it ? Regards, Willy Regards, PiBa-NL (Pieter)
regtest, response lenght check failure for /reg-tests/http-capture/h00000.vtc with HTX enabled, using 2.0-dev1
Hi List, Christopher, With 2.0-dev1-6c1b667 and 2.0-dev1-12a7184 i get the 'failure' below when running reg-tests with HTX enabled. (without HTX the test passes) Seems this commit made it return different results: http://git.haproxy.org/?p=haproxy.git;a=commit;h=b8d2ee040aa21f2906a4921e5e1c7afefb7e I 'think' the syslog output for a single request/response should remain the same with/without htx? Or should the size check be less strict or accept 1 of 2 possible outcomes with/without htx.? Regards, PiBa-NL (Pieter) S 0.0 syslog|<134>Feb 26 20:42:52 haproxy[56313]: ::1:46091 [26/Feb/2019:20:42:52.065] fe be/srv 0/0/0/2/2 200 17473 - - 1/1/0/0/0 0/0 {HPhx8n59qjjNBLjP} {htb56qDdCcbRVTfS} "GET / HTTP/1.1" ** S 0.0 === expect ~ "[^:\\[ ]\\[${h_pid}\\]: .* .* fe be/srv .* 200 176... S 0.0 EXPECT FAILED ~ "[^:\[ ]\[56313\]: .* .* fe be/srv .* 200 17641 - - .* .* {HPhx8n59qjjNBLjP} {htb56qDdCcbRVTfS} "GET / HTTP/1\.1""
Re: h1-client to h2-server host header / authority conversion failure.?
Hi Willy, Op 2-2-2019 om 0:01 schreef Willy Tarreau: On Fri, Feb 01, 2019 at 09:43:13PM +0100, PiBa-NL wrote: The 'last' part is in TCP mode, and is intended like that to allow me to run tcpdump/wireshark on the un-encrypted traffic, and being certain that haproxy would not modify it before sending. But maybe the test contained a 'half done' edit as well, ill attach a new test now. OK. I've not tried 1.9 .. but did try with '2.0-dev0-ff5dd74 2019/01/31', that should contain the fix as well right.?. I just rechecked and no, these ones were added after ff5dd74 : 9c9da5e MINOR: muxes: Don't bother to LIST_DEL(&conn->list) before calling conn_ dc21ff7 MINOR: debug: Add an option that causes random allocation failures. 3c4e19f BUG/MEDIUM: backend: always release the previous connection into its own 3e45184 BUG/MEDIUM: htx: check the HTX compatibility in dynamic use-backend rule 9c4f08a BUG/MINOR: tune.fail-alloc: Don't forget to initialize ret. 1da41ec BUG/MINOR: backend: check srv_conn before dereferencing it 5be92ff BUG/MEDIUM: mux-h2: always omit :scheme and :path for the CONNECT method 053c157 BUG/MEDIUM: mux-h2: always set :authority on request output 32211a1 BUG/MEDIUM: stream: Don't forget to free s->unique_id in stream_free(). It's 053c157 which fixes it. You scared me, I thought I had messed up with the commit :-) I tested again here and it still works for me. Cheers, Willy Sorry, indeed all 4 tests pass. ( Using 2.0-dev0-32211a1 2019/02/01 ) I must have mixed the git-id to sync up with in my makefile, thought i picked the last one.. Sorry for the noise! Thanks for fixing and re-checking :) Regards, PiBa-NL (Pieter)
Re: h1-client to h2-server host header / authority conversion failure.?
Hi Willy, List, Just a little check, was below mail received properly with the 6 attachments (vtc/vtc/log/png/png/pcapng) .? (As it didn't show up on the mail-archive.) Regards, PiBa-NL (Pieter) Op 26-1-2019 om 21:04 schreef PiBa-NL: Hi Willy, Op 25-1-2019 om 17:04 schreef Willy Tarreau: Hi Pieter, On Fri, Jan 25, 2019 at 01:01:19AM +0100, PiBa-NL wrote: Hi List, Attached a regtest which i 'think' should pass. ** s1 0.0 === expect tbl.dec[1].key == ":authority" s1 0.0 EXPECT tbl.dec[1].key (host) == ":authority" failed It seems to me the Host <> Authority conversion isn't happening properly.? But maybe i'm just making a mistake in the test case... I was using HA-Proxy version 2.0-dev0-f7a259d 2019/01/24 with this test. The test was inspired by the attempt to connect to mail google com , as discussed in the "haproxy 1.9.2 with boringssl" mail thread.. Not sure if this is the main problem, but it seems suspicious to me.. It's not as simple, :authority is only required for CONNECT and is optional for other methods with Host as a fallback. Clients are encouraged to use it instead of the Host header field, according to paragraph 8.1.2.3, but there is nothing indicating that a gateway may nor should build one from scratch when translating HTTP/1.1 to HTTP/2. In fact the authority part is generally not present in the URIs we receive as a gateway, so what we'd put there would be completely reconstructed from the host header field. I don't even know if all servers are fine with authority only instead of Host. Please note, I'm not against changing this, I just want to be sure we actually fix something and that we don't break anything. Thus if you have any info indicating there is an issue with this one missing, it could definitely help. Thanks! Willy Today ive given it another shot. (connecting to mail google com). Is there a way in haproxy to directly 'manipulate' the h2 headers? Setting h2 header with set-header :authority didn't seem to work.? See attached some logs a packetcapture and a vtc that uses google's servers itself. It seems google replies "Header: :status: 400 Bad Request" But leaves me 'guessing' why it would be invalid, also the 'body' doesn't get downloaded but haproxy terminates the connection, which curl then reports as missing bytes.. There are a few differences between the 2 get requests, authority and scheme.. But i also wonder if that is the actual packet with the issue, H2 isnt quite a simple as H1 used to be ;). Also with "h2-client-mail google vtc" the first request succeeds, but the second where the Host header is used fails. I think this shows there is a 'need' for the :authority header to be present? Or i mixed something up... p.s. Wireshark doesnt nicely show/dissect the http2 requests made by vtest, probably because for example the first magic packet is spread out over multiple tcp packets, is there a way to make it send them in 1 go, or make haproxy 'buffer' the short packets into a bigger complete packets, i tried putting a little listen/bind/server section in the request path, but it just forwarded the small packets as is.. Regards, PiBa-NL (Pieter)
h1-client to h2-server host header / authority conversion failure.?
Hi List, Attached a regtest which i 'think' should pass. ** s1 0.0 === expect tbl.dec[1].key == ":authority" s1 0.0 EXPECT tbl.dec[1].key (host) == ":authority" failed It seems to me the Host <> Authority conversion isn't happening properly.? But maybe i'm just making a mistake in the test case... I was using HA-Proxy version 2.0-dev0-f7a259d 2019/01/24 with this test. The test was inspired by the attempt to connect to mail.google.com , as discussed in the "haproxy 1.9.2 with boringssl" mail thread.. Not sure if this is the main problem, but it seems suspicious to me.. Regards, PiBa-NL (Pieter) varnishtest "Check H1 client to H2 server with HTX." feature ignore_unknown_macro syslog Slog_1 -repeat 1 -level info { recv } -start server s1 -repeat 2 { rxpri stream 0 { txsettings rxsettings txsettings -ack } -run stream 1 { rxreq expect tbl.dec[1].key == ":authority" expect tbl.dec[1].value == "domain.tld" txresp } -run } -start haproxy h1 -conf { global log ${Slog_1_addr}:${Slog_1_port} len 2048 local0 debug err defaults mode http timeout client 2s timeout server 2s timeout connect 1s log global option http-use-htx frontend fe1 option httplog bind "fd@${fe1}" default_backend b1 backend b1 server s1 ${s1_addr}:${s1_port} proto h2 frontend fe2 option httplog bind "fd@${fe2}" proto h2 default_backend b2 backend b2 server s2 ${s1_addr}:${s1_port} proto h2 } -start client c1 -connect ${h1_fe1_sock} { txreq -url "/" -hdr "host: domain.tld" rxresp expect resp.status == 200 } -run client c2 -connect ${h1_fe2_sock} { txpri stream 0 { txsettings -hdrtbl 0 rxsettings } -run stream 1 { txreq -req GET -url /3 -litIdxHdr inc 1 huf "domain.tld" rxresp expect resp.status == 200 } -run } -run #syslog Slog_1 -wait
Re: haproxy 1.9.2 with boringssl
Hi Aleksandar, Just FYI. Op 22-1-2019 om 22:08 schreef Aleksandar Lazic: But this could be a know bug and is fixed in the current git - ## Starting vtest ## Testing with haproxy version: 1.9.2 #top TEST ./reg-tests/mailers/k_healthcheckmail.vtc FAILED (7.808) exit=2 1 tests failed, 0 tests skipped, 32 tests passed ## Gathering results ## ## Test case: ./reg-tests/mailers/k_healthcheckmail.vtc ## ## test results in: "/tmp/haregtests-2019-01-22_20-56-31.QMI0Ue/vtc.5740.39907fe1" c27.0 EXPECT resp.http.mailsreceived (11) == "16" failed This was indeed identified as a bug, and is fixed in current master. The impact of this was rather low though, and this specific issue of a few 'missing' mails under certain configuration circumstances existed for years before it was spotted with the regtest. https://www.mail-archive.com/haproxy@formilux.org/msg32190.html http://git.haproxy.org/?p=haproxy.git;a=commit;h=774c486cece942570b6a9d16afe236a16ee12079 Regards, PiBa-NL (Pieter)
Re: [PATCH] REG-TEST: mailers: add new test for 'mailers' section
Hi Christopher, Op 21-1-2019 om 15:28 schreef Christopher Faulet: Hi Pieter, About the timing issue, could you try the following patch please ? With it, I can run the regtest about email alerts without any error. Thanks, -- Christopher Faulet The regtest works for me as well with this patch. Without needing the 'timeout mail' setting. I think we can call it fixed once committed. Thanks, PiBa-NL (Pieter)
Re: stats webpage crash, htx and scope filter, [PATCH] REGTEST is included
Hi Willy, Christopher, Op 16-1-2019 om 17:32 schreef Willy Tarreau: On Wed, Jan 16, 2019 at 02:28:56PM +0100, Christopher Faulet wrote: here is a new patch, again. Willy, I hope it will be good for the release 1.9.2. This one works :). OK so I've mergd it now, thank you! Willy Op 14-1-2019 om 11:17 schreef Christopher Faulet: If it's ok for you, I'll also merge your regtest. Can you add the regtest as well into the git repo? Regards, PiBa-NL (Pieter)
Re: stats webpage crash, htx and scope filter, [PATCH] REGTEST is included
Hi Christopher, Op 15-1-2019 om 10:48 schreef Christopher Faulet: Le 14/01/2019 à 21:53, PiBa-NL a écrit : Hi Christopher, Op 14-1-2019 om 11:17 schreef Christopher Faulet: Le 12/01/2019 à 23:23, PiBa-NL a écrit : Hi List, I've configured haproxy with htx and when i try to filter the stats webpage. Sending this request: "GET /?;csv;scope=b1" to '2.0-dev0-762475e 2019/01/10' it will crash with the trace below. 1.9.0 and 1.9.1 are also affected. Can someone take a look? Thanks in advance. A regtest is attached that reproduces the behavior, and which i think could be included into the haproxy repository. Pieter, Here is the patch that should fix this issue. This was "just" an oversight when the stats applet has been adapted to support the HTX. If it's ok for you, I'll also merge your regtest. Thanks It seems the patch did not change/fix the crash.? Below looks pretty much the same as previously. Did i fail to apply the patch properly.? It seems to have 'applied' properly checking a few lines of the touched code manually. As for the regtest, yes please merge that if its okay as-is, perhaps after the fix is also ready :). Hi Pieter, Sorry, I made my patch too quickly. It seemed ok, but obviously not... This new one should do the trick. Well.. 'something' changed, still crashing though.. but at a different place. Regards, PiBa-NL (Pieter) Program terminated with signal SIGSEGV, Segmentation fault. #0 0x004d3770 in htx_sl_p2 (sl=0x0) at include/common/htx.h:237 237 return ist2(HTX_SL_P2_PTR(sl), HTX_SL_P2_LEN(sl)); (gdb) bt full #0 0x004d3770 in htx_sl_p2 (sl=0x0) at include/common/htx.h:237 No locals. #1 0x004d3665 in htx_sl_req_uri (sl=0x0) at include/common/htx.h:252 No locals. #2 0x004d1125 in stats_scope_ptr (appctx=0x802678540, si=0x8026416d8) at src/stats.c:268 req = 0x802641410 htx = 0x80271df80 uri = {ptr = 0x60932e "H\213E\320H\211E\370H\213E\370H\201\304\260", len = 4304914720} p = 0x4802631048 0x4802631048> #3 0x004d8505 in stats_send_htx_redirect (si=0x8026416d8, htx=0x8027c8e40) at src/stats.c:3162 scope_ptr = 0x5f80f5 <__pool_get_first+21> "H\211E\310H\203}\310" scope_txt = "\000\342\377\377\377\177\000\000\351}M\000\000\000\000\000x\024d\002\b\000\000\000x\024d\002" s = 0x802641400 uri = 0x802638000 appctx = 0x802678540 sl = 0x8027c8e40 flags = 8 #4 0x004d60fb in htx_stats_io_handler (appctx=0x802678540) at src/stats.c:3337 si = 0x8026416d8 s = 0x802641400 req = 0x802641410 res = 0x802641470 req_htx = 0x8027c8e40 res_htx = 0x8027c8e40 #5 0x004d2d36 in http_stats_io_handler (appctx=0x802678540) at src/stats.c:3393 si = 0x8026416d8 s = 0x802641400 req = 0x802641410 res = 0x802641470 #6 0x005f7d5f in task_run_applet (t=0x802656780, context=0x802678540, state=16385) at src/applet.c:85 app = 0x802678540 si = 0x8026416d8 #7 0x005f3023 in process_runnable_tasks () at src/task.c:435 t = 0x802656780 state = 16385 ctx = 0x802678540 process = 0x5f7cc0 t = 0x802656780 max_processed = 200 #8 0x00516ca2 in run_poll_loop () at src/haproxy.c:2620 next = 0 exp = 1394283990 #9 0x005138f8 in run_thread_poll_loop (data=0x8026310e8) at src/haproxy.c:2685 start_lock = 0 ptif = 0x936d40 ptdf = 0x0 #10 0x0050ff26 in main (argc=4, argv=0x7fffeb08) at src/haproxy.c:3314 tids = 0x8026310e8 threads = 0x8026310f0 i = 1 old_sig = {__bits = {0, 0, 0, 0}} blocked_sig = {__bits = {4227856759, 4294967295, 4294967295, 4294967295}} err = 0 retry = 200 limit = {rlim_cur = 4051, rlim_max = 4051} errmsg = "\000\353\377\377\377\177\000\000\060\353\377\377\377\177\000\000\b\353\377\377\377\177\000\000\004\000\000\000\000\000\000\000\376\310\311\070\333\207d\000`9\224\000\000\000\000\000\000\353\377\377\377\177\000\000\060\353\377\377\377\177\000\000\b\353\377\377\377\177\000\000\004\000\000\000\000\000\000\000\240\352\377\377\377\177\000\000R\201\000\002\b\000\000\000\001\000\000" pidfd = -1
Re: stats webpage crash, htx and scope filter, [PATCH] REGTEST is included
Hi Christopher, Op 14-1-2019 om 11:17 schreef Christopher Faulet: Le 12/01/2019 à 23:23, PiBa-NL a écrit : Hi List, I've configured haproxy with htx and when i try to filter the stats webpage. Sending this request: "GET /?;csv;scope=b1" to '2.0-dev0-762475e 2019/01/10' it will crash with the trace below. 1.9.0 and 1.9.1 are also affected. Can someone take a look? Thanks in advance. A regtest is attached that reproduces the behavior, and which i think could be included into the haproxy repository. Pieter, Here is the patch that should fix this issue. This was "just" an oversight when the stats applet has been adapted to support the HTX. If it's ok for you, I'll also merge your regtest. Thanks It seems the patch did not change/fix the crash.? Below looks pretty much the same as previously. Did i fail to apply the patch properly.? It seems to have 'applied' properly checking a few lines of the touched code manually. As for the regtest, yes please merge that if its okay as-is, perhaps after the fix is also ready :). Regards, PiBa-NL (Pieter) Program terminated with signal SIGSEGV, Segmentation fault. #0 0x005658e7 in strnistr (str1=0x802631048 "fe1", len_str1=3, str2=0x271dfcc , len_str2=3) at src/standard.c:3657 3657 while (toupper(*start) != toupper(*str2)) { (gdb) bt full #0 0x005658e7 in strnistr (str1=0x802631048 "fe1", len_str1=3, str2=0x271dfcc , len_str2=3) at src/standard.c:3657 pptr = 0x271dfcc sptr = 0x6995d3 "text/plain" start = 0x802631048 "fe1" slen = 3 plen = 3 tmp1 = 0 tmp2 = 4294958728 #1 0x004d09ff in stats_dump_proxy_to_buffer (si=0x8026416d8, htx=0x8027c8e40, px=0x8026b3c00, uri=0x802638000) at src/stats.c:2087 scope_ptr = 0x271dfcc 0x271dfcc> appctx = 0x802678380 s = 0x802641400 rep = 0x802641470 sv = 0x8027c8e40 svs = 0x343e1e0 l = 0x4d3a8f flags = 0 #2 0x004d49e9 in stats_dump_stat_to_buffer (si=0x8026416d8, htx=0x8027c8e40, uri=0x802638000) at src/stats.c:2664
Re: Get client IP
Hi, Op 13-1-2019 om 13:11 schreef Aleksandar Lazic: Hi. Am 13.01.2019 um 12:17 schrieb Vũ Xuân Học: Hi, Please help me to solve this problem. I use HAProxy version 1.5.18, SSL transparent mode and I can not get client IP in my .net mvc website. With mode http, I can use option forwardfor to catch client ip but with tcp mode, my web read X_Forwarded_For is null. My diagram: Client => Firewall => HAProxy => Web I read HAProxy document, try to use send-proxy. But when use send-proxy, I can access my web. This is my config: frontend test2233 bind *:2233 option forwardfor default_backend testecus backend testecus mode http server web1 192.168.0.151:2233 check Above config work, and I can get the client IP That's good as it's `mode http` therefore haproxy can see the http traffic. Indeed it can insert the http forwardfor header with 'mode http'. Config with SSL: frontend ivan bind 192.168.0.4:443 mode tcp option tcplog #option forwardfor reqadd X-Forwarded-Proto:\ https This can't work as you use `mode tcp` and therefore haproxy can't see the http traffic. From my point of view have you now 2 options. * use https termination on haproxy. Then you can add this http header. Thats one option indeed. * use accept-proxy in the bind line. This option requires that the firewall is able to send the PROXY PROTOCOL header to haproxy. https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#5.1-accept-proxy I dont expect a firewall to send such a header. And if i understand correctly the 'webserver' would need to be configured to accept proxy-protocol. The modification to make in haproxy would be to configure send-proxy[-v2-ssl-cn] http://cbonte.github.io/haproxy-dconv/1.9/snapshot/configuration.html#5.2-send-proxy And how to configure it with for example nginx: https://wakatime.com/blog/23-how-to-scale-ssl-with-haproxy-and-nginx The different modes are described in the doc https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4-mode Here is a blog post about basic setup of haproxy with ssl https://www.haproxy.com/blog/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/ acl tls req.ssl_hello_type 1 tcp-request inspect-delay 5s tcp-request content accept if tls # Define hosts acl host_1 req.ssl_sni -i ebh.vn acl host_2 req.ssl_sni hdr_end(host) -i einvoice.com.vn use_backend eBH if host_1 use_backend einvoice443 if host_2 backend eBH mode tcp balance roundrobin option ssl-hello-chk server web1 192.168.0.153:443 maxconn 3 check #cookie web1 server web1 192.168.0.154:443 maxconn 3 check #cookie web2 Above config doesn’t work, and I can not get the client ip. I try server web1 192.168.0.153:443 send-proxy and try server web1 192.168.0.153:443 send-proxy-v2 but I can’t access my web. This is expected as the Firewall does not send the PROXY PROTOCOL header and the bind line is not configured for that. Firewall's by themselves will never use proxy-protocol at all. That it doesn't work with send-proxy on the haproxy server line is likely because the webservice that is receiving the traffic isn't configured to accept the proxy protocol. How to configure a ".net mvc website" to accept that is something i don't know if it is even possible at all.. Many thanks, Best regards Aleks Thanks & Best Regards! * VU XUAN HOC Regards, PiBa-NL (Pieter)
stats webpage crash, htx and scope filter, [PATCH] REGTEST is included
Hi List, I've configured haproxy with htx and when i try to filter the stats webpage. Sending this request: "GET /?;csv;scope=b1" to '2.0-dev0-762475e 2019/01/10' it will crash with the trace below. 1.9.0 and 1.9.1 are also affected. Can someone take a look? Thanks in advance. A regtest is attached that reproduces the behavior, and which i think could be included into the haproxy repository. Regards, PiBa-NL (Pieter) Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00564fe7 in strnistr (str1=0x802631048 "fe1", len_str1=3, str2=0x804e3bf4c , len_str2=3) at src/standard.c:3657 3657 while (toupper(*start) != toupper(*str2)) { (gdb) bt full #0 0x00564fe7 in strnistr (str1=0x802631048 "fe1", len_str1=3, str2=0x804e3bf4c , len_str2=3) at src/standard.c:3657 pptr = 0x804e3bf4c 0x804e3bf4c> sptr = 0x80271df80 "\330?" start = 0x802631048 "fe1" slen = 3 plen = 3 tmp1 = 0 tmp2 = 4294959392 #1 0x004d01d3 in stats_dump_proxy_to_buffer (si=0x8026416d8, htx=0x8027c8e40, px=0x8026b3c00, uri=0x802638000) at src/stats.c:2079 appctx = 0x802678380 s = 0x802641400 rep = 0x802641470 sv = 0x8027c8e40 svs = 0x33be1e0 l = 0x4d31df flags = 0 #2 0x004d4139 in stats_dump_stat_to_buffer (si=0x8026416d8, htx=0x8027c8e40, uri=0x802638000) at src/stats.c:2652 appctx = 0x802678380 rep = 0x802641470 px = 0x8026b3c00 #3 0x004d56bb in htx_stats_io_handler (appctx=0x802678380) at src/stats.c:3299 si = 0x8026416d8 s = 0x802641400 req = 0x802641410 res = 0x802641470 req_htx = 0x8027c8e40 res_htx = 0x8027c8e40 #4 0x004d2546 in http_stats_io_handler (appctx=0x802678380) at src/stats.c:3367 si = 0x8026416d8 s = 0x802641400 req = 0x802641410 res = 0x802641470 #5 0x005f729f in task_run_applet (t=0x8026566e0, context=0x802678380, state=16385) at src/applet.c:85 app = 0x802678380 si = 0x8026416d8 #6 0x005f2533 in process_runnable_tasks () at src/task.c:435 t = 0x8026566e0 state = 16385 ctx = 0x802678380 process = 0x5f7200 t = 0x8026566e0 max_processed = 199 #7 0x005163b2 in run_poll_loop () at src/haproxy.c:2619 next = 0 exp = 1137019023 #8 0x00513008 in run_thread_poll_loop (data=0x8026310f0) at src/haproxy.c:2684 start_lock = 0 ptif = 0x935d40 ptdf = 0x0 #9 0x0050f636 in main (argc=4, argv=0x7fffeb08) at src/haproxy.c:3313 tids = 0x8026310f0 threads = 0x8026310f8 i = 1 old_sig = {__bits = {0, 0, 0, 0}} blocked_sig = {__bits = {4227856759, 4294967295, 4294967295, 4294967295}} err = 0 retry = 200 limit = {rlim_cur = 4052, rlim_max = 4052} errmsg = "\000\353\377\377\377\177\000\000\060\353\377\377\377\177\000\000\b\353\377\377\377\177\000\000\004\000\000\000\000\000\000\000t\240\220?\260|6\224`)\224\000\000\000\000\000\000\353\377\377\377\177\000\000\060\353\377\377\377\177\000\000\b\353\377\377\377\177\000\000\004\000\000\000\000\000\000\000\240\352\377\377\377\177\000\000R\201\000\002\b\000\000\000\001\000\000" pidfd = -1 From 838ecb4e153c1d859d0a49e0554ff050ff10033c Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Sat, 12 Jan 2019 21:57:48 +0100 Subject: [PATCH] REGTEST: checks basic stats webpage functionality This regtest verifies that the stats webpage can be used to change a server state to maintenance or drain, and that filtering the page scope will result in a filtered page. --- .../h_webstats-scope-and-post-change.vtc | 83 +++ 1 file changed, 83 insertions(+) create mode 100644 reg-tests/webstats/h_webstats-scope-and-post-change.vtc diff --git a/reg-tests/webstats/h_webstats-scope-and-post-change.vtc b/reg-tests/webstats/h_webstats-scope-and-post-change.vtc new file mode 100644 index ..a77483b5 --- /dev/null +++ b/reg-tests/webstats/h_webstats-scope-and-post-change.vtc @@ -0,0 +1,83 @@ +varnishtest "Webgui stats page check filtering with scope and changing server state" +#REQUIRE_VERSION=1.6 + +feature ignore_unknown_macro + +server s1 { +} -start + +haproxy h1 -conf { + global +stats socket /tmp/haproxy.socket level admin + + defaults +mode http +${no-htx} option http-use-htx + + frontend fe1 +bind "fd@${fe1}" +stats enable +stats refresh 5s +stats uri / +stats admin if TRUE + + backend b1 +server srv1 ${s1_addr}:${s1_port} +server srv2 ${s1_addr}:${s1_port} +server srv3 ${s1_addr}:${s1_port} + + backend b2 +server srv1 ${s1_addr}:${s1_port} +server srv2 ${s1_addr}:${s1_port} +
Re: Lots of mail from email alert on 1.9.x
Hi Willy, Olivier, Op 12-1-2019 om 13:11 schreef Willy Tarreau: Hi Pieter, it is needed to prepend this at the beginning of chk_report_conn_err() : if (!check->server) return; We need to make sure that check->server is properly tested everywhere. With a bit of luck this one was the only remnant. Thanks! Willy With the check above added, mail alerts seem to work properly here, or just as good as they used to anyhow. Once the patches and above addition get committed, that leaves the other 'low priority' issue of needing a short timeout to send the exact amount of 'expected' mails. EXPECT resp.http.mailsreceived (10) == "16" failed To be honest i only noticed it due to making the regtest, and double-checking what to expect.. When i validated mails on my actual environment it seems to work properly. (Though the server i took out to test has a health-check with a 60 second interval..) Anyhow its been like this for years afaik, i guess it wont matter much if stays like this a bit longer. Regards, PiBa-NL (Pieter)
Re: Lots of mail from email alert on 1.9.x
Hi Olivier, Op 11-1-2019 om 19:17 schreef Olivier Houchard: Ok so erm, I'd be lying if I claimed I enjoy working on the check code, or that I understand it fully. However, after talking with Willy and Christopher, I think I may have comed with an acceptable solution, and the attached patch should fix it (at least by getting haproxy to segfault, but it shouldn't mailbomb you anymore). Pieter, I'd be very interested to know if it still work with your setup. It's a different way of trying to fix what you tried ot fix with 1714b9f28694d750d446917672dd59c46e16afd7 I'd like to be sure I didn't break it for you again:) Regards, Olivier (Slightly modified patches, I think there were a potential race condition when running with multiple threads). Olivier Thanks for this 'change in behavior' ;). Indeed the mailbomb is fixed, and it seems the expected mails get generated and delivered, but a segfault also happens on occasion. Not with the regtest as it was, but with a few minor modifications (adding a unreachable mailserver, and giving it a little more time seems to be the most reliable reproduction a.t.m.) it will crash consistently after 11 seconds.. So i guess the patch needs a bit more tweaking. Regards, PiBa-NL (Pieter) Core was generated by `haproxy -d -f /tmp/vtc.37274.4b8a1a3a/h1/cfg'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00500955 in chk_report_conn_err (check=0x802616a10, errno_bck=0, expired=1) at src/checks.c:689 689 dns_trigger_resolution(check->server->dns_requester); (gdb) bt full #0 0x00500955 in chk_report_conn_err (check=0x802616a10, errno_bck=0, expired=1) at src/checks.c:689 cs = 0x8027de0c0 conn = 0x802683180 err_msg = 0x80266d0c0 " at step 1 of tcp-check (connect)" chk = 0x80097b848 step = 1 comment = 0x0 #1 0x005065a5 in process_chk_conn (t=0x802656640, context=0x802616a10, state=513) at src/checks.c:2261 check = 0x802616a10 proxy = 0x8026c3000 cs = 0x8027de0c0 conn = 0x802683180 rv = 0 ret = 0 expired = 1 #2 0x0050596e in process_chk (t=0x802656640, context=0x802616a10, state=513) at src/checks.c:2330 check = 0x802616a10 #3 0x004fe0a2 in process_email_alert (t=0x802656640, context=0x802616a10, state=513) at src/checks.c:3210 check = 0x802616a10 q = 0x802616a00 alert = 0x7fffe340 #4 0x005f2523 in process_runnable_tasks () at src/task.c:435 t = 0x802656640 state = 513 ctx = 0x802616a10 process = 0x4fdeb0 t = 0x8026566e0 max_processed = 200 #5 0x005163a2 in run_poll_loop () at src/haproxy.c:2619 next = 1062130135 exp = 1062129684 #6 0x00512ff8 in run_thread_poll_loop (data=0x8026310f0) at src/haproxy.c:2684 start_lock = 0 ptif = 0x935d40 ---Type to continue, or q to quit--- ptdf = 0x0 #7 0x0050f626 in main (argc=4, argv=0x7fffead8) at src/haproxy.c:3313 tids = 0x8026310f0 threads = 0x8026310f8 i = 1 old_sig = {__bits = {0, 0, 0, 0}} blocked_sig = {__bits = {4227856759, 4294967295, 4294967295, 4294967295}} err = 0 retry = 200 limit = {rlim_cur = 4046, rlim_max = 4046} errmsg = "\000\352\377\377\377\177\000\000\000\353\377\377\377\177\000\000\330\352\377\377\377\177\000\000\004\000\000\000\000\000\000\00 0\b\250\037\315})5:`)\224\000\000\000\000\000\320\352\377\377\377\177\000\000\000\353\377\377\377\177\000\000\330\352\377\377\377\177\000\000\004 \000\000\000\000\000\00 reg-tests/mailers/k_healthcheckmail.vtc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/reg-tests/mailers/k_healthcheckmail.vtc b/reg-tests/mailers/k_healthcheckmail.vtc index d3af3589..820191c8 100644 --- a/reg-tests/mailers/k_healthcheckmail.vtc +++ b/reg-tests/mailers/k_healthcheckmail.vtc @@ -48,6 +48,7 @@ defaults # timeout mail 20s # timeout mail 200ms mailer smtp1 ${h1_femail_addr}:${h1_femail_port} + mailer smtp2 ipv4@192.0.2.100:1025 } -start @@ -62,7 +63,7 @@ client c1 -connect ${h1_luahttpservice_sock} { delay 2 server s2 -repeat 5 -start -delay 5 +delay 10 client c2 -connect ${h1_luahttpservice_sock} { timeout 2
Re: Lots of mail from email alert on 1.9.x
Hi Johan, Olivier, Willy, Op 10-1-2019 om 17:00 schreef Johan Hendriks: I just updated to 1.9.1 on my test system. We noticed that when a server fails we now get tons of mail, and with tons we mean a lot. After a client backend server fails we usually get 1 mail on 1.8.x now with 1.9.1 within 1 minute we have the following. mailq | grep -B2 l...@testdomain.nl | grep '^[A-F0-9]' | awk '{print $1}' | sed 's/*//' | postsuper -d - postsuper: Deleted: 19929 messages My setting from the backend part is as follows. email-alert mailers alert-mailers email-alert from l...@testdomain.nl email-alert to not...@testdomain.nl server webserver09 11.22.33.44:80 check Has something changed in 1.9.x (it was on 1.9.0 also) regards Johan Hendriks Its a 'known issue' see: https://www.mail-archive.com/haproxy@formilux.org/msg32290.html a 'regtest' is added in that mail thread also to aid developers in reproducing the issue and validating a possible fix. @Olivier, Willy, may i assume this mailbomb feature is 'planned' to get fixed in 1.9.2 ? (perhaps a bugtracker with a 'target version' would be nice ;) ?) Regards, PiBa-NL (Pieter)
Re: [PATCH] REGTEST: filters: add compression test
Thank you Christopher & Frederic. Op 9-1-2019 om 14:47 schreef Christopher Faulet: Le 09/01/2019 à 10:43, Frederic Lecaille a écrit : On 1/8/19 11:25 PM, PiBa-NL wrote: Hi Frederic, Hi Pieter, Op 7-1-2019 om 10:13 schreef Frederic Lecaille: On 12/23/18 11:38 PM, PiBa-NL wrote: As requested hereby the regtest send for inclusion into the git repository. It is OK like that. Note that you patch do not add reg-test/filters/common.pem which could be a symlink to ../ssl/common.pem. Also note that since 8f16148Christopher's commit, we add such a line where possible: ${no-htx} option http-use-htx We should also rename your test files to reg-test/filters/h0.* Thank you. Fred. Together with these changes you have supplied me already off-list, i've also added a " --max-time 15" for the curl request, that should be sufficient for most systems to complete the 3 second testcase, and allows the shell command to complete without varnishtest killing it after a timeout and not showing any of the curl output.. One last question, currently its being added to a new folder: reg-test/filters/ , perhaps it should be in reg-test/compression/ ? If you agree that needs changing i guess that can be done upon committing it? I have modified your patch to move your new files to reg-test/compression. I have also applied this to it ;) : 's/\r$//' Note that the test fails on my FreeBSD system when using HTX when using '2.0-dev0-251a6b7 2019/01/08', i'm not aware it ever worked (i didn't test it with HTX before..). top 15.2 shell_out|curl: (28) Operation timed out after 15036 milliseconds with 187718 bytes received Ok, I will take some time to have a look at this BSD specific issue. Note that we can easily use the CLI at the end of the script to troubleshooting anything. Log attached.. Would it help to log it with the complete "filter trace name BEFORE / filter compression / filter trace name AFTER" ? Or are there other details i could try and gather? I do not fill at ease enough on compression/filter topics to reply to your question Pieter ;) Nevertheless I think your test deserve to be merged. *The patch to be merged is attached to this mail*. Thank a lot Pieter. Thanks Fred and Pieter, now merged. I've just updated the patch to add the list of required options in the VTC file. Hereby just a little confirmation that this works well now :) also in my tests. Regards, PiBa-NL (Pieter)
coredump in h2_process_mux with 1.9.0-8223050
Hi List, Willy, Got a coredump of 1.9.0-8223050 today, see below. Would this be 'likely' the same one with the 'PRIORITY' that 1.9.1 fixes? I don't have any idea what the exact circumstance request/response was.. Anyhow i updated my system to 2.0-dev0-251a6b7 for the moment, lets see if something strange happens again. Might take a few days though, IF it still occurs.. Regards, PiBa-NL (Pieter) Core was generated by `/usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x004b91c7 in h2_process_mux (h2c=0x802657480) at src/mux_h2.c:2434 2434 src/mux_h2.c: No such file or directory. (gdb) bt full #0 0x004b91c7 in h2_process_mux (h2c=0x802657480) at src/mux_h2.c:2434 h2s = 0x80262c7a0 h2s_back = 0x80262ca40 #1 0x004b844d in h2_send (h2c=0x802657480) at src/mux_h2.c:2560 flags = 0 conn = 0x8026dc300 done = 0 sent = 1 #2 0x004b8a49 in h2_process (h2c=0x802657480) at src/mux_h2.c:2640 conn = 0x8026dc300 #3 0x004b32e1 in h2_wake (conn=0x8026dc300) at src/mux_h2.c:2715 h2c = 0x802657480 #4 0x005c8158 in conn_fd_handler (fd=7) at src/connection.c:190 conn = 0x8026dc300 flags = 0 io_available = 0 #5 0x005e3c7c in fdlist_process_cached_events (fdlist=0x9448f0 ) at src/fd.c:441 fd = 7 old_fd = 7 e = 117 #6 0x005e377c in fd_process_cached_events () at src/fd.c:459 No locals. #7 0x00514296 in run_poll_loop () at src/haproxy.c:2655 next = 762362654 exp = 762362654 #8 0x00510b78 in run_thread_poll_loop (data=0x802615970) at src/haproxy.c:2684 start_lock = 0 ptif = 0x92ed10 ptdf = 0x0 #9 0x0050d1a6 in main (argc=6, argv=0x7fffec60) at src/haproxy.c:3313 tids = 0x802615970 threads = 0x802615998 i = 1 old_sig = {__bits = {0, 0, 0, 0}} blocked_sig = {__bits = {4227856759, 4294967295, 4294967295, 4294967295}} err = 0 retry = 200 limit = {rlim_cur = 2040, rlim_max = 2040} errmsg = "\000\354\377\377\377\177\000\000\230\354\377\377\377\177\000\000`\354\377\377\377\177\000\000\006\000\000\000\000\000\000\000\f\373\353\230\373\032\351~\240\270\223\000\000\000\000\000X\354\377\377\377\177\000\000\230\354\377\377\377\177\000\000`\354\377\377\377\177\000\000\006\000\000\000\000\000\000\000\000\354\377\377\377\177\000\000\302z\000\002\b\000\000\000\001\000\000" pidfd = 17 (gdb)
Re: [PATCH] REGTEST: filters: add compression test
Hi Frederic, Op 7-1-2019 om 10:13 schreef Frederic Lecaille: On 12/23/18 11:38 PM, PiBa-NL wrote: As requested hereby the regtest send for inclusion into the git repository. It is OK like that. Note that you patch do not add reg-test/filters/common.pem which could be a symlink to ../ssl/common.pem. Also note that since 8f16148Christopher's commit, we add such a line where possible: ${no-htx} option http-use-htx We should also rename your test files to reg-test/filters/h0.* Thank you. Fred. Together with these changes you have supplied me already off-list, i've also added a " --max-time 15" for the curl request, that should be sufficient for most systems to complete the 3 second testcase, and allows the shell command to complete without varnishtest killing it after a timeout and not showing any of the curl output.. One last question, currently its being added to a new folder: reg-test/filters/ , perhaps it should be in reg-test/compression/ ? If you agree that needs changing i guess that can be done upon committing it? Note that the test fails on my FreeBSD system when using HTX when using '2.0-dev0-251a6b7 2019/01/08', i'm not aware it ever worked (i didn't test it with HTX before..). top 15.2 shell_out|curl: (28) Operation timed out after 15036 milliseconds with 187718 bytes received Log attached.. Would it help to log it with the complete "filter trace name BEFORE / filter compression / filter trace name AFTER" ? Or are there other details i could try and gather? Regards, PiBa-NL (Pieter) From 793e770b399157a1549a2655612a29845b165dd6 Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Sun, 23 Dec 2018 21:21:51 +0100 Subject: [PATCH] REGTEST: filters: add compression test This test checks that data transferred with compression is correctly received at different download speeds --- reg-tests/filters/common.pem | 1 + reg-tests/filters/s0.lua | 19 ++ reg-tests/filters/s0.vtc | 59 3 files changed, 79 insertions(+) create mode 12 reg-tests/filters/common.pem create mode 100644 reg-tests/filters/s0.lua create mode 100644 reg-tests/filters/s0.vtc diff --git a/reg-tests/filters/common.pem b/reg-tests/filters/common.pem new file mode 12 index ..a4433d56 --- /dev/null +++ b/reg-tests/filters/common.pem @@ -0,0 +1 @@ +../ssl/common.pem \ No newline at end of file diff --git a/reg-tests/filters/s0.lua b/reg-tests/filters/s0.lua new file mode 100644 index ..2cc874b9 --- /dev/null +++ b/reg-tests/filters/s0.lua @@ -0,0 +1,19 @@ + +local data = "abcdefghijklmnopqrstuvwxyz" +local responseblob = "" +for i = 1,1 do + responseblob = responseblob .. "\r\n" .. i .. data:sub(1, math.floor(i % 27)) +end + +http01applet = function(applet) + local response = responseblob + applet:set_status(200) + applet:add_header("Content-Type", "application/javascript") + applet:add_header("Content-Length", string.len(response)*10) + applet:start_response() + for i = 1,10 do +applet:send(response) + end +end + +core.register_service("fileloader-http01", "http", http01applet) diff --git a/reg-tests/filters/s0.vtc b/reg-tests/filters/s0.vtc new file mode 100644 index ..231344a6 --- /dev/null +++ b/reg-tests/filters/s0.vtc @@ -0,0 +1,59 @@ +# Checks that compression doesnt cause corruption.. + +varnishtest "Compression validation" +#REQUIRE_VERSION=1.6 + +feature ignore_unknown_macro + +haproxy h1 -conf { +global +# log stdout format short daemon + lua-load${testdir}/s0.lua + +defaults + modehttp + log global + ${no-htx} option http-use-htx + option httplog + +frontend main-https + bind"fd@${fe1}" ssl crt ${testdir}/common.pem + compression algo gzip + compression type text/html text/plain application/json application/javascript + compression offload + use_backend TestBack if TRUE + +backend TestBack + server LocalSrv ${h1_fe2_addr}:${h1_fe2_port} + +listen fileloader + mode http + bind "fd@${fe2}" + http-request use-service lua.fileloader-http01 +} -start + +shell { +HOST=${h1_fe1_addr} +if [ "${h1_fe1_addr}" = "::1" ] ; then +HOST="\[::1\]" +fi + +md5=$(which md5 || which md5sum) + +if [ -z $md5 ] ; then +echo "MD5 checksum utility not found" +exit 1 +fi + +expectchecksum="4d9c62aa5370b8d5f84f17ec2e78f483" + +for opt in "" "--limit-rate 300K" "--limit-rate 500K" ; do +checksum=$(curl --max-time 15 --compressed -k "https://$HOST:${h1_fe1_port}"; $opt | $m
Re: regtests - with option http-use-htx
Hi Frederic, Op 8-1-2019 om 16:27 schreef Frederic Lecaille: On 12/15/18 4:52 PM, PiBa-NL wrote: Hi List, Willy, Trying to run some existing regtests with added option: option http-use-htx Using: HA-Proxy version 1.9-dev10-c11ec4a 2018/12/15 I get the below issues sofar: based on /reg-tests/connection/b0.vtc Takes 8 seconds to pass, in a slightly modified manor 1.1 > 2.0 expectation for syslog. This surely needs a closer look? # top TEST ./htx-test/connection-b0.vtc passed (8.490) based on /reg-tests/stick-table/b1.vtc Difference here is the use=1 vs use=0 , maybe that is better, but then the 'old' expectation seems wrong, and the bug is the case without htx? Note that the server s1 never responds. Furthermore, c1 client is run with -run argument. This means that we wait for its termination before running accessing CLI. Then we check that there is no consistency issue with the stick-table: if the entry has expired we get only this line: table: http1, type: ip, size:1024, used:0 if not we get these two lines: table: http1, type: ip, size:1024, used:1 .* use=0 ... here used=1 means there is still an entry in the stick-table, and use=0 means it is not currently in use (I guess this is because the client has closed its connection). I do not reproduce your issue with this script both on Linux and FreeBSD 11 both with or without htx. Did you try with the 'old' development version (1.9-dev10-c11ec4a 2018/12/15), i think in current version its already fixed see my own test results also below. h1 0.0 CLI recv|0x8026612c0: key=127.0.0.1 use=1 exp=0 gpt0=0 gpc0=0 gpc0_rate(1)=0 conn_rate(1)=1 http_req_cnt=1 http_req_rate(1)=1 http_err_cnt=0 http_err_rate(1)=0 h1 0.0 CLI recv| h1 0.0 CLI expect failed ~ "table: http1, type: ip, size:1024, used:(0|1\n0x[0-9a-f]*: key=127\.0\.0\.1 use=0 exp=[0-9]* gpt0=0 gpc0=0 gpc0_rate\(1\)=0 conn_rate\(1\)=1 http_req_cnt=1 http_req_rate\(1\)=1 http_err_cnt=0 http_err_rate\(10000\)=0)\n$" Regards, PiBa-NL (Pieter) I tried again today with both 2.0-dev0-251a6b7 and 1.9.0-8223050 and 1.9-dev10-c11ec4a : HA-Proxy version 2.0-dev0-251a6b7 2019/01/08 - https://haproxy.org/ ## Without HTX # top TEST ./PB-TEST/2018/connection-b0.vtc passed (0.146) # top TEST ./PB-TEST/2018/stick-table-b1.vtc passed (0.127) 0 tests failed, 0 tests skipped, 2 tests passed ## With HTX # top TEST ./PB-TEST/2018/connection-b0.vtc passed (0.147) # top TEST ./PB-TEST/2018/stick-table-b1.vtc passed (0.127) 0 tests failed, 0 tests skipped, 2 tests passed HA-Proxy version 1.9.0-8223050 2018/12/19 - https://haproxy.org/ ## Without HTX # top TEST ./PB-TEST/2018/connection-b0.vtc passed (0.150) # top TEST ./PB-TEST/2018/stick-table-b1.vtc passed (0.128) 0 tests failed, 0 tests skipped, 2 tests passed ## With HTX # top TEST ./PB-TEST/2018/connection-b0.vtc passed (0.148) # top TEST ./PB-TEST/2018/stick-table-b1.vtc passed (0.127) 0 tests failed, 0 tests skipped, 2 tests passed HA-Proxy version 1.9-dev10-c11ec4a 2018/12/15 Copyright 2000-2018 Willy Tarreau ## Without HTX # top TEST ./PB-TEST/2018/connection-b0.vtc passed (0.146) # top TEST ./PB-TEST/2018/stick-table-b1.vtc passed (0.127) 0 tests failed, 0 tests skipped, 2 tests passed ## With HTX # top TEST ./PB-TEST/2018/connection-b0.vtc passed (8.646) * top 0.0 TEST ./PB-TEST/2018/stick-table-b1.vtc starting h1 0.0 CLI recv|# table: http1, type: ip, size:1024, used:1 h1 0.0 CLI recv|0x80262a200: key=127.0.0.1 use=1 exp=0 gpt0=0 gpc0=0 gpc0_rate(1)=0 conn_rate(1)=1 http_req_cnt=1 http_req_rate(1)=1 http_err_cnt=0 http_err_rate(1)=0 h1 0.0 CLI recv| h1 0.0 CLI expect failed ~ "table: http1, type: ip, size:1024, used:(0|1\n0x[0-9a-f]*: key=127\.0\.0\.1 use=0 exp=[0-9]* gpt0=0 gpc0=0 gpc0_rate\(1\)=0 conn_rate\(1\)=1 http_req_cnt=1 http_req_rate\(1\)=1 http_err_cnt=0 http_err_rate\(1\)=0)\n$" * top 0.0 RESETTING after ./PB-TEST/2018/stick-table-b1.vtc ** h1 0.0 Reset and free h1 haproxy 92940 # top TEST ./PB-TEST/2018/stick-table-b1.vtc FAILED (0.127) exit=2 1 tests failed, 0 tests skipped, 1 tests passed With the 'old' 1.9-dev10 version and with HTX i can still reproduce the "passed (8.646)" and "use=1".. But both 1.9.0 and 2.0-dev don't show that behavior. I have not 'bisected' further, but i don't think there is anything to do a.t.m. regarding this old (already fixed) issue. Regards, PiBa-NL (Pieter)
Re: compression in defaults happens twice with 1.9.0
Hi Christopher, Op 7-1-2019 om 16:32 schreef Christopher Faulet: Le 06/01/2019 à 16:22, PiBa-NL a écrit : Hi List, Using both 1.9.0 and 2.0-dev0-909b9d8 compression happens twice when configured in defaults. This was noticed by user walle303 on IRC. Seems like a bug to me as 1.8.14 does not show this behavior. Attached a little regtest that reproduces the issue. Can someone take a look, thanks in advance. Hi Pieter, Here is the patch that should fix this issue. Could you confirm please ? Thanks Works for me. Thanks! Regards, PiBa-NL (Pieter)
Re: [PATCH] REG-TEST: mailers: add new test for 'mailers' section
Hi Willy, Op 7-1-2019 om 15:25 schreef Willy Tarreau: Hi Pieter, On Sun, Jan 06, 2019 at 04:38:21PM +0100, PiBa-NL wrote: The 23654 mails received for a failed server is a bit much.. I agree. I really don't know much how the mails work to be honest, as I have never used them. I remember that we reused a part of the tcp-check infrastructure because by then it offered a convenient way to proceed with send/expect sequences. Maybe there's something excessive in the sequence there, such as a certain status code being expected at the end while the mail succeeds, I don't know. Given that this apparently has always been broken, For 1 part its always been broken (needing the short mailer timeout to send all expected mails), for the other part, at least until 1.8.14 it used to NOT send thousands of mails so that would be a regression in the current 1.9 version that should get fixed on a shorter term. I'm hesitant between merging this in the slow category or the broken one. My goal with "broken" was to keep the scripts that trigger broken behaviours that need to be addressed, rather than keep broken scripts. Indeed keeping broken scripts wouldn't be help-full in the long run, unless there is still the intent to fix them. It isn't what the makefile says about 'LEVEL 5' though. It says its for 'broken scripts' and to quickly disable them, not as you write here for scripts that show 'broken haproxy behavior'. My goal is to make sure we never consider it normal to have failures in the regular test suite, otherwise you know how it becomes, just like compiler warnings, people say "oh I didn't notice this new error in the middle of all other ones". Agreed, though i will likely fall into repeat some day, apology in advance ;).. I guess we could 'fix' the regtest by specifying the 'timeout mail 200', that would fix it for 1.7 and 1.8.. And might help for 1.9 regressiontests and to get it fixed to at least not send thousands of mails. We might forget about the short time requirement then though, which seems strange as well. And the test wouldn't be 1.6 compatible as it doesn't have that setting at all. Thus probably the best thing to do is to use it at level 5 so that it's easier to work on the bug without triggering false positives when doing regression testing. What's your opinion ? With a changed description for 'level 5' being 'shows broken haproxy behavior, to be fixed in a future release' i think it would fit in there nicely. Can you change the starting letter of the .vtc test (and the .lua and reference to that) to 'k' during committing? Or shall i re-send it? p.s. What do you think about the 'naming' of the test? 'k_healthcheckmail.vtc' or 'k0.vtc' personally i don't think the 'numbering' of tests makes them easier to use.?. thanks, Willy Regards, PiBa-NL (Pieter)
Re: [PATCH] REG-TEST: mailers: add new test for 'mailers' section
Hi, 2 weeks passed without reply, so a little hereby a little 'bump'.. I know everyone has been busy, but would be nice to get test added or at least the biggest issue of the 'mailbomb' fixed before next release. If its 'scheduled' to get looked at later thats okay. Just making sure it aint forgotten about :). The 23654 mails received for a failed server is a bit much.. c2 7.5 EXPECT resp.http.mailsreceived (23654) == "16" failed Regards, PiBa-NL (Pieter) Op 23-12-2018 om 23:37 schreef PiBa-NL: Changed subject of patch requirement to 'REGTEST'. Op 23-12-2018 om 21:17 schreef PiBa-NL: Hi List, Attached a new test to verify that the 'mailers' section is working properly. Currently with 1.9 the mailers sends thousands of mails for my setup... As the test is rather slow i have marked it with a starting letter 's'. Note that the test also fails on 1.6/1.7/1.8 but can be 'fixed' there by adding a 'timeout mail 200ms'.. (except on 1.6 which doesn't have that setting.) I don't think that should be needed though if everything was working properly? If the test could be committed, and related issues exposed fixed that would be neat ;) Thanks in advance, PiBa-NL (Pieter)
compression in defaults happens twice with 1.9.0
Hi List, Using both 1.9.0 and 2.0-dev0-909b9d8 compression happens twice when configured in defaults. This was noticed by user walle303 on IRC. Seems like a bug to me as 1.8.14 does not show this behavior. Attached a little regtest that reproduces the issue. Can someone take a look, thanks in advance. Regards, PiBa-NL (Pieter) s1 0.0 txresp|!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_ s1 0.0 txresp|"#$%&'()*+,-./0123456789:;<=>?@ABCD *** s1 0.0 shutting fd 4 ** s1 0.0 Ending *** h1 0.0 debug|:b1.srvrep[000a:adfd]: HTTP/1.1 200 OK *** h1 0.0 debug|:b1.srvhdr[000a:adfd]: Content-Length: 100 *** h1 0.0 debug|:b1.srvcls[000a:adfd] c1 0.0 rxhdr|HTTP/1.1 200 OK\r c1 0.0 rxhdr|Content-Encoding: gzip\r c1 0.0 rxhdr|Transfer-Encoding: chunked\r c1 0.0 rxhdr|Co57\r c1 0.0 rxhdr|\037\213\010 c1 0.0 rxhdrlen = 78 c1 0.0 http[ 0] |HTTP/1.1 c1 0.0 http[ 1] |200 c1 0.0 http[ 2] |OK c1 0.0 http[ 3] |Content-Encoding: gzip c1 0.0 http[ 4] |Transfer-Encoding: chunked c1 0.0 http[ 5] |Co57 c1 0.0 http[ 6] |\037\213\010 c1 10.2 HTTP rx timeout (fd:7 1 ms) # Checks compression defined in defaults doesnt happen twice varnishtest "Compression in defaults" feature ignore_unknown_macro server s1 { rxreq txresp -bodylen 100 } -start haproxy h1 -conf { defaults mode http compression algo gzip frontend fe1 bind "fd@${fe_1}" default_backend b1 backend b1 server srv1 ${s1_addr}:${s1_port} } -start client c1 -connect ${h1_fe_1_sock} { txreq -url "/" -hdr "Accept-Encoding: gzip" rxresp expect resp.status == 200 expect resp.http.content-encoding == "gzip" expect resp.http.transfer-encoding == "chunked" gunzip expect resp.bodylen == 100 } -run server s1 -wait
Re: htx with compression issue, "Gunzip error: Body lacks gzip magics"
Hi Christopher, Willy, Op 2-1-2019 om 15:37 schreef Christopher Faulet: Le 29/12/2018 à 01:29, PiBa-NL a écrit : compression with htx, and a slightly delayed body content it will prefix some rubbish and corrupt the gzip header.. Hi Pieter, In fact, It is not a bug related to the compression. But a pure HTX one, about the defragmentation when we need space to store data. Here is a patch. It fixes the problem for me. Okay so the compression somehow 'triggers' this defragmentation to happen, are there simpler ways to make that happen 'on demand' ? Willy, if it is ok for you, I can merge it in upstream and backport it in 1.9. -- Christopher Faulet The patch fixes the reg-test for me as well, I guess its good to go :). Thanks. Regards, PiBa-NL (Pieter)
htx with compression issue, "Gunzip error: Body lacks gzip magics"
Hi List, When using compression with htx, and a slightly delayed body content it will prefix some rubbish and corrupt the gzip header.. Below output i get with attached test.. Removing http-use-htx 'fixes' the test. This happens with both 1.9.0 and todays commit a2dbeb2, not sure if this ever worked before.. c1 0.1 len|1A\r c1 0.1 chunk|\222\7\0\0\0\377\377\213\10\0\0\0\0\0\4\3JLJN\1\0\0\0\377\377 c1 0.1 len|0\r c1 0.1 bodylen = 26 ** c1 0.1 === expect resp.status == 200 c1 0.1 EXPECT resp.status (200) == "200" match ** c1 0.1 === expect resp.http.content-encoding == "gzip" c1 0.1 EXPECT resp.http.content-encoding (gzip) == "gzip" match ** c1 0.1 === gunzip c1 0.1 Gunzip error: Body lacks gzip magics Can someone take a look? Thanks in advance. Regards, PiBa-NL (Pieter) # Checks htx with compression and a short delay between headers and data send by the server varnishtest "Connection counters check" feature ignore_unknown_macro server s1 { rxreq txresp -nolen -hdr "Content-Length: 4" delay 0.05 send "abcd" } -start haproxy h1 -conf { global stats socket /tmp/haproxy.socket level admin defaults mode http option http-use-htx frontend fe1 bind "fd@${fe1}" compression algo gzip #filter trace name BEFORE-HTTP-COMP #filter compression #filter trace name AFTER-HTTP-COMP default_backend b1 backend b1 server srv1 ${s1_addr}:${s1_port} } -start # configure port for lua to call fe4 client c1 -connect ${h1_fe1_sock} { txreq -url "/" -hdr "Accept-Encoding: gzip" rxresp expect resp.status == 200 expect resp.http.content-encoding == "gzip" gunzip expect resp.body == "abcd" } -run
Re: [PATCH] REGTEST: filters: add compression test
Added LUA requirement into the test.. Op 23-12-2018 om 23:05 schreef PiBa-NL: Hi Frederic, As requested hereby the regtest send for inclusion into the git repository. Without randomization and with your .diff applied. Also outputting expected and actual checksum if the test fails so its clear that that is the issue detected. Is it okay like this? Should the blob be bigger? As you mentioned needing a 10MB output to reproduce the original issue on your machine? Regards, PiBa-NL (Pieter) From 29c3b9d344f360503bcd30f48558ca8a51df92ed Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Sun, 23 Dec 2018 21:21:51 +0100 Subject: [PATCH] REGTEST: filters: add compression test This test checks that data transferred with compression is correctly received at different download speeds --- reg-tests/filters/b5.lua | 19 reg-tests/filters/b5.vtc | 59 2 files changed, 78 insertions(+) create mode 100644 reg-tests/filters/b5.lua create mode 100644 reg-tests/filters/b5.vtc diff --git a/reg-tests/filters/b5.lua b/reg-tests/filters/b5.lua new file mode 100644 index ..6dbe1d33 --- /dev/null +++ b/reg-tests/filters/b5.lua @@ -0,0 +1,19 @@ + +local data = "abcdefghijklmnopqrstuvwxyz" +local responseblob = "" +for i = 1,1 do + responseblob = responseblob .. "\r\n" .. i .. data:sub(1, math.floor(i % 27)) +end + +http01applet = function(applet) + local response = responseblob + applet:set_status(200) + applet:add_header("Content-Type", "application/javascript") + applet:add_header("Content-Length", string.len(response)*10) + applet:start_response() + for i = 1,10 do +applet:send(response) + end +end + +core.register_service("fileloader-http01", "http", http01applet) diff --git a/reg-tests/filters/b5.vtc b/reg-tests/filters/b5.vtc new file mode 100644 index ..2f4982cb --- /dev/null +++ b/reg-tests/filters/b5.vtc @@ -0,0 +1,59 @@ +# Checks that compression doesnt cause corruption.. + +varnishtest "Compression validation" +#REQUIRE_VERSION=1.6 +#REQUIRE_OPTIONS=LUA + +feature ignore_unknown_macro + +haproxy h1 -conf { +global +# log stdout format short daemon + lua-load${testdir}/b5.lua + +defaults + modehttp + log global + option httplog + +frontend main-https + bind"fd@${fe1}" ssl crt ${testdir}/common.pem + compression algo gzip + compression type text/html text/plain application/json application/javascript + compression offload + use_backend TestBack if TRUE + +backend TestBack + server LocalSrv ${h1_fe2_addr}:${h1_fe2_port} + +listen fileloader + mode http + bind "fd@${fe2}" + http-request use-service lua.fileloader-http01 +} -start + +shell { +HOST=${h1_fe1_addr} +if [ "${h1_fe1_addr}" = "::1" ] ; then +HOST="\[::1\]" +fi + +md5=$(which md5 || which md5sum) + +if [ -z $md5 ] ; then +echo "MD5 checksum utility not found" +exit 1 +fi + +expectchecksum="4d9c62aa5370b8d5f84f17ec2e78f483" + +for opt in "" "--limit-rate 300K" "--limit-rate 500K" ; do +checksum=$(curl --compressed -k "https://$HOST:${h1_fe1_port}"; $opt | $md5 | cut -d ' ' -f1) +if [ "$checksum" != "$expectchecksum" ] ; then + echo "Expecting checksum $expectchecksum" + echo "Received checksum: $checksum" + exit 1; +fi +done + +} -run -- 2.18.0.windows.1
Re: [PATCH] REG-TEST: mailers: add new test for 'mailers' section
Changed subject of patch requirement to 'REGTEST'. Op 23-12-2018 om 21:17 schreef PiBa-NL: Hi List, Attached a new test to verify that the 'mailers' section is working properly. Currently with 1.9 the mailers sends thousands of mails for my setup... As the test is rather slow i have marked it with a starting letter 's'. Note that the test also fails on 1.6/1.7/1.8 but can be 'fixed' there by adding a 'timeout mail 200ms'.. (except on 1.6 which doesn't have that setting.) I don't think that should be needed though if everything was working properly? If the test could be committed, and related issues exposed fixed that would be neat ;) Thanks in advance, PiBa-NL (Pieter) From 8d63f5a39a9b4b326b636e42ccafcf0c2173d752 Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Sun, 23 Dec 2018 21:06:31 +0100 Subject: [PATCH] REGTEST: mailers: add new test for 'mailers' section This test verifies the mailers section works properly by checking that it sends the proper amount of mails when health-checks are changing and or marking a server up/down The test currently fails on all versions of haproxy i tried with varying results. 1.9.0 produces thousands of mails.. 1.8.14 only sends 1 mail, needs a 200ms 'timeout mail' to succeed 1.7.11 only sends 1 mail, needs a 200ms 'timeout mail' to succeed 1.6 only sends 1 mail, (does not have the 'timeout mail' setting implemented) --- reg-tests/mailers/shealthcheckmail.lua | 105 + reg-tests/mailers/shealthcheckmail.vtc | 75 ++ 2 files changed, 180 insertions(+) create mode 100644 reg-tests/mailers/shealthcheckmail.lua create mode 100644 reg-tests/mailers/shealthcheckmail.vtc diff --git a/reg-tests/mailers/shealthcheckmail.lua b/reg-tests/mailers/shealthcheckmail.lua new file mode 100644 index ..9c75877b --- /dev/null +++ b/reg-tests/mailers/shealthcheckmail.lua @@ -0,0 +1,105 @@ + +local vtc_port1 = 0 +local mailsreceived = 0 +local mailconnectionsmade = 0 +local healthcheckcounter = 0 + +core.register_action("bug", { "http-res" }, function(txn) + data = txn:get_priv() + if not data then + data = 0 + end + data = data + 1 + print(string.format("set to %d", data)) + txn.http:res_set_status(200 + data) + txn:set_priv(data) +end) + +core.register_service("luahttpservice", "http", function(applet) + local response = "?" + local responsestatus = 200 + if applet.path == "/setport" then + vtc_port1 = applet.headers["vtcport1"][0] + response = "OK" + end + if applet.path == "/svr_healthcheck" then + healthcheckcounter = healthcheckcounter + 1 + if healthcheckcounter < 2 or healthcheckcounter > 6 then + responsestatus = 403 + end + end + + applet:set_status(responsestatus) + if applet.path == "/checkMailCounters" then + response = "MailCounters" + applet:add_header("mailsreceived", mailsreceived) + applet:add_header("mailconnectionsmade", mailconnectionsmade) + end + applet:start_response() + applet:send(response) +end) + +core.register_service("fakeserv", "http", function(applet) + applet:set_status(200) + applet:start_response() +end) + +function RecieveAndCheck(applet, expect) + data = applet:getline() + if data:sub(1,expect:len()) ~= expect then + core.Info("Expected: "..expect.." but got:"..data:sub(1,expect:len())) + applet:send("Expected: "..expect.." but got:"..data.."\r\n") + return false + end + return true +end + +core.register_service("mailservice", "tcp", function(applet) + core.Info("# Mailservice Called #") + mailconnectionsmade = mailconnectionsmade + 1 + applet:send("220 Welcome\r\n") + local data + + if RecieveAndCheck(applet, "EHLO") == false then + return + end + applet:send("250 OK\r\n") + if RecieveAndCheck(applet, "MAIL FROM:") == false then + return + end + applet:send("250 OK\r\n") + if RecieveAndCheck(applet, "RCPT TO:") == false then + return + end + applet:send("250 OK\r\n") + if RecieveAndCheck(applet, "DATA") == false then + return + end + applet:send("354 OK\r\n") + core.Info(" Send your mailbody") + local endofmail = false +
[PATCH] REGTEST: filters: add compression test
Hi Frederic, As requested hereby the regtest send for inclusion into the git repository. Without randomization and with your .diff applied. Also outputting expected and actual checksum if the test fails so its clear that that is the issue detected. Is it okay like this? Should the blob be bigger? As you mentioned needing a 10MB output to reproduce the original issue on your machine? Regards, PiBa-NL (Pieter) From 64460dfeacef3d04af4243396007a606c2e5dbf7 Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Sun, 23 Dec 2018 21:21:51 +0100 Subject: [PATCH] REGTEST: filters: add compression test This test checks that data transferred with compression is correctly received at different download speeds --- reg-tests/filters/b5.lua | 19 reg-tests/filters/b5.vtc | 58 2 files changed, 77 insertions(+) create mode 100644 reg-tests/filters/b5.lua create mode 100644 reg-tests/filters/b5.vtc diff --git a/reg-tests/filters/b5.lua b/reg-tests/filters/b5.lua new file mode 100644 index ..6dbe1d33 --- /dev/null +++ b/reg-tests/filters/b5.lua @@ -0,0 +1,19 @@ + +local data = "abcdefghijklmnopqrstuvwxyz" +local responseblob = "" +for i = 1,1 do + responseblob = responseblob .. "\r\n" .. i .. data:sub(1, math.floor(i % 27)) +end + +http01applet = function(applet) + local response = responseblob + applet:set_status(200) + applet:add_header("Content-Type", "application/javascript") + applet:add_header("Content-Length", string.len(response)*10) + applet:start_response() + for i = 1,10 do +applet:send(response) + end +end + +core.register_service("fileloader-http01", "http", http01applet) diff --git a/reg-tests/filters/b5.vtc b/reg-tests/filters/b5.vtc new file mode 100644 index ..5216cdaf --- /dev/null +++ b/reg-tests/filters/b5.vtc @@ -0,0 +1,58 @@ +# Checks that compression doesnt cause corruption.. + +varnishtest "Compression validation" +#REQUIRE_VERSION=1.6 + +feature ignore_unknown_macro + +haproxy h1 -conf { +global +# log stdout format short daemon + lua-load${testdir}/b5.lua + +defaults + modehttp + log global + option httplog + +frontend main-https + bind"fd@${fe1}" ssl crt ${testdir}/common.pem + compression algo gzip + compression type text/html text/plain application/json application/javascript + compression offload + use_backend TestBack if TRUE + +backend TestBack + server LocalSrv ${h1_fe2_addr}:${h1_fe2_port} + +listen fileloader + mode http + bind "fd@${fe2}" + http-request use-service lua.fileloader-http01 +} -start + +shell { +HOST=${h1_fe1_addr} +if [ "${h1_fe1_addr}" = "::1" ] ; then +HOST="\[::1\]" +fi + +md5=$(which md5 || which md5sum) + +if [ -z $md5 ] ; then +echo "MD5 checksum utility not found" +exit 1 +fi + +expectchecksum="4d9c62aa5370b8d5f84f17ec2e78f483" + +for opt in "" "--limit-rate 300K" "--limit-rate 500K" ; do +checksum=$(curl --compressed -k "https://$HOST:${h1_fe1_port}"; $opt | $md5 | cut -d ' ' -f1) +if [ "$checksum" != "$expectchecksum" ] ; then + echo "Expecting checksum $expectchecksum" + echo "Received checksum: $checksum" + exit 1; +fi +done + +} -run -- 2.18.0.windows.1
[PATCH] REG-TEST: mailers: add new test for 'mailers' section
Hi List, Attached a new test to verify that the 'mailers' section is working properly. Currently with 1.9 the mailers sends thousands of mails for my setup... As the test is rather slow i have marked it with a starting letter 's'. Note that the test also fails on 1.6/1.7/1.8 but can be 'fixed' there by adding a 'timeout mail 200ms'.. (except on 1.6 which doesn't have that setting.) I don't think that should be needed though if everything was working properly? If the test could be committed, and related issues exposed fixed that would be neat ;) Thanks in advance, PiBa-NL (Pieter) From 49a605bfadaafe25de0f084c7d1d449eef9c23aa Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Sun, 23 Dec 2018 21:06:31 +0100 Subject: [PATCH] REG-TEST: mailers: add new test for 'mailers' section This test verifies the mailers section works properly by checking that it sends the proper amount of mails when health-checks are changing and or marking a server up/down The test currently fails on all versions of haproxy i tried with varying results. 1.9.0 produces thousands of mails.. 1.8.14 only sends 1 mail, needs a 200ms 'timeout mail' to succeed 1.7.11 only sends 1 mail, needs a 200ms 'timeout mail' to succeed 1.6 only sends 1 mail, (does not have the 'timeout mail' setting implemented) --- reg-tests/mailers/shealthcheckmail.lua | 105 + reg-tests/mailers/shealthcheckmail.vtc | 75 ++ 2 files changed, 180 insertions(+) create mode 100644 reg-tests/mailers/shealthcheckmail.lua create mode 100644 reg-tests/mailers/shealthcheckmail.vtc diff --git a/reg-tests/mailers/shealthcheckmail.lua b/reg-tests/mailers/shealthcheckmail.lua new file mode 100644 index ..9c75877b --- /dev/null +++ b/reg-tests/mailers/shealthcheckmail.lua @@ -0,0 +1,105 @@ + +local vtc_port1 = 0 +local mailsreceived = 0 +local mailconnectionsmade = 0 +local healthcheckcounter = 0 + +core.register_action("bug", { "http-res" }, function(txn) + data = txn:get_priv() + if not data then + data = 0 + end + data = data + 1 + print(string.format("set to %d", data)) + txn.http:res_set_status(200 + data) + txn:set_priv(data) +end) + +core.register_service("luahttpservice", "http", function(applet) + local response = "?" + local responsestatus = 200 + if applet.path == "/setport" then + vtc_port1 = applet.headers["vtcport1"][0] + response = "OK" + end + if applet.path == "/svr_healthcheck" then + healthcheckcounter = healthcheckcounter + 1 + if healthcheckcounter < 2 or healthcheckcounter > 6 then + responsestatus = 403 + end + end + + applet:set_status(responsestatus) + if applet.path == "/checkMailCounters" then + response = "MailCounters" + applet:add_header("mailsreceived", mailsreceived) + applet:add_header("mailconnectionsmade", mailconnectionsmade) + end + applet:start_response() + applet:send(response) +end) + +core.register_service("fakeserv", "http", function(applet) + applet:set_status(200) + applet:start_response() +end) + +function RecieveAndCheck(applet, expect) + data = applet:getline() + if data:sub(1,expect:len()) ~= expect then + core.Info("Expected: "..expect.." but got:"..data:sub(1,expect:len())) + applet:send("Expected: "..expect.." but got:"..data.."\r\n") + return false + end + return true +end + +core.register_service("mailservice", "tcp", function(applet) + core.Info("# Mailservice Called #") + mailconnectionsmade = mailconnectionsmade + 1 + applet:send("220 Welcome\r\n") + local data + + if RecieveAndCheck(applet, "EHLO") == false then + return + end + applet:send("250 OK\r\n") + if RecieveAndCheck(applet, "MAIL FROM:") == false then + return + end + applet:send("250 OK\r\n") + if RecieveAndCheck(applet, "RCPT TO:") == false then + return + end + applet:send("250 OK\r\n") + if RecieveAndCheck(applet, "DATA") == false then + return + end + applet:send("354 OK\r\n") + core.Info(" Send your mailbody") + local endofmail = false + local subject = "" + while endofmail ~= true do + data = apple
d94f877 causes timeout in a basic connection test 1.9-dev11_d94f877
Hi List, Christopher, Seems like d94f877 causes timeout in a pretty 'basic' connection test that transfers a little bit of data .? Or at least attached test fails to complete for me.. # top TEST ./PB-TEST/basic_connection.vtc TIMED OUT (kill -9) # top TEST ./PB-TEST/basic_connection.vtc FAILED (120.236) signal=9 Please can you take a look :) Thanks in advance. Regards, PiBa-NL (Pieter) # Checks a simple request varnishtest "Checks a simple request" feature ignore_unknown_macro server s1 { rxreq txresp -bodylen 42202 } -start haproxy h1 -conf { global stats socket /tmp/haproxy.socket level admin defaults mode http log global option httplog timeout connect 3s timeout client 4s timeout server 4s frontend fe1 bind "fd@${fe_1}" default_backend b1 backend b1 http-reuse never server srv1 ${s1_addr}:${s1_port} #pool-max-conn 0 } -start shell { HOST=${h1_fe_1_addr} if [ "${h1_fe_1_addr}" = "::1" ] ; then HOST="\[::1\]" fi curl -v -k "http://$HOST:${h1_fe_1_port}/CurlTest1"; } -run server s1 -wait
Re: corruption of data with compression in 1.9-dev10
Hi Christopher, Fix confirmed. top 2.5 shell_out|File1 all OK top 2.5 shell_out|File2 all OK top 2.5 shell_out|File3 all OK Thank you! Regards, PiBa-NL (Pieter)
Re: regtests - with option http-use-htx
Hi Willy, Op 15-12-2018 om 17:06 schreef Willy Tarreau: Hi Pieter, On Sat, Dec 15, 2018 at 04:52:10PM +0100, PiBa-NL wrote: Hi List, Willy, Trying to run some existing regtests with added option: option http-use-htx Using: HA-Proxy version 1.9-dev10-c11ec4a 2018/12/15 I get the below issues sofar: based on /reg-tests/connection/b0.vtc Takes 8 seconds to pass, in a slightly modified manor 1.1 > 2.0 expectation for syslog. This surely needs a closer look? # top TEST ./htx-test/connection-b0.vtc passed (8.490) It looks exactly like another issue we've found when a content-length is missing but the close is not seen, which is the same in your case with the first proxy returning the 503 error page by default. Christopher told me he understands what's happening in this situation (at least for the one we've met), I'm CCing him in case this report fuels this thoughts. Ok thanks. based on /reg-tests/stick-table/b1.vtc Difference here is the use=1 vs use=0 , maybe that is better, but then the 'old' expectation seems wrong, and the bug is the case without htx? h1 0.0 CLI recv|0x8026612c0: key=127.0.0.1 use=1 exp=0 gpt0=0 gpc0=0 gpc0_rate(1)=0 conn_rate(1)=1 http_req_cnt=1 http_req_rate(1)=1 http_err_cnt=0 http_err_rate(1)=0 h1 0.0 CLI recv| h1 0.0 CLI expect failed ~ "table: http1, type: ip, size:1024, used:(0|1\n0x[0-9a-f]*: key=127\.0\.0\.1 use=0 exp=[0-9]* gpt0=0 gpc0=0 gpc0_rate\(1\)=0 conn_rate\(1\)=1 http_req_cnt=1 http_req_rate\(1\)=1 http_err_cnt=0 http_err_rate\(1\)=0)\n$" Hmmm here I think we're really hitting corner cases depending on whether the tracked counters are released before or after the logs are emitted. In the case of htx, the logs are emitted slightly later than before, which may induce this. Quite honestly I'd be inclined to set use=[01] here in the regex to cover the race condition that exists in both cases, as there isn't any single good value. Christopher, are you also OK with this ? I can do the patch if you're OK. Its not about emitting logs, its querying the stats admin socket, and even with a added 'delay 2' before doing so the results seem to show the same difference with/without htx. I don't think its a matter of 'timing' .? Regards, PiBa-NL (Pieter) ** c1 0.0 === expect resp.status == 503 c1 0.0 EXPECT resp.status (503) == "503" match *** c1 0.0 closing fd 7 ** c1 0.0 Ending ** top 0.0 === delay 2 *** top 0.0 delaying 2 second(s) ** top 2.1 === haproxy h1 -cli { ** h1 2.1 CLI starting ** h1 2.1 CLI waiting *** h1 2.1 CLI connected fd 7 from ::1 26202 to ::1 26153 ** h1 2.1 === send "show table http1" h1 2.1 CLI send|show table http1 ** h1 2.1 === expect ~ "table: http1, type: ip, size:1024, used:(0|1\\n0x[... *** h1 2.1 debug|0001:GLOBAL.accept(0005)=000b from [::1:26202] ALPN= h1 2.1 CLI connection normally closed *** h1 2.1 CLI closing fd 7 *** h1 2.1 debug|0001:GLOBAL.srvcls[adfd:] *** h1 2.1 debug|0001:GLOBAL.clicls[adfd:] *** h1 2.1 debug|0001:GLOBAL.closed[adfd:] h1 2.1 CLI recv|# table: http1, type: ip, size:1024, used:1 h1 2.1 CLI recv|0x8026612c0: key=127.0.0.1 use=1 exp=0 gpt0=0 gpc0=0 gpc0_rate(1)=0 conn_rate(1)=1 http_req_cnt=1 http_req_rate(1)=1 http_err_cnt=0 http_err_rate(1)=0 h1 2.1 CLI recv| h1 2.1 CLI expect failed ~ "table: http1, type: ip, size:1024, used:(0|1\n0x[0-9a-f]*: key=127\.0\.0\.1 use=0 exp=[0-9]* gpt0=0 gpc0=0 gpc0_rate\(1\)=0 conn_rate\(1\)=1 http_req_cnt=1 http_req_rate\(1\)=1 http_err_cnt=0 http_err_rate\(1\)=0)\n$"
regtests - with option http-use-htx
Hi List, Willy, Trying to run some existing regtests with added option: option http-use-htx Using: HA-Proxy version 1.9-dev10-c11ec4a 2018/12/15 I get the below issues sofar: based on /reg-tests/connection/b0.vtc Takes 8 seconds to pass, in a slightly modified manor 1.1 > 2.0 expectation for syslog. This surely needs a closer look? # top TEST ./htx-test/connection-b0.vtc passed (8.490) based on /reg-tests/stick-table/b1.vtc Difference here is the use=1 vs use=0 , maybe that is better, but then the 'old' expectation seems wrong, and the bug is the case without htx? h1 0.0 CLI recv|0x8026612c0: key=127.0.0.1 use=1 exp=0 gpt0=0 gpc0=0 gpc0_rate(1)=0 conn_rate(1)=1 http_req_cnt=1 http_req_rate(1)=1 http_err_cnt=0 http_err_rate(1)=0 h1 0.0 CLI recv| h1 0.0 CLI expect failed ~ "table: http1, type: ip, size:1024, used:(0|1\n0x[0-9a-f]*: key=127\.0\.0\.1 use=0 exp=[0-9]* gpt0=0 gpc0=0 gpc0_rate\(1\)=0 conn_rate\(1\)=1 http_req_cnt=1 http_req_rate\(1\)=1 http_err_cnt=0 http_err_rate\(1\)=0)\n$" Regards, PiBa-NL (Pieter) #commit b406b87 # BUG/MEDIUM: connection: don't store recv() result into trash.data # # Cyril Bonté discovered that the proxy protocol randomly fails since # commit 843b7cb ("MEDIUM: chunks: make the chunk struct's fields match # the buffer struct"). This is because we used to store recv()'s return # code into trash.data which is now unsigned, so it never compares as # negative against 0. Let's clean this up and test the result itself # without storing it first. varnishtest "PROXY protocol random failures" feature ignore_unknown_macro syslog Slog_1 -repeat 8 -level info { recv expect ~ "Connect from .* to ${h1_ssl_addr}:${h1_ssl_port}" recv expect ~ "ssl-offload-http/http .* \"POST /[1-8] HTTP/2\\.0\"" } -start haproxy h1 -conf { global nbproc 4 nbthread 4 tune.ssl.default-dh-param 2048 stats bind-process 1 log ${Slog_1_addr}:${Slog_1_port} len 2048 local0 debug err defaults mode http timeout client 1s timeout server 1s timeout connect 1s log global option http-use-htx listen http bind-process 1 bind unix@${tmpdir}/http.socket accept-proxy name ssl-offload-http option forwardfor listen ssl-offload-http option httplog bind-process 2-4 bind "fd@${ssl}" ssl crt ${testdir}/common.pem ssl no-sslv3 alpn h2,http/1.1 server http unix@${tmpdir}/http.socket send-proxy } -start shell { HOST=${h1_ssl_addr} if [ "$HOST" = "::1" ] ; then HOST="\[::1\]" fi for i in 1 2 3 4 5 6 7 8 ; do urls="$urls https://$HOST:${h1_ssl_port}/$i"; done curl -i -k -d 'x=x' $urls & wait $! } syslog Slog_1 -wait # commit 3e60b11 # BUG/MEDIUM: stick-tables: Decrement ref_cnt in table_* converters # # When using table_* converters ref_cnt was incremented # and never decremented causing entries to not expire. # # The root cause appears to be that stktable_lookup_key() # was called within all sample_conv_table_* functions which was # incrementing ref_cnt and not decrementing after completion. # # Added stktable_release() to the end of each sample_conv_table_* # function and reworked the end logic to ensure that ref_cnt is # always decremented after use. # # This should be backported to 1.8 varnishtest "stick-tables: Test expirations when used with table_*" # As some macros for haproxy are used in this file, this line is mandatory. feature ignore_unknown_macro # Do nothing. server s1 { } -start haproxy h1 -conf { # Configuration file of 'h1' haproxy instance. defaults mode http timeout connect 5s timeout server 30s timeout client 30s option http-use-htx frontend http1 bind "fd@${my_frontend_fd}" stick-table size 1k expire 1ms type ip store conn_rate(10s),http_req_cnt,http_err_cnt,http_req_rate(10s),http_err_rate(10s),gpc0,gpc0_rate(10s),gpt0 http-request track-sc0 req.hdr(X-Forwarded-For) http-request redirect location https://${s1_addr}:${s1_port}/ if { req.hdr(X-Forwarded-For),table_http_req_cnt(http1) -m int lt 0 } http-request redirect location https://${s1_addr}:${s1_port}/ if { req.hdr(X-Forwarded-For),table_trackers(http1) -m int lt 0 } http-request redirect location https://${s1_addr}:${s1_port}/ if { req.hdr(X-Forwarded-For),in_table(http1) -m int lt 0 } http-request redirect location https://${s1_addr}:${s1_port}/ if { req.hdr(X-Forwarded-For),table_bytes_in_rate(http1) -m int lt 0 } http-request redirect location https://${s1_addr}:${s1_port}/ if { req.hdr(X-Forwa
Re: Quick update on 1.9
Hi Willy, Op 15-12-2018 om 6:15 schreef Willy Tarreau: - Compression corrupts data(Christopher is investigating): https://www.mail-archive.com/haproxy@formilux.org/msg32059.html This one was fixed, he had to leave quickly last evening so he couldn't respond, but it was due to some of my changes to avoid copies, I failed to grasp some corner cases of htx. Could it be it is not fixed/committed in the git repository? (b.t.w. i don't use htx in the vtc testfile ..). As "6e0d8ae BUG/MINOR: mworker: don't use unitialized mworker_proc struct master" seems to be the latest commit and the .vtc file still produces files with different hashes for the 3 curl commands for me. Besides that and as usual thanks for your elaborate response on all the other subjects :). Regards, PiBa-NL (Pieter)
Re: Quick update on 1.9
Hi Willy, Op 14-12-2018 om 22:32 schreef Willy Tarreau: if we manage to get haproxy.org to work reasonably stable this week- end, it will be a sign that we can release it. There are still several known issues that should be addressed before 'release' imho. - Compression corrupts data(Christopher is investigating): https://www.mail-archive.com/haproxy@formilux.org/msg32059.html - Dispatch server crashes haproxy (i found it today): https://www.mail-archive.com/haproxy@formilux.org/msg32078.html - stdout logging makes syslog logging fail (i mentioned it before, but i thought lets 'officially' re-report it now): https://www.mail-archive.com/haproxy@formilux.org/msg32079.html - As you mention haproxy serving the haproxy.org website apparently crashed several times today when you tried a recent build.. I think a week of running without a single crash would be a better indicator than a single week-end that a release could be imminent.?. - Several of the '/checks/' regtests don't work. Might be a problem with varnishtest though, not sure.. But you already discovered that. And thats just the things i am aware off a.t.m.. I'm usually not 'scared' to run a -dev version on my production box for a while and try a few new experimental features that seem useful to me over a weekend, but i do need to have the idea that it will 'work' as good as the version i update from, and to me it just doesn't seem there yet. (i would really like the compression to be functional before i try again..) So with still several known bugs to solve imho its not yet a good time to release it as a 'stable' version already in few days time.?. Or did i misunderstand the 'sign' to release, is it one of several signs that needs to be checked.?. I think a -dev11 or perhaps a -RC if someone likes that term, would probably be more appropriate, before distro's start including the new release expecting stability while actually bringing a seemingly largish potential of breaking some features that used to work. So even current new commits are still introducing new breakage, while shortly before release i would expect mostly little fixes to issues to get committed. That 'new' features arn't 100% stable, that might not be a blocker. But existing features that used to work properly should imho not get released in a broken state.. my 2 cent. Regards, PiBa-NL (Pieter)
stdout logging makes syslog logging fail.. 1.9-dev10-6e0d8ae
Hi List, Willy, stdout logging makes syslog logging fail.. regtest that reproduces the issue attached. Attached test (a modification of /log/b0.vtc) fails, by just adding a stdout logger: *** h1 0.0 debug|[ALERT] 348/000831 (51048) : sendmsg()/writev() failed in logger #2: Socket operation on non-socket (errno=38), which apparently modifies the syslog behavior.? Tested with version 1.9-dev10-6e0d8ae, but i think it never worked since stdout logging was introduced. Regards, PiBa-NL (Pieter) # commit d02286d # BUG/MINOR: log: pin the front connection when front ip/ports are logged # # Mathias Weiersmueller reported an interesting issue with logs which Lukas # diagnosed as dating back from commit 9b061e332 (1.5-dev9). When front # connection information (ip, port) are logged in TCP mode and the log is # emitted at the end of the connection (eg: because %B or any log tag # requiring LW_BYTES is set), the log is emitted after the connection is # closed, so the address and ports cannot be retrieved anymore. # # It could be argued that we'd make a special case of these to immediately # retrieve the source and destination addresses from the connection, but it # seems cleaner to simply pin the front connection, marking it "tracked" by # adding the LW_XPRT flag to mention that we'll need some of these elements # at the last moment. Only LW_FRTIP and LW_CLIP are affected. Note that after # this change, LW_FRTIP could simply be removed as it's not used anywhere. # # Note that the problem doesn't happen when using %[src] or %[dst] since # all sample expressions set LW_XPRT. varnishtest "Wrong ip/port logging" feature ignore_unknown_macro server s1 { rxreq txresp } -start syslog Slg_1 -level notice { recv recv recv info expect ~ \"dip\":\"${h1_fe_1_addr}\",\"dport\":\"${h1_fe_1_port}.*\"ts\":\"[cC]D\",\" } -start haproxy h1 -conf { global log stdout format short daemon log ${Slg_1_addr}:${Slg_1_port} local0 defaults log global timeout connect 3000 timeout client 1 timeout server 1 frontend fe1 bind "fd@${fe_1}" mode tcp log-format {\"dip\":\"%fi\",\"dport\":\"%fp\",\"c_ip\":\"%ci\",\"c_port\":\"%cp\",\"fe_name\":\"%ft\",\"be_name\":\"%b\",\"s_name\":\"%s\",\"ts\":\"%ts\",\"bytes_read\":\"%B\"} default_backendbe_app backend be_app server app1 ${s1_addr}:${s1_port} } -start client c1 -connect ${h1_fe_1_sock} { txreq -url "/" delay 0.02 } -run syslog Slg_1 -wait
crash with regtest: /reg-tests/connection/h00001.vtc after commit f157384
Hi List, Willy, Current 1.9-dev master ( 6e0d8ae ) crashes with regtest: /reg-tests/connection/h1.vtc stack below, it fails after commit f157384. Can someone check? Thanks. Regards, PiBa-NL (Pieter) Program terminated with signal 11, Segmentation fault. #0 0x0057f34f in connect_server (s=0x802616500) at src/backend.c:1384 1384 HA_ATOMIC_ADD(&srv->counters.connect, 1); (gdb) bt full #0 0x0057f34f in connect_server (s=0x802616500) at src/backend.c:1384 cli_conn = (struct connection *) 0x8026888c0 srv_conn = (struct connection *) 0x802688a80 old_conn = (struct connection *) 0x0 srv_cs = (struct conn_stream *) 0x8027b8180 srv = (struct server *) 0x0 reuse = 0 reuse_orphan = 0 err = 0 i = 5 #1 0x004a8acc in sess_update_stream_int (s=0x802616500) at src/stream.c:928 conn_err = 8 srv = (struct server *) 0x0 si = (struct stream_interface *) 0x802616848 req = (struct channel *) 0x802616510 #2 0x004a37c2 in process_stream (t=0x80265c320, context=0x802616500, state=257) at src/stream.c:2302 srv = (struct server *) 0x0 s = (struct stream *) 0x802616500 sess = (struct session *) 0x8027be000 rqf_last = 9469954 rpf_last = 2147483648 rq_prod_last = 7 rq_cons_last = 0 rp_cons_last = 7 rp_prod_last = 0 req_ana_back = 0 req = (struct channel *) 0x802616510 res = (struct channel *) 0x802616570 si_f = (struct stream_interface *) 0x802616808 si_b = (struct stream_interface *) 0x802616848 #3 0x005e9da7 in process_runnable_tasks () at src/task.c:432 t = (struct task *) 0x80265c320 state = 257 ctx = (void *) 0x802616500 process = (struct task *(*)(struct task *, void *, unsigned short)) 0x4a0480 t = (struct task *) 0x80265c320 max_processed = 200 #4 0x00511592 in run_poll_loop () at src/haproxy.c:2620 next = 0 exp = 0 #5 0x0050dc00 in run_thread_poll_loop (data=0x802637080) at src/haproxy.c:2685 ---Type to continue, or q to quit--- start_lock = {lock = 0, info = {owner = 0, waiters = 0, last_location = {function = 0x0, file = 0x0, line = 0}}} ptif = (struct per_thread_init_fct *) 0x92ee30 ptdf = (struct per_thread_deinit_fct *) 0x0 #6 0x0050a2b6 in main (argc=4, argv=0x7fffea48) at src/haproxy.c:3314 tids = (unsigned int *) 0x802637080 threads = (pthread_t *) 0x802637088 i = 1 old_sig = {__bits = 0x7fffe770} blocked_sig = {__bits = 0x7fffe780} err = 0 retry = 200 limit = {rlim_cur = 4042, rlim_max = 4042} errmsg = 0x7fffe950 "" pidfd = -1 Current language: auto; currently minimal haproxy -vv HA-Proxy version 1.9-dev10-6e0d8ae 2018/12/14 Copyright 2000-2018 Willy Tarreau Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -DDEBUG_THREAD -DDEBUG_MEMORY -pipe -g -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-address-of-packed-member -Wno-null-dereference -Wno-unused-label -DFREEBSD_PORTS -DFREEBSD_PORTS OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017 Running on OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 Built with Lua version : Lua 5.3.4 Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY Built with zlib version : 1.2.11 Running on zlib version : 1.2.11 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with PCRE version : 8.40 2017-01-11 Running on PCRE version : 8.40 2017-01-11 PCRE library supports JIT : yes Encrypted password support via crypt(3): yes Built with multi-threading support. Available polling systems : kqueue : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use kqueue. Available multiplexer protocols : (protocols marked as cannot be specified using 'proto' keyword) h2 : mode=HTTP side=FE h2 : mode=HTX side=FE|BE : mode=HTX side=FE|BE : mode=TCP|HTTP side=FE|BE Available filters : [SPOE] spoe [COMP] compression [CACHE] cache [TRACE] trace # commit d02286d # BUG/MINOR: log: pin the front
Re: corruption of data with compression in 1.9-dev10
Hi Christopher, Op 12-12-2018 om 13:49 schreef Christopher Faulet: Le 12/12/2018 à 12:07, Pi Ba a écrit : Found the issue on the 10th (I think commit 56b0348).. so yesterday's commit isn't the (only) problem.. tested with commit 0007d0a the issue also happens. Reverting only below mentioned commit I can't easily do atm. I'll check more closely this evening. Hum, I don't understand, the commit 56b0348 fixes a bug in the H2 multiplexer. you don't use it in your test-case. It should be totally unrelated. It wasn't that specific commit that broke my test, its just that i picked that one because i could... It just that was they day i started to try 1.9-X on my production environment.. Having spend a bit more time checking/compiling various commits i found that this is the commit that 'broke' the testcase that was attached (some of the contents is misplaced/repeated..): http://git.haproxy.org/?p=haproxy.git;a=commit;h=d247be0620c35ea0a43074fd88c6a520629c1823 p.s. I also send you off-list the full output of the test with the added configuration options: filter trace name BEFORE / filter compression / filter trace name AFTER .. (resulting in a 90MB log..) Regards, PiBa-NL (Pieter)
corruption of data with compression in 1.9-dev10
Hi List, Didn't have time yet to bisect when it went wrong. But attached testfile produces the following output after 3 curl requests at different speeds, this seems to trigger a problem as the hash of the downloaded content is nolonger the same as it should be, (in my actual environment its a 2MB javascript file that comes from a iis server behind haproxy.). Took already a few hours more than desired to come up with a seemingly reliable reproduction. 1.9-dev10 is the first one i put on my production environment as i think release is imminent so it 'should' be pretty stable ;), (yes i know..i shouldn't assume..) before it was using 1.8.14.. So was quick to revert to that 1.8 again :). Using these settings: compression algo gzip compression type text/html text/plain application/json application/javascript compression offload When these compression settings are disabled, it completes successfully.. top 2.4 shell_cmd| exit 1 top 2.4 shell_cmd| fi top 2.5 shell_out|File1 all OK top 2.5 shell_out|File2 not OK 7798551c02a37ce89c77fc18fc415e5b top 2.5 shell_out|File3 not OK 3146c4c9fce4da750558bfd9387ffc3b top 2.5 shell_status = 0x0001 top 2.5 shell_exit not as expected: got 0x0001 wanted 0x * top 2.5 RESETTING after ./PB-TEST/ulticompres/b5.vtc ** h1 2.5 Reset and free h1 haproxy 51853 ** h1 2.5 Wait ** h1 2.5 Stop HAproxy pid=51853 h1 2.5 STDOUT poll 0x11 h1 2.5 Kill(2)=0: No error: 0 ** h1 2.6 WAIT4 pid=51853 status=0x0002 (user 0.253496 sys 0.00) * top 2.6 TEST ./PB-TEST/ulticompres/b5.vtc FAILED # top TEST ./PB-TEST/ulticompres/b5.vtc FAILED (2.581) exit=2 haproxy -v HA-Proxy version 1.9-dev10-3815b22 2018/12/11 Copyright 2000-2018 Willy Tarreau Can anyone confirm? Or perhaps even fix ;) Ill try and dig a little more tomorrow evening :). Thanks in advance, PiBa-NL (Pieter) local data = "abcdefghijklmnopqrstuvwxyz" local responseblob = "" math.randomseed(1) for i = 1,1 do responseblob = responseblob .. "\r\n" .. i .. data:sub(1, math.floor(math.random(4,26))) end http01applet = function(applet) local response = responseblob applet:set_status(200) applet:add_header("Content-Type", "application/javascript") applet:add_header("Content-Length", string.len(response)*10) applet:start_response() for i = 1,10 do applet:send(response) end end core.register_service("fileloader-http01", "http", http01applet) # Checks that compression doesnt cause corruption.. varnishtest "Compression validation" feature ignore_unknown_macro haproxy h1 -conf { global # log stdout format short daemon lua-load${testdir}/b5.lua defaults modehttp log global option httplog frontend main-https bind"fd@${fe1}" ssl crt ${testdir}/common.pem compression algo gzip compression type text/html text/plain application/json application/javascript compression offload use_backend TestBack if TRUE backend TestBack server LocalSrv ${h1_fe2_addr}:${h1_fe2_port} listen fileloader mode http bind "fd@${fe2}" http-request use-service lua.fileloader-http01 } -start shell { HOST=${h1_fe1_addr} if [ "${h1_fe1_addr}" = "::1" ] ; then HOST="\[::1\]" fi curl --compressed -k "https://$HOST:${h1_fe1_port}"; -o ${tmpdir}/outputfile1.bin curl --compressed -k "https://$HOST:${h1_fe1_port}"; -o ${tmpdir}/outputfile3.bin --limit-rate 300K curl --compressed -k "https://$HOST:${h1_fe1_port}"; -o ${tmpdir}/outputfile2.bin --limit-rate 500K } -run shell { md5sum=$(md5 -q ${tmpdir}/outputfile1.bin) if [ "$md5sum" = "f0d51d274ebc7696237efec272a38c41" ] then echo "File1 all OK" else echo "File1 not OK $md5sum " testfailed=1 fi md5sum=$(md5 -q ${tmpdir}/outputfile2.bin) if [ "$md5sum" = "f0d51d274ebc7696237efec272a38c41" ] then echo "File2 all OK" else echo "File2 not OK $md5sum " testfailed=1 fi md5sum=$(md5 -q ${tmpdir}/outputfile3.bin) if [ "$md5sum" = "f0d51d274ebc7696237efec272a38c41" ] then echo "File3 all OK" else echo "File3 not OK $md5sum " testfailed=1 fi if [ -n "$testfailed" ]; then exit 1 fi } -run
Re: regtest failure for /log/b00000.vtc, tcp health-check makes and closes a connection to s1 server without valid http-request
Hi Willy, Op 8-12-2018 om 23:49 schreef Willy Tarreau: Hi Pieter, Just let me know which patch you prefer me to apply, I'm fine with your options. The patch i prefer would be 'c', to remove the 'check' from the server line. As it simply removes all possibly check related 'issues'. Willy Regards, PiBa-NL (Pieter)
regtest failure for /log/b00000.vtc, tcp health-check makes and closes a connection to s1 server without valid http-request
Hi List, Willy, The regtest /reg-tests/log/b0.vtc, is failing for me as shown below, and attached: *** s1 0.0 accepted fd 5 127.0.0.1 29538 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) *** c1 0.0 closing fd 8 ** c1 0.0 Ending * top 0.0 RESETTING after ./reg-tests/log/b0.vtc This happens because the health-check makes a tcp connection, then disconnects, but s1 server expects a http-request. So to fix this, i propose to apply 1 out of 3 possible fixes i could imagine, each one does fix the test when executed. a- use s2 server specifically for the tcp health-check b- use a option httpchk, and repeat s1 server twice c- remove the health-check completely. I think option C is probably the cleanest and most fail-safe way. And am 'pretty sure' that the health-check isn't actually needed to reproduce the original issue. Anyhow health-checks could be a source of random test-failures when the system is really slow it might need 2 checks during a test, and normal varnishtest server's only processes 1 connection unless specified differently, or using a 's0 -dispatch'. Or on second (fourth? / last) thought, is there a bug somewhere as the tcp-health-check 'should' abort the connection even before the 3way-tcp-handshake is completed? And as such s1 should not see that first connection?? (Is that also possible/valid for a FreeBSD system? Or would that be a linux trick?) Regards, PiBa-NL (Pieter) top 0.0 extmacro def pwd=/usr/ports/net/haproxy-devel/work/haproxy-eb2bbba top 0.0 extmacro def localhost=127.0.0.1 top 0.0 extmacro def bad_backend=127.0.0.1 14768 top 0.0 extmacro def bad_ip=192.0.2.255 top 0.0 macro def testdir=/usr/ports/net/haproxy-devel/work/haproxy-eb2bbba/./reg-tests/log top 0.0 macro def tmpdir=/tmp/2018-12-08_21-44-28.0VTEYU/vtc.97431.55aa9ba1 *top 0.0 TEST ./reg-tests/log/b0.vtc starting ** top 0.0 === varnishtest "Wrong ip/port logging" *top 0.0 TEST Wrong ip/port logging ** top 0.0 === feature ignore_unknown_macro ** top 0.0 === server s1 { ** s10.0 Starting server s10.0 macro def s1_addr=127.0.0.1 s10.0 macro def s1_port=45615 s10.0 macro def s1_sock=127.0.0.1 45615 *s10.0 Listen on 127.0.0.1 45615 ** top 0.0 === syslog Slg_1 -level notice { ** Slg_1 0.0 Starting syslog server Slg_1 0.0 macro def Slg_1_addr=127.0.0.1 Slg_1 0.0 macro def Slg_1_port=16363 Slg_1 0.0 macro def Slg_1_sock=127.0.0.1 16363 *Slg_1 0.0 Bound on 127.0.0.1 16363 ** s10.0 Started on 127.0.0.1 45615 ** top 0.0 === haproxy h1 -conf { ** Slg_1 0.0 Started on 127.0.0.1 16363 (level: 5) ** Slg_1 0.0 === recv h10.0 macro def h1_cli_sock=::1 58187 h10.0 macro def h1_cli_addr=::1 h10.0 macro def h1_cli_port=58187 h10.0 setenv(cli, 6) h10.0 macro def h1_fe_1_sock=::1 33423 h10.0 macro def h1_fe_1_addr=::1 h10.0 macro def h1_fe_1_port=33423 h10.0 setenv(fe_1, 7) h10.0 conf|global h10.0 conf|\tstats socket /tmp/2018-12-08_21-44-28.0VTEYU/vtc.97431.55aa9ba1/h1/stats.sock level admin mode 600 h10.0 conf|stats socket "fd@${cli}" level admin h10.0 conf| h10.0 conf|global h10.0 conf|log 127.0.0.1:16363 local0 h10.0 conf| h10.0 conf|defaults h10.0 conf|log global h10.0 conf|timeout connect 3000 h10.0 conf|timeout client 1 h10.0 conf|timeout server 1 h10.0 conf| h10.0 conf|frontend fe1 h10.0 conf|bind "fd@${fe_1}" h10.0 conf|mode tcp h10.0 conf|log-format {\"dip\":\"%fi\",\"dport\":\"%fp\",\"c_ip\":\"%ci\",\"c_port\":\"%cp\",\"fe_name\":\"%ft\",\"be_name\":\"%b\",\"s_name\":\"%s\",\"ts\":\"%ts\",\"bytes_read\":\"%B\"} h10.0 conf|default_backendbe_app h10.0 conf| h10.0 conf|backend be_app h10.0 conf|server app1 127.0.0.1:45615 check ** h10.0 haproxy_start h10.0 opt_worker 0 opt_daemon 0 opt_check_mode 0 h10.0 argv|exec /usr/ports/net/haproxy-devel/work/haproxy-eb2bbba/haproxy -d -f /tmp/2018-12-08_21-44-28.0VTEYU/vtc.97431.55aa9ba1/h1/cfg h10.0 XXX 9 @586 *** h10.0 PID: 97435 h10.0 macro def h1_pid=97435 h10.0 macro def h1_name=/tmp/2018-12-08_21-44-28.0VTEYU/vtc.97431.55aa9ba1/h1 ** top 0.0 === client c1 -connect ${h1_fe_1_sock} { ** c10.0 Starting client ** c10.0 Waiting for client *** c10.0 Connect to ::1 33423 *** c10.0 connected f
[PATCH] REGTEST/MINOR: skip seamless-reload test with abns socket on freebsd
Hi List, Willy, Added below to the reg-tests/seamless-reload/b0.vtc, I'm sure there are other targets to be excluded, but I'm not sure which. Or should i have listed 'all' targets except the 5 linux22 - linux2628 specific versions to be excluded? Only thing I know for sure it doesn't work on FreeBSD currently.. I can re-spin the patch with all non-linux targets listed if desired. # expose-fd is available starting at version 1.8 #REQUIRE_VERSION=1.8 # abns@ sockets are not available on freebsd #EXCLUDE_TARGETS=freebsd Regards, PiBa-NL (Pieter) From 3ef9f57b274b350b69d747f8f92fedfb8d283092 Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Sat, 8 Dec 2018 20:51:16 +0100 Subject: [PATCH] REGTEST/MINOR: skip seamless-reload test with abns socket on freebsd abns sockets are not available on freebsd as such mark the test to skip this OS and expose-fd was implemented first in 1.8 so require that --- reg-tests/seamless-reload/b0.vtc | 5 + 1 file changed, 5 insertions(+) diff --git a/reg-tests/seamless-reload/b0.vtc b/reg-tests/seamless-reload/b0.vtc index 498e0c61..8f7acf64 100644 --- a/reg-tests/seamless-reload/b0.vtc +++ b/reg-tests/seamless-reload/b0.vtc @@ -11,6 +11,11 @@ varnishtest "Seamless reload issue with abns sockets" feature ignore_unknown_macro +# expose-fd is available starting at version 1.8 +#REQUIRE_VERSION=1.8 +# abns@ sockets are not available on freebsd +#EXCLUDE_TARGETS=freebsd + haproxy h1 -W -conf { global stats socket ${tmpdir}/h1/stats level admin expose-fd listeners -- 2.18.0.windows.1
[PATCH] REGTEST/MINOR: remove double body specification for server txresp
Hi List, I'm getting a regtest failure for the h1.vtc test, seemingly because varnishtest doesn't like the definition. Error (full log attached): s1 0.0 http[23] |be-hdr-crc: 3634102538 s1 0.0 bodylen = 0 ** s1 0.0 === txresp \ s1 0.0 Assert error in http_tx_parse_args(), vtc_http.c line 870: Condition(body == nullbody) not true. *** h1 1.0 debug|:be.srvcls[000b:adfd] *** h1 1.0 debug|:be.clicls[000b:adfd] This comes from the following definition: txresp \ -status 234 \ -hdr "hdr1: val1" \ -hdr "hdr2: val2a" \ -hdr "hdr2: val2b" \ -hdr "hdr3: val3a, val3b" \ -hdr "hdr4:" \ -bodylen 14 \ -body "This is a body" Where both -bodylen and -body are defined, while it seems to me these 2 settings are 'conflicting', as the '-bodylen' generates a kinda random body content. While '-body' defines the exact string to send as a body.. Seems to me that the bodylen should be removed? Patch that does that attached. Regards, PiBa-NL From e786be564e7dca1e3b347b6cc9e0af05c85e975b Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Sat, 8 Dec 2018 19:48:37 +0100 Subject: [PATCH] REGTEST/MINOR: remove double body specification for server txresp fix http-rules/h0.vtc / http-rules/h0.vtc as both 'bodylen' and 'body' are specified, these settings conflict with each other as they both generate/present the body to send. --- reg-tests/http-rules/h0.vtc | 1 - reg-tests/http-rules/h1.vtc | 1 - 2 files changed, 2 deletions(-) diff --git a/reg-tests/http-rules/h0.vtc b/reg-tests/http-rules/h0.vtc index 25388f8a..aedb41ff 100644 --- a/reg-tests/http-rules/h0.vtc +++ b/reg-tests/http-rules/h0.vtc @@ -25,7 +25,6 @@ server s1 { -hdr "hdr2:val2b" \ -hdr "hdr3: val3a, val3b" \ -hdr "hdr4:" \ - -bodylen 14 \ -body "This is a body" expect req.method == "GET" diff --git a/reg-tests/http-rules/h1.vtc b/reg-tests/http-rules/h1.vtc index 80522a1b..ca86f1b9 100644 --- a/reg-tests/http-rules/h1.vtc +++ b/reg-tests/http-rules/h1.vtc @@ -24,7 +24,6 @@ server s1 { -hdr "hdr2:val2b" \ -hdr "hdr3: val3a, val3b" \ -hdr "hdr4:" \ - -bodylen 14 \ -body "This is a body" expect req.method == "GET" -- 2.18.0.windows.1 top 0.0 extmacro def pwd=/usr/ports/net/haproxy-devel/work/haproxy-eb2bbba top 0.0 extmacro def localhost=127.0.0.1 top 0.0 extmacro def bad_backend=127.0.0.1 28101 top 0.0 extmacro def bad_ip=192.0.2.255 top 0.0 macro def testdir=/usr/ports/net/haproxy-devel/work/haproxy-eb2bbba/./reg-tests/http-rules top 0.0 macro def tmpdir=/tmp/2018-12-08_19-21-06.FYVFcT/vtc.86314.04fe4024 *top 0.0 TEST ./reg-tests/http-rules/h1.vtc starting ** top 0.0 === varnishtest "Composite HTTP manipulation test (H1 and H2 cle... *top 0.0 TEST Composite HTTP manipulation test (H1 and H2 clear to H1 clear) ** top 0.0 === feature ignore_unknown_macro ** top 0.0 === server s1 { ** s10.0 Starting server s10.0 macro def s1_addr=127.0.0.1 s10.0 macro def s1_port=23305 s10.0 macro def s1_sock=127.0.0.1 23305 *s10.0 Listen on 127.0.0.1 23305 ** top 0.0 === haproxy h1 -conf { ** s10.0 Started on 127.0.0.1 23305 *** s10.0 Iteration 0 h10.0 macro def h1_cli_sock=::1 23306 h10.0 macro def h1_cli_addr=::1 h10.0 macro def h1_cli_port=23306 h10.0 setenv(cli, 5) h10.0 macro def h1_feh1_sock=::1 23307 h10.0 macro def h1_feh1_addr=::1 h10.0 macro def h1_feh1_port=23307 h10.0 setenv(feh1, 6) h10.0 macro def h1_feh2_sock=::1 23308 h10.0 macro def h1_feh2_addr=::1 h10.0 macro def h1_feh2_port=23308 h10.0 setenv(feh2, 7) h10.0 conf|global h10.0 conf|\tstats socket /tmp/2018-12-08_19-21-06.FYVFcT/vtc.86314.04fe4024/h1/stats.sock level admin mode 600 h10.0 conf|stats socket "fd@${cli}" level admin h10.0 conf| h10.0 conf|defaults h10.0 conf|\tmode http h10.0 conf|\ttimeout connect 1s h10.0 conf|\ttimeout client 1s h10.0 conf|\ttimeout server 1s h10.0 conf| h10.0 conf|frontend fe h10.0 conf|\tbind "fd@${feh1}" h10.0 conf|\tbind "fd@${feh2}" proto h2 h10.0 conf| h10.0 conf|\t requests h10.0 conf|\thttp-request set-var(req.method) method h10.0 conf|\thttp-request set-var(req.uri)url h10.0 conf|\thtt
Re: behavior change when enabling HTX (in regtest /connection/b00000.vtc)
Hi Willy, Op 3-12-2018 om 4:29 schreef Willy Tarreau: Hi Pieter, On Mon, Dec 03, 2018 at 12:30:37AM +0100, PiBa-NL wrote: Hi List, When running regtest /connection/b0.vtc with the added setting below: defaults option http-use-htx The test no longer completes successfully, is this 'by design' ? It seems its not closing the server connection? Or at least not logging so, which i guess is why the log line isn't emitted at the same 'expected' time? It's not expected that the connection is not closed, After some additional testing I think the connection is and was properly closed. Just the logging was a bit strange. however since the log happens at slightly different stages it could be possible that it's just a matter of event logging. In any case we'll have to have a look. Weird additional issue was that adding a 'option htttplog' changed the number of syslog lines produced. Instead of 2 connect lines, it shows 1 http log line. That is with the 'old' -dev9 version, current master branch seems to have fixed this already. By the way we've been thinking how to easily run the same tests with and without HTX. I wanted to support "ifdef" in the config but it's a bit late now. For the time being, I found that we could do something very ugly like having something like this in the config : ${HTX} http-use-htx and set the "HTX" variable to either "option" or "desc" (latter being to ignore the keyword by making it the proxy's description). With minor changes to the config parser (dropping leading empty words), we could have "$NOHTX" option http-use-htx and set "NOHTX" to "no" to disable it. I'm not sure how feasible this is in the end. I like the ability to also just run a test directly outside of the run-regtests.sh script. And get the same pass/fail result. When using such a $NOHTX 'trick' The test would need to always be run twice with different parameters to perform a 'complete' test of all haproxy features in the current build. Also what would the minimum required haproxy version for such a test become, would it be 1.9 or 1.8 which wouldn't know what 'no option http-use-htx' means.? And even then the specific /connection/b0.vtc test currently succeeds only with htx if also the HTTP/1.1 is changed to HTTP/2.0 in the expected syslog output. Which i'm not sure if it is or isn't a desired effect/change.?. When testing with 1.6 the /connection/b0.vtc needs to be run without threads. And with 1.5 the config would need to change again to not include fd@ sockets, which seems impossible though as varnishtest automatically adds a 'admin socket' into the config utilizing such a fd@ socket. In the end it probably becomes tricky to use 1 'master set' of tests that can be run against any older version, without duplicating tests or including some more advanced ifdef constructions as you already mentioned. Which i think would need to be supported by the 'testing framework', not by haproxy itself. Maybe 'vtest' will eventually have such abilities? Or perhaps the vtc test file would need a little pre-processing, before passing it to varnishtest. Though that doesn't really make things easier/faster either.. Cheers, Willy Regards, PiBa-NL (Pieter)
behavior change when enabling HTX (in regtest /connection/b00000.vtc)
Hi List, When running regtest /connection/b0.vtc with the added setting below: defaults option http-use-htx The test no longer completes successfully, is this 'by design' ? It seems its not closing the server connection? Or at least not logging so, which i guess is why the log line isn't emitted at the same 'expected' time? Regards, PiBa-NL (Pieter)
regtest failure for /cache/h00000.vtc, config parsing fails? after commit 7805e2b
Hi Christopher, List A recent commit http://git.haproxy.org/?p=haproxy.git;a=commit;h=7805e2bc1faf04169866c801087fd794535ecbb2 seems to have broken config parsing for some configurations as seen with reg-test: /cache/h0.vtc Using: haproxy version: 1.9-dev8-7805e2b *** h1 0.0 debug|[ALERT] 334/200950 (30141) : Proxy 'test': unable to find the cache 'my_cache' referenced by http-response cache-store rule. *** h1 0.0 debug|[ALERT] 334/200950 (30141) : Proxy 'test': unable to find the cache 'my_cache' referenced by http-request cache-use rule. *** h1 0.0 debug|[ALERT] 334/200950 (30141) : Proxy 'test': unable to find the cache 'my_cache' referenced by the filter 'cache'. *** h1 0.0 debug|[ALERT] 334/200950 (30141) : Fatal errors found in configuration. Can you take a look? Thanks in advance :). Regards, PiBa-NL (Pieter)
[PATCH] REGTEST: lua: check socket functionality from a lua-task
Hi List, Willy, Frederic, Adis, Attached the same reg-test as send previously, this time as a .patch . Created after the issue was reported here: https://www.mail-archive.com/haproxy@formilux.org/msg31924.html but should be part of the general tests when running regression-checks. It can be back-ported until haproxy 1.6. It does fail on the current 1.9-dev8-51e01b5 version, and back until 1.9-dev5-3e1f68b . Hope its okay like this :). Regards, PiBa-NL (Pieter) From 29b2e82eb8461a2994841ba5460583c8acf5cddc Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Fri, 30 Nov 2018 21:01:01 +0100 Subject: [PATCH] REGTEST: lua: check socket functionality from a lua-task Adding a new test /reg-tests/lua/b4.vtc which checks if the core.tcp() socket basic functions properly when used from a lua-task --- reg-tests/lua/b4.lua | 44 reg-tests/lua/b4.vtc | 34 +++ 2 files changed, 78 insertions(+) create mode 100644 reg-tests/lua/b4.lua create mode 100644 reg-tests/lua/b4.vtc diff --git a/reg-tests/lua/b4.lua b/reg-tests/lua/b4.lua new file mode 100644 index ..3ad14fe5 --- /dev/null +++ b/reg-tests/lua/b4.lua @@ -0,0 +1,44 @@ + +local vtc_port = 0 + +core.register_service("fakeserv", "http", function(applet) + vtc_port = applet.headers["vtcport"][0] + core.Info("APPLET START") + local response = "OK" + applet:add_header("Server", "haproxy/webstats") + applet:add_header("Content-Length", string.len(response)) + applet:add_header("Content-Type", "text/html") + applet:start_response() + applet:send(response) + core.Info("APPLET DONE") +end) + +local function cron() + -- wait for until the correct port is set through the c0 request.. + while vtc_port == 0 do + core.msleep(1) + end + core.Debug('CRON port:' .. vtc_port) + + local socket = core.tcp() + local success = socket:connect("127.0.0.1", vtc_port) + core.Info("SOCKET MADE ".. (success or "??")) + if success ~= 1 then + core.Info("CONNECT SOCKET FAILED?") + return + end + local request = "GET / HTTP/1.1\r\n\r\n" + core.Info("SENDING REQUEST") + socket:send(request) + local result = "" + repeat + core.Info("4") + local d = socket:receive("*a") + if d ~= nil then + result = result .. d + end + until d == nil or d == 0 + core.Info("Received: "..result) +end + +core.register_task(cron) \ No newline at end of file diff --git a/reg-tests/lua/b4.vtc b/reg-tests/lua/b4.vtc new file mode 100644 index ..91b5dde3 --- /dev/null +++ b/reg-tests/lua/b4.vtc @@ -0,0 +1,34 @@ +varnishtest "Lua: check socket functionality from a lua-task" +feature ignore_unknown_macro + +#REQUIRE_OPTIONS=LUA +#REQUIRE_VERSION=1.6 + +server s1 { +rxreq +txresp -bodylen 20 +} -start + +haproxy h1 -conf { +global +lua-load ${testdir}/b4.lua + +frontend fe1 +mode http +bind "fd@${fe1}" +default_backend b1 + +backend b1 +mode http +http-request use-service lua.fakeserv + +} -start + +client c0 -connect ${h1_fe1_sock} { +txreq -url "/" -hdr "vtcport: ${s1_port}" +rxresp +expect resp.status == 200 +} -run + + +server s1 -wait \ No newline at end of file -- 2.18.0.windows.1
Re: BUG: Lua tasks can't use client sockets after bf89ff3d
Hi Frederic, Adis, Op 29-11-2018 om 14:53 schreef Frederic Lecaille: Hi Adis, On 11/29/18 10:03 AM, Adis Nezirovic wrote: On Thu, Nov 29, 2018 at 09:03:34AM +0100, Willy Tarreau wrote: OK thanks, I'll take a look at it once I've flushed my pending stuff on H2+HTX :-( Great, I had my morning coffee and visited my optometrist, so here is a fixed test script (correctly setting Host header). P.S. Lua usually suffers trying to do things in tasks, I don't think this is the first time something gets broken. Can we make reg test with Lua script (maybe strip out LuaSocket requirement)? Yes. There already exist LUA reg tests in reg-tests/lua directory. Fred. Indeed some LUA tests already exists, but it didn't check to use a socket from a task. Attached a new test which does, and does indeed fail on versions since the mentioned commit. Should i make a patch out of it for inclusion in git? Or can you guys do that once the fix is also ready.? i think it was the preferred to get bugfix+regtest 'linked' then ? Regards, PiBa-NL (Pieter) varnishtest "Lua: txn:get_priv() scope" feature ignore_unknown_macro #REQUIRE_OPTIONS=LUA #REQUIRE_VERSION=1.6 server s1 { rxreq txresp -bodylen 20 } -start haproxy h1 -conf { global lua-load ${testdir}/b4.lua frontend fe1 mode http bind "fd@${fe1}" default_backend b1 backend b1 mode http http-request use-service lua.fakeserv } -start client c0 -connect ${h1_fe1_sock} { txreq -url "/" -hdr "vtcport: ${s1_port}" rxresp expect resp.status == 200 } -run server s1 -wait local vtc_port = 0 core.register_service("fakeserv", "http", function(applet) vtc_port = applet.headers["vtcport"][0] core.Info("APPLET START") local response = "OK" applet:add_header("Server", "haproxy/webstats") applet:add_header("Content-Length", string.len(response)) applet:add_header("Content-Type", "text/html") applet:start_response() applet:send(response) core.Info("APPLET DONE") end) local function cron() -- wait for until the correct port is set through the c0 request.. while vtc_port == 0 do core.msleep(1) end core.Debug('CRON port:' .. vtc_port) local socket = core.tcp() local success = socket:connect("127.0.0.1", vtc_port) core.Info("SOCKET MADE ".. (success or "??")) if success ~= 1 then core.Info("CONNECT SOCKET FAILED?") return end local request = "GET / HTTP/1.1\r\n\r\n" core.Info("SENDING REQUEST") socket:send(request) local result = "" repeat core.Info("4") local d = socket:receive("*a") if d ~= nil then result = result .. d end until d == nil or d == 0 core.Info("Received: "..result) end core.register_task(cron)
reg-test failure for /connection/b00000.vtc after commit 3e1f68b
Hi Olivier, List, It seems one of the reg-tests /connection/b0.vtc is failing after this recent commit. http://git.haproxy.org/?p=haproxy.git;a=commit;h=3e1f68bcf9adfcd30e3316b0822c2626cc2a6a84 Using HA-Proxy version 1.9-dev8-3e1f68b 2018/11/29 Some of the output looks like this: *** h1 0.0 debug|Using kqueue() as the polling mechanism. Slog_1 0.0 syslog|<133>Nov 29 22:44:25 haproxy[79765]: Proxy http started. Slog_1 0.0 syslog|<133>Nov 29 22:44:25 haproxy[79765]: Proxy ssl-offload-http started. *** h1 0.0 debug|:ssl-offload-http.accept(0005)=000d from [::1:59078] ALPN=h2 *** h1 0.0 debug|:ssl-offload-http.clireq[000d:]: POST /1 HTTP/1.1 *** h1 0.0 debug|:ssl-offload-http.clihdr[000d:]: user-agent: curl/7.60.0 *** h1 0.0 debug|:ssl-offload-http.clihdr[000d:]: accept: */* *** h1 0.0 debug|:ssl-offload-http.clihdr[000d:]: content-length: 3 *** h1 0.0 debug|:ssl-offload-http.clihdr[000d:]: content-type: application/x-www-form-urlencoded *** h1 0.0 debug|:ssl-offload-http.clihdr[000d:]: host: [::1]:37611 *** h1 0.0 debug|:ssl-offload-http.srvcls[000d:adfd] *** h1 0.0 debug|:ssl-offload-http.clicls[000d:adfd] *** h1 0.0 debug|:ssl-offload-http.closed[000d:adfd] Slog_1 0.0 syslog|<134>Nov 29 22:44:25 haproxy[79765]: ::1:59078 [29/Nov/2018:22:44:25.752] ssl-offload-http~ ssl-offload-http/http 0/0/0/-1/1 400 187 - - CH-- 1/1/0/0/0 0/0 "POST /1 HTTP/1.1" ** Slog_1 0.0 === expect ~ "Connect from .* to ${h1_ssl_addr}:${h1_ssl_port}" Slog_1 0.0 EXPECT FAILED ~ "Connect from .* to ::1:37611" ... top 0.1 shell_out| % Total % Received % Xferd Average Speed Time Time Time Current top 0.1 shell_out| Dload Upload Total Spent Left Speed top 0.1 shell_out|\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 93 0 90 100 3 5625 187 --:--:-- --:--:-- --:--:-- 5812 top 0.1 shell_out|HTTP/2 400 \r top 0.1 shell_out|cache-control: no-cache\r top 0.1 shell_out|content-type: text/html\r top 0.1 shell_out|\r top 0.1 shell_out|400 Bad request top 0.1 shell_out|Your browser sent an invalid request. top 0.1 shell_out| While it should look like this where offloaded traffic is forwarded to a second http frontend: *** h1 0.0 debug|:ssl-offload-http.accept(0005)=000d from [::1:48710] ALPN=h2 *** h1 0.0 debug|:ssl-offload-http.clireq[000d:]: POST /1 HTTP/1.1 *** h1 0.0 debug|:ssl-offload-http.clihdr[000d:]: user-agent: curl/7.60.0 *** h1 0.0 debug|:ssl-offload-http.clihdr[000d:]: accept: */* *** h1 0.0 debug|:ssl-offload-http.clihdr[000d:]: content-length: 3 *** h1 0.0 debug|:ssl-offload-http.clihdr[000d:]: content-type: application/x-www-form-urlencoded *** h1 0.0 debug|:ssl-offload-http.clihdr[000d:]: host: [::1]:48188 *** h1 0.0 debug|0001:http.accept(0008)=0017 from [::1:48710] ALPN= *** h1 0.0 debug|0001:http.clireq[0017:]: POST /1 HTTP/1.1 *** h1 0.0 debug|0001:http.clihdr[0017:]: user-agent: curl/7.60.0 *** h1 0.0 debug|0001:http.clihdr[0017:]: accept: */* *** h1 0.0 debug|0001:http.clihdr[0017:]: content-length: 3 *** h1 0.0 debug|0001:http.clihdr[0017:]: content-type: application/x-www-form-urlencoded *** h1 0.0 debug|0001:http.clihdr[0017:]: host: [::1]:48188 Slog_1 0.0 syslog|<134>Nov 29 22:18:35 haproxy[70605]: Connect from ::1:48710 to ::1:48188 (http/HTTP) ** Slog_1 0.0 === expect ~ "Connect from .* to ${h1_ssl_addr}:${h1_ssl_port}" Do you see the same? Is more info needed? Thanks in advance :) Regards, PiBa-NL (Pieter)
Re: [PATCH] REGTEST/MINOR: script: add run-regtests.sh script
Hi Frederic, Op 29-11-2018 om 19:18 schreef Frederic Lecaille: On 11/29/18 8:47 AM, Willy Tarreau wrote: On Thu, Nov 29, 2018 at 05:36:35AM +0100, Willy Tarreau wrote: However I'm well aware that it's easier to work on improvements once the script is merged, so what I've done now is to merge it and create a temporary "reg-tests2" target in the makefile to use it without losing the existing one. This way everyone can work in parallel, and once the few issues seem reliably addressed, we can definitely replace the make target. Unfortunately ENOCOFFEE struck me this morning and I forgot to commit my local changes so I merged the unmodified version which replaces the "reg-test" target. Thus now we're condemned to quickly fix these small issues :-) Pieter, I am having a look at all these issues. Regards, Fred. If that means i don't have to do anything at this moment, thank you! (i suppose your turnaround time from issue>fix will also be shorter than waiting for my spare evening hour..) Ill start checking some requirements of existing .vtc . And perhaps writing new one's. Regards, PiBa-NL (Pieter)
Re: [PATCH] REGTEST/MINOR: script: add run-regtests.sh script
Hi Frederic, Willy, Op 27-11-2018 om 15:00 schreef Frederic Lecaille: On 11/27/18 10:44 AM, Frederic Lecaille wrote: On 11/27/18 9:52 AM, Willy Tarreau wrote: Hi guys, On Tue, Nov 27, 2018 at 09:45:25AM +0100, Frederic Lecaille wrote: I put the script in the /reg-tests/ folder. Maybe it should have been besides the Makefile in the / root ? Yes I think it should be placed at the same level as the Makefile. Well, we already have a "scripts" directory with the stuff used for release and backport management. I think it perfectly has its place there. /scripts/ sounds good. Note that the reg tests must be run from the Makefile with "reg-tests" target and possibly other arguments/variables. Willy recently added REG_TEST_FILES variable. I've changed the the script to include the LEVEL parameter almost the way the Makefile used it, changed behavior though so without the parameter it it runs all tests. I am sorry Pieter a remaining detail I should have mentioned before: + for i in $(find $TESTDIR/ -type d -name "vtc.*"); + do + echo "## $(cat $i/INFO) ##" + echo "## test results in: $i" + grep -- $i/LOG + + echo "## $(cat $i/INFO) ##" >> $TESTDIR/failedtests.log + echo "## test results in: $i" >> $TESTDIR/failedtests.log + grep -- $i/LOG >> $TESTDIR/failedtests.log + echo >> $TESTDIR/failedtests.log + done may be shortened thanks to tee command like that: cat <<- EOF | tee $TESDIR/failedtests.log . . EOF Removed some spaces for indentation which became part of the output. I have tested you script. For me it is OK. Good job! Thank you a lot Pieter. OK just let me know what to do with this, should I merge it as-is and expect minor updates later, or do you or Pieter want to resend an updated version ? I can adapt, let me know. I have modified Pieter's patch for the modification mentioned above. Seems to work ;) Willy, Here is a better patch which takes into an account the modification above and yours (the script is added in "tests" directory). I think Willy mentioned a 'scripts' directory? Changed patch to include that as well. You can merge it as-is. Regards, Fred New path attached, which includes a LEVEL check. And a modification of the Makefile to call the ./scripts/run-regtests.sh Please can someone check it again before merging.?. Thanks guys :). Regards, PiBa-NL (Pieter) From 989bf7ccbfd849deed450291121cdcc68796ba64 Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Tue, 27 Nov 2018 22:26:38 +0100 Subject: [PATCH] REGTEST/MINOR: script: add run-regtests.sh script Some tests require a minimal haproxy version or compilation options to be able to run successfully. This script allows to add 'requirements' to tests to check so they will automatically be skipped if a requirement is not met. The script supports several parameters to slightly modify its behavior including the directories to search for tests. Also some features are not available for certain OS's these can also be 'excluded', this should allow for the complete set of test cases to be run on any OS against any haproxy release without 'expected failures'. The test .vtc files will need to be modified to include their 'requirements' by listing including text options as shown below: #EXCLUDE_TARGETS=dos,freebsd,windows #REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA #REQUIRE_VERSION=0.0 #REQUIRE_VERSION_BELOW=99.9, When excluding a OS by its TARGET, please do make a comment why the test can not succeed on that TARGET. --- Makefile| 25 +--- scripts/run-regtests.sh | 317 2 files changed, 318 insertions(+), 24 deletions(-) create mode 100644 scripts/run-regtests.sh diff --git a/Makefile b/Makefile index 6d7a0159..aa6d66b9 100644 --- a/Makefile +++ b/Makefile @@ -1094,28 +1094,5 @@ opts: # LEVEL 3 scripts are low interest scripts (prefixed with 'l' letter). # LEVEL 4 scripts are in relation with bugs they help to reproduce (prefixed with 'b' letter). reg-tests: - $(Q)if [ ! -x "$(VARNISHTEST_PROGRAM)" ]; then \ - echo "Please make the VARNISHTEST_PROGRAM variable point to the location of the varnishtest program."; \ - exit 1; \ - fi - $(Q)export LEVEL=$${LEVEL:-1}; \ - if [ $$LEVEL = 1 ] ; then \ - EXPR='h*.vtc'; \ - elif [ $$LEVEL = 2 ] ; then \ - EXPR='s*.vtc'; \ - elif [ $$LEVEL = 3 ] ; then \ - EXPR='l*.vtc'; \ - elif [ $$LEVEL = 4 ] ; then \ - EXPR='b*.vtc'; \ - fi ; \ - if [ -n "$(REG_TEST_FILES)" ] ; then \ - err=0; \ - for n in $(REG_TEST_FILES); do \ - HAPROXY_P
[PATCH] REGTEST/MINOR: script: add run-regtests.sh script
Hi Frederic, Willy, Added the varnishtest script we have been discussing as a .patch this time. I put the script in the /reg-tests/ folder. Maybe it should have been besides the Makefile in the / root ? Also i put a bit of comments into the commit. I hope it is okay like this? If not, feel free to comment on them or change them as required. Once this one is 'accepted' ill create a few patches for the existing .vtc files to include their requirements. (at least the more obvious ones..) Regards, PiBa-NL (Pieter) From 4432c10a0a822619c152aa187f18b2f6478ac565 Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Sun, 25 Nov 2018 16:46:44 +0100 Subject: [PATCH] REGTEST/MINOR: script: add run-regtests.sh script Some tests require a minimal haproxy version or compilation options to be able to run successfully. This script allows to add 'requirements' to tests to check so they will automatically be skipped if a requirement is not met. The script supports several parameters to slightly modify its behavior including the directories to search for tests. Also some features are not available for certain OS's these can also be 'excluded', this should allow for the complete set of test cases to be run on any OS against any haproxy release without 'expected failures'. The test .vtc files will need to be modified to include their 'requirements' by listing including text options as shown below: #EXCLUDE_TARGETS=dos,freebsd,windows #REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA #REQUIRE_VERSION=0.0 #REQUIRE_VERSION_BELOW=99.9, When excluding a OS by its TARGET, please do make a comment why the test can not succeed on that TARGET. --- reg-tests/run-regtests.sh | 303 ++ 1 file changed, 303 insertions(+) create mode 100644 reg-tests/run-regtests.sh diff --git a/reg-tests/run-regtests.sh b/reg-tests/run-regtests.sh new file mode 100644 index ..1094117f --- /dev/null +++ b/reg-tests/run-regtests.sh @@ -0,0 +1,303 @@ +#!/usr/bin/env sh + +if [ "$1" = "--help" ]; then + cat << EOF +### run-regtests.sh ### + Running run-regtests.sh --help shows this information about how to use it + + Run without parameters to run all tests in the current folder (including subfolders) +run-regtests.sh + + Provide paths to run tests from (including subfolders): +run-regtests.sh ./tests1 ./tests2 + + Parameters: +--j , To run varnishtest with multiple jobs / threads for a faster overall result + run-regtests.sh ./fasttest --j 16 + +--v, to run verbose + run-regtests.sh --v, disables the default varnishtest 'quiet' parameter + +--varnishtestparams , passes custom ARGS to varnishtest + run-regtests.sh --varnishtestparams "-n 10" + + Including text below into a .vtc file will check for its requirements + related to haproxy's target and compilation options +# Below targets are not capable of completing this test succesfully +#EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd + +#EXCLUDE_TARGETS=dos,freebsd,windows + +# Below option is required to complete this test succesfully +#REQUIRE_OPTION=OPENSSL, this test needs OPENSSL compiled in. + +#REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA + +# To define a range of versions that a test can run with: +#REQUIRE_VERSION=0.0 +#REQUIRE_VERSION_BELOW=99.9 + + Configure environment variables to set the haproxy and varnishtest binaries to use +setenv HAPROXY_PROGRAM /usr/local/sbin/haproxy +setenv VARNISHTEST_PROGRAM /usr/local/bin/varnishtest +EOF + return +fi + +_startswith() { + _str="$1" + _sub="$2" + echo "$_str" | grep "^$_sub" >/dev/null 2>&1 +} + +_findtests() { + set -f + for i in $( find "$1" -name "*.vtc" ); do +skiptest= +require_version="$(grep "#REQUIRE_VERSION=" "$i" | sed -e 's/.*=//')" +require_version_below="$(grep "#REQUIRE_VERSION_BELOW=" "$i" | sed -e 's/.*=//')" +require_options="$(grep "#REQUIRE_OPTIONS=" "$i" | sed -e 's/.*=//')" +exclude_targets=",$(grep "#EXCLUDE_TARGETS=" "$i" | sed -e 's/.*=//')," + +if [ -n "$require_version" ]; then + if [ $(_version "$HAPROXY_VERSION") -lt $(_version "$require_version") ]; then +echo " Skip $i because option haproxy is version: $HAPROXY_VERSION" +echo "REASON: this test requires at least version: $require_version" +skiptest=1 + fi +fi +if [ -n "$require_version_below" ]; then + if [ $(_version "$HAPROXY_VERSION") -ge $(_version "$require_version_below") ]; then +echo " Skip $i because option
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, I still have a ' ' newline, with the IFS= but the \n and \012 didnt seem to work there.. Strangely on my PC with both bash and dash I do not have to change IFS value to parse HAPROXY_VERSION, TARGET and OPTIONS with "read" internal command. Reading version,target and options works fine indeed without, however the loop over test files fails if any .vtc file has a space character in its filename. Or should we 'forbid' that with documentation.? Or is there another better workaround for that? I do not think it is a good idea to build TESTDIR like that: TESTRUNDATETIME="$(date '+%Y-%m-%d_%H-%M-%S')" TESTDIR=${TMPDIR:-/tmp}/varnishtest_haproxy/$TESTRUNDATETIME mkdir -p $TESTDIR What if we run the tests several times at the same time? Well they would have to run at the same second. Not sure if that would be wise to do.. But mktemp now solved that i guess :) at least for the directory name part.. Please have a look to mkstemp utility. Without the 's' right? Done, combined with the rundatetime which i do like, so its 'readable' and unique, best of both ways i think?. Remaining details: cat $i/LOG | grep -- should be replaced by grep -- $i/LOG I guess ill never learn do do this right the first time around ;). Fixed. Note that you script is plenty of '\r' characters with no newline character at the end of file position: Not sure what the 'correct' way would be. I think there is a CR LF everywhere at the moment? And the scripts hashbang tries to point to 'sh', will this be a issue.? (acme.sh does the same, and seems to be run an lots of systems..) And if so what can i best do to avoid issues? Also note that some shells do not like == operator (at line 3): Used a single = now. When I do not set both HAPROY_PROGRAM I get this output with a script with a successful result. Checks added to avoid this issue for both haproxy and varnishtest so they are checked to exist. Next round :). Regards, PiBa-NL (Pieter) #!/usr/bin/env sh if [ "$1" = "--help" ]; then cat << EOF ### run-regtests.sh ### Running run-regtests.sh --help shows this information about how to use it Run without parameters to run all tests in the current folder (including subfolders) run-regtests.sh Provide paths to run tests from (including subfolders): run-regtests.sh ./tests1 ./tests2 Parameters: --j , To run varnishtest with multiple jobs / threads for a faster overall result run-regtests.sh ./fasttest --j 16 --v, to run verbose run-regtests.sh --v, disables the default varnishtest 'quiet' parameter --varnishtestparams , passes custom ARGS to varnishtest run-regtests.sh --varnishtestparams "-n 10" Including text below into a .vtc file will check for its requirements related to haproxy's target and compilation options # Below targets are not capable of completing this test succesfully #EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd #EXCLUDE_TARGETS=dos,freebsd,windows # Below option is required to complete this test succesfully #REQUIRE_OPTION=OPENSSL, this test needs OPENSSL compiled in. #REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA # To define a range of versions that a test can run with: #REQUIRE_VERSION=0.0 #REQUIRE_VERSION_BELOW=99.9 Configure environment variables to set the haproxy and varnishtest binaries to use setenv HAPROXY_PROGRAM /usr/local/sbin/haproxy setenv VARNISHTEST_PROGRAM /usr/local/bin/varnishtest EOF return fi _startswith() { _str="$1" _sub="$2" echo "$_str" | grep "^$_sub" >/dev/null 2>&1 } _findtests() { #find "$1" -name "*.vtc" | while read i; do set -f for i in $( find "$1" -name "*.vtc" ); do #echo "TESTcheck '$i'" skiptest= require_version="$(grep "#REQUIRE_VERSION=" "$i" | sed -e 's/.*=//')" require_version_below="$(grep "#REQUIRE_VERSION_BELOW=" "$i" | sed -e 's/.*=//')" require_options="$(grep "#REQUIRE_OPTIONS=" "$i" | sed -e 's/.*=//')" exclude_targets=",$(grep "#EXCLUDE_TARGETS=" "$i" | sed -e 's/.*=//')," if [ -n "$require_version" ]; then if [ $(_version "$HAPROXY_VERSION") -lt $(_version "$require_version") ]; then echo " Skip $i because option haproxy is version: $HAPROXY_VERSION" echo "REASON: this test requires at least version: $require_version" skiptest=1 fi fi if [ -n "$require_version_below" ]; then if [ $(_version "$HAPROXY_VERSION") -ge $(
Re: varnishtest with H2>HTX>H1(keep-alive)
Hi Christopher, Willy, Op 20-11-2018 om 12:09 schreef Christopher Faulet: Hi, The H2 is not yet compatible with the HTX for now. So you should never use both in same time. However, this configuration error should be detected during the configuration parsing, to avoid runtime errors. Here is a patch to do so. I'll merge it. Thanks -- Christopher Faulet Thanks the 'old' config which tried to combine H2 and HTX is now rejected. (as expected) I guess i misinterpreted the 'need' for HTX for the H2 conversion and features which i thought would include the new keep-alive this version brings, but i guess those are separate things. Keepalive for a H1 backend coming from a H2 frontend works fine without using that option. New testcase attached, one that actually works! , regarding H2 > H1 with keepalive. (without HTX option though..) It shows as 'passed' when run. I did notice there is one line regarding the 'double logging' I have got configured though which I'm not sure is supposed to happen, its seems to be because i'm having both stdout and :514 logging should that not be possible?: *** h1 0.0 debug|[ALERT] 323/233813 (5) : sendmsg()/writev() failed in logger #2: Socket operation on non-socket (errno=38) Partial config: global log stdout format raw daemon log :1514 local0 I'm using "HA-Proxy version 1.9-dev7-7ff4f14 2018/11/20" this time. Or is it (again) something i'm configuring wrongly ? ;) . Regards, PiBa-NL (Pieter) # h2 with h1 backend connection reuse check varnishtest "h2 with h1 backend connection reuse check" feature ignore_unknown_macro #REQUIRE_VERSION=1.9 server s1 { rxreq txresp -gziplen 200 rxreq txresp -gziplen 200 } -start server s2 { stream 0 { rxsettings txsettings -ack } -run stream 1 { rxreq txresp -bodylen 200 } -run stream 3 { rxreq txresp -bodylen 200 } -run } -start server s3 -repeat 2 { rxreq txresp -gziplen 200 } -start server s4 { timeout 3 rxreq txresp -gziplen 200 rxreq txresp -gziplen 200 } -start server s5 { timeout 3 rxreq txresp -gziplen 200 rxreq txresp -gziplen 200 } -start haproxy h1 -conf { global #nbthread 3 log stdout format raw daemon log :1514 local0 stats socket /tmp/haproxy.socket level admin defaults mode http #option dontlog-normal log global option httplog timeout connect 3s timeout client 40s timeout server 40s listen fe1 bind "fd@${fe1}" server srv1 ${s1_addr}:${s1_port} listen fe3 bind "fd@${fe3}" proto h2 server srv3 ${s3_addr}:${s3_port} listen fe4 bind "fd@${fe4}" proto h2 server srv4 ${s4_addr}:${s4_port} listen fe5 bind "fd@${fe5}" ssl crt /usr/ports/net/haproxy-devel/test/common.pem alpn h2 server srv5 ${s5_addr}:${s5_port} } -start client c1 -connect ${h1_fe1_sock} { txreq -url "/1" rxresp expect resp.status == 200 txreq -url "/2" rxresp expect resp.status == 200 } -start client c1 -wait client c2 -connect ${s2_sock} { stream 0 { txsettings -hdrtbl 0 rxsettings } -run stream 1 { txreq -req GET -url /3 rxresp expect resp.status == 200 } -run stream 3 { txreq -req GET -url /4 rxresp expect resp.status == 200 } -run } -start client c2 -wait client c3 -connect ${h1_fe3_sock} { stream 0 { txsettings -hdrtbl 0 rxsettings } -run stream 1 { txreq -req GET -url /3 rxresp expect resp.status == 200 } -run stream 3 { txreq -req GET -url /4 rxresp expect resp.status == 200 } -run } -start client c3 -wait client c4 -connect ${h1_fe4_sock} { stream 0 { txsettings -hdrtbl 0 rxsettings } -run stream 1 { txreq -req GET -url /3 rxresp expect resp.status == 200 } -run stream 3 { txreq -req GET -url /4 rxresp expect resp.status == 200 } -run } -start client c4 -wait shell { HOST=${h1_fe5_addr} if [ "${h1_fe5_addr}" = "::1" ] ; then HOST="\[::1\]" fi curl --http2 -i -k https://$HOST:${h1_fe5_port}/CuRLtesT_1/ https://$HOST:${h1_fe5_port}/CuRLtesT_2/ } server s1 -wait server s2 -wait server s3 -wait server s4 -wait server s5 -wait
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Willy, Hello Pieter, Do you intend to finalize this script? We would like to use it in haproxy sources. Note that varnishtest already uses TMPDIR variable in place of /tmp if it is set in the environment. Thanks again. Fred. Thanks for your advices and comments, to be honest i haven't looked at the script for several days, got distracted by other things ;). So sorry for late reply. Just cleaned it up a bit. I guess its ready for another review. I still have a ' ' newline, with the IFS= but the \n and \012 didnt seem to work there.. I've tried to incorporate all suggestions. Lemme know if/what i missed :) Regards, PiBa-NL (Pieter) #!/usr/bin/env sh if [ "$1" == "--help" ]; then cat << EOF ### run-regtests.sh ### Running run-regtests.sh --help shows this information about how to use it Run without parameters to run all tests in the current folder (including subfolders) run-regtests.sh Provide paths to run tests from (including subfolders): run-regtests.sh ./tests1 ./tests2 Parameters: --j , To run varnishtest with multiple jobs / threads for a faster overall result run-regtests.sh ./fasttest --j 16 --v, to run verbose run-regtests.sh --v, disables the default varnishtest 'quiet' parameter --varnishtestparams , passes custom ARGS to varnishtest run-regtests.sh --varnishtestparams "-n 10" Including text below into a .vtc file will check for its requirements related to haproxy's target and compilation options # Below targets are not capable of completing this test succesfully #EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd #EXCLUDE_TARGETS=dos,freebsd,windows # Below option is required to complete this test succesfully #REQUIRE_OPTION=OPENSSL, this test needs OPENSSL compiled in. #REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA # To define a range of versions that a test can run with: #REQUIRE_VERSION=0.0 #REQUIRE_VERSION_BELOW=99.9 Configure environment variables to set the haproxy and varnishtest binaries to use setenv HAPROXY_PROGRAM /usr/local/sbin/haproxy setenv VARNISHTEST_PROGRAM /usr/local/bin/varnishtest EOF return fi _startswith() { _str="$1" _sub="$2" echo "$_str" | grep "^$_sub" >/dev/null 2>&1 } _findtests() { #find "$1" -name "*.vtc" | while read i; do IFS=' ' set -f for i in $( find "$1" -name "*.vtc" ); do #echo "TESTcheck '$i'" skiptest= require_version="$(grep "#REQUIRE_VERSION=" "$i" | sed -e 's/.*=//')" require_version_below="$(grep "#REQUIRE_VERSION_BELOW=" "$i" | sed -e 's/.*=//')" require_options="$(grep "#REQUIRE_OPTIONS=" "$i" | sed -e 's/.*=//')" exclude_targets=",$(grep "#EXCLUDE_TARGETS=" "$i" | sed -e 's/.*=//')," if [ -n "$require_version" ]; then if [ $(_version "$HAPROXY_VERSION") -lt $(_version "$require_version") ]; then echo " Skip $i because option haproxy is version: $HAPROXY_VERSION" echo "REASON: this test requires at least version: $require_version" skiptest=1 fi fi if [ -n "$require_version_below" ]; then if [ $(_version "$HAPROXY_VERSION") -ge $(_version "$require_version_below") ]; then echo " Skip $i because option haproxy is version: $HAPROXY_VERSION" echo "REASON: this test requires a version below: $require_version_below" skiptest=1 fi fi if [ -n "$( echo "$exclude_targets" | grep ",$TARGET," )" ]; then echo " Skip $i because exclude_targets" echo "REASON: exclude_targets '$exclude_targets' contains '$TARGET'" skiptest=1 fi #echo "REQUIRE_OPTIONS : $require_options" for requiredoption in $(echo $require_options | tr "," "\012" ); do if [ -z "$( echo "$OPTIONS" | grep "USE_$requiredoption=1" )" ] then echo " Skip $i because option $requiredoption not found" echo -n "REASON: " echo -n "$required" | sed -e 's/.*,//' -e 's/^[[:space:]]//' echo skiptest=1 fi done for required in "$(grep "#REQUIRE_OPTION=" "$i")"; do if [ -z "$required" ] then continue fi requiredoption=$(echo "$required" | sed -e 's/.*=//' -e 's/,.*//') if [ -z "$
Re: varnishtest with H2>HTX>H1(keep-alive)
Hi Willy, Op 19-11-2018 om 4:37 schreef Willy Tarreau: Hi Pieter, On Mon, Nov 19, 2018 at 01:07:44AM +0100, PiBa-NL wrote: Hi List, I'm trying (and failing?) to write a H2>HTX>H1(keepalive) test. Using haproxy 1.9-dev6-05b9b64. Test vtc attached, i added the 'option http-use-htx' to the fe4 frontend/backend. Is there anything else that should be changed? For HTX, you need dev7. Ah crap, i 'thought' i took the latest commit from the online branch. And stopped looking properly. I must have been a few minutes to soon or something, after i read the dev7 mail.. (and i should have checked i did actually compile the expected version..) (which i totally didn't do in the excitement about htx and the late time..) Or is my way of making the H2 request incorrect? Though the 3 tests before it 'seem' to work alright. I've never tested varnishtest on h2 yet, I don't know precisely how it's supposed to be configured, nor what minimum version is required for this. From an haproxy perspective, your config looks OK. By the way, in my opinion you should remove "option http-keep-alive" since it's the default That was a remnant of a previous try to get keep-alive with h2 which then wasn't supposed to work with dev5 yet. , and I think the config would be more readable by replacing the "frontend"+"backend" couples with a single "listen" for each. Okay true, listen sections would make the config more readable :) . I'm just used to make a frontend+backend for almost 'everything'.. (Usually i have multiple backends behind one frontend anyhow..) And also i 'think' it shows more clearly in the logging output if it did or didn't get passed from frontend to the backend, but maybe thats just my imagination. Below the output i get, with a unexpected '500' status, and with a IC-- on the logline... It also seems it never contacted the s4 server. Indeed, it faced an internal while trying to make the connection. What I think happened is that H2 decoded the request (and logged it), but the upper layer stream failed to do anything from it since it's configured with HTX and HTX is not supported in your version. Please note that even with dev7 you don't have H2 over HTX yet. You definitely need to update to dev7 to eliminate a number of candidate bugs. Without the htx option it does make 1 request to the s4, and the second expected request tries to make a second connection. (the 'old' way..) Without the latest changes from dev7 it's expecetd since by default, server-side keep-alive is lost when H2 is used on the frontend (the connection used to be tied to the stream, so since you have a distinct stream for each request, you used to lose the connection). In dev7 the server-side idle connection moved to the session which is attached to the front connection so the keep-alive will still work. Okay new try with "1.9-dev7-1611965". Hoping this helps, Willy However that still doesn't work yet (as also already seen by Frederic): ** c4 0.2 === txreq -req GET -url /3 *** c4 0.2 tx: stream: 1, type: HEADERS (1), flags: 0x05, size: 37 ** c4 0.2 === rxresp *** h1 0.2 debug|0007:fe4.accept(000e)=0010 from [::1:13402] ALPN= h1 0.2 STDOUT poll 0x11 *** c4 0.2 HTTP2 rx EOF (fd:6 read: No error: 0) c4 0.2 could not get frame header ** c4 0.2 Ending stream 1 *** c4 0.2 closing fd 6 ** c4 0.2 Ending * top 0.2 RESETTING after ./PB-TEST/h2-keepalive-backend.vtc ** h1 0.2 Reset and free h1 haproxy 31909 ** h1 0.2 Wait ** h1 0.2 Stop HAproxy pid=31909 ** h1 0.2 WAIT4 pid=31909 status=0x008b (user 0.013928 sys 0.00) * h1 0.2 Expected exit: 0x0 signal: 0 core: 0 h1 0.2 Bad exit status: 0x008b exit 0x0 signal 11 core 128 * top 0.2 failure during reset # top TEST ./PB-TEST/h2-keepalive-backend.vtc FAILED (0.169) exit=2 root@freebsd11:/usr/ports/net/haproxy-devel # haproxy -v HA-Proxy version 1.9-dev7-1611965 2018/11/19 Copyright 2000-2018 Willy Tarreau So i guess the question remains, is the test configured wrongly, or is some other improvement still needed? (I guess improvement in this case surely is needed as crashing is never the right way, even if the input might be 'wrong'.) Regards, PiBa-NL (Pieter)
varnishtest with H2>HTX>H1(keep-alive)
Hi List, I'm trying (and failing?) to write a H2>HTX>H1(keepalive) test. Using haproxy 1.9-dev6-05b9b64. Test vtc attached, i added the 'option http-use-htx' to the fe4 frontend/backend. Is there anything else that should be changed? Or is my way of making the H2 request incorrect? Though the 3 tests before it 'seem' to work alright. Below the output i get, with a unexpected '500' status, and with a IC-- on the logline... It also seems it never contacted the s4 server. Without the htx option it does make 1 request to the s4, and the second expected request tries to make a second connection. (the 'old' way..) Thanks in advance. Regards PiBa-NL (Pieter) ** c4 0.2 === txreq -req GET -url /3 *** c4 0.2 tx: stream: 1, type: HEADERS (1), flags: 0x05, size: 37 ** c4 0.2 === rxresp *** h1 0.2 debug|0007:fe4.accept(000e)=0011 from [::1:25432] ALPN= *** h1 0.2 debug|0007:fe4.clireq[0011:]: GET /3 HTTP/1.1 *** h1 0.2 debug|0007:b4.clicls[0011:adfd] *** h1 0.2 debug|0007:b4.closed[0011:adfd] *** h1 0.2 debug|::1:25432 [19/Nov/2018:01:03:11.550] fe4 b4/srv4 0/0/-1/-1/0 500 203 - - IC-- 2/1/0/0/0 0/0 "GET /3 HTTP/1.1" *** c4 0.2 rx: stream: 1, type: HEADERS (1), flags: 0x04, size: 26 *** c4 0.2 flag: END_TYPE_HEADERS c4 0.2 header[ 0]: :status : 500 c4 0.2 header[ 1]: cache-control : no-cache c4 0.2 header[ 2]: content-type : text/html *** c4 0.2 rx: stream: 1, type: DATA (0), flags: 0x00, size: 96 *** c4 0.2 rx: stream: 1, type: DATA (0), flags: 0x01, size: 0 *** c4 0.2 flag: END_STREAM c4 0.2 s1 - no data ** c4 0.2 === expect resp.status == 200 c4 0.2 EXPECT resp.status (500) == "200" failed # h2 with h1 backend connection reuse check # the c3 > h1 > s3 test works (wrongly?) because haproxy breaks connection to the server, and creates a new one.. # the c4 > h1 > s4 test fails because haproxy breaks connection to the server, while it should keep the connection alive. varnishtest "h2 with h1 backend connection reuse check" feature ignore_unknown_macro server s1 { rxreq txresp -gziplen 200 rxreq txresp -gziplen 200 } -start server s2 { stream 0 { rxsettings txsettings -ack } -run stream 1 { rxreq txresp -bodylen 200 } -run stream 3 { rxreq txresp -bodylen 200 } -run } -start server s3 -repeat 2 { rxreq txresp -gziplen 200 } -start server s4 { timeout 3 rxreq txresp -gziplen 200 rxreq txresp -gziplen 200 } -start haproxy h1 -conf { global #nbthread 3 log stdout format raw daemon #log :1514 local0 stats socket /tmp/haproxy.socket level admin defaults mode http #option dontlog-normal log global option httplog timeout connect 3s timeout client 40s timeout server 40s frontend fe1 bind "fd@${fe1}" default_backend b1 backend b1 option http-keep-alive server srv1 ${s1_addr}:${s1_port} frontend fe3 bind "fd@${fe3}" proto h2 default_backend b3 backend b3 server srv3 ${s3_addr}:${s3_port} frontend fe4 bind "fd@${fe4}" proto h2 default_backend b4 option http-use-htx backend b4 option http-keep-alive server srv4 ${s4_addr}:${s4_port} option http-use-htx } -start client c1 -connect ${h1_fe1_sock} { txreq -url "/1" rxresp expect resp.status == 200 txreq -url "/2" rxresp expect resp.status == 200 } -start client c1 -wait client c2 -connect ${s2_sock} { stream 0 { txsettings -hdrtbl 0 rxsettings } -run stream 1 { txreq -req GET -url /3 rxresp expect resp.status == 200 } -run stream 3 { txreq -req GET -url /4 rxresp expect resp.status == 200 } -run } -start client c2 -wait client c3 -connect ${h1_fe3_sock} { stream 0 { txsettings -hdrtbl 0 rxsettings } -run stream 1 { txreq -req GET -url /3 rxresp expect resp.status == 200 } -run stream 3 { txreq -req GET -url /4 rxresp expect resp.status == 200 } -run } -start client c3 -wait client c4 -connect ${h1_fe4_sock} { stream 0 { txsettings -hdrtbl 0 rxsettings } -run stream 1 { txreq -req GET -url /3 rxresp expect resp.status == 200 } -run stream 3 { txreq -req GET -url /4 rxresp expect resp.status == 200 } -run } -start client c4 -wait server s1 -wait server s2 -wait server s3 -wait server s4 -wait
Re: Some patches about the master worker
Hi William, Something seems to have been broken by below patch series. (when using threads.?.) *** h1 0.0 debug|[ALERT] 309/191142 (6588) : Current worker #1 (6589) exited with code 134 (Abort trap) *** h1 0.0 debug|[ALERT] 309/191142 (6588) : exit-on-failure: killing every workers with SIGTERM Yes i'm using -W in the varnishtest, and yes that adds a 1 second delay or something, but that never prevented this test from succeeding for me before.. Can you take a look? Regards, PiBa-NL (Pieter) Op 6-11-2018 om 18:33 schreef Willy Tarreau: On Tue, Nov 06, 2018 at 05:37:09PM +0100, William Lallemand wrote: Some improvements for the master-worker. Thanks, whole series merged. I've replaced the warning with a qfprintf() as we discussed so that it's less scary at boot :-) I think we'd benefit from having ha_notice(), ha_info() and ha_debug() in complement to the existing ha_alert() and ha_warning(). This would greatly help display runtime info. And then you could replace your reload warning with a more suitable notice. Willy # Checks that request and connection counters are properly kept varnishtest "Connection counters check" feature ignore_unknown_macro server s1 { rxreq expect req.http.TESTsize == 10 txresp } -repeat 4 -start haproxy h1 -W -conf { global nbthread 3 stats socket /tmp/haproxy.socket level admin defaults mode http log global option httplog timeout connect 3s timeout client 40s timeout server 40s frontend fe1 maxconn 200 bind "fd@${fe_1}" acl donelooping hdr(TEST) -m len 10 http-request set-header TEST "%[hdr(TEST)]x" use_backend b2 if donelooping default_backend b1 backend b1 server srv1 ${h1_fe_1_addr}:${h1_fe_1_port} backend b2 fullconn 200 # haproxy 1.8 does not have the ,length converter. #acl OK hdr(TEST) -m len 500 #http-request deny deny_status 200 if OK #http-request deny deny_status 400 # haproxy 1.9 does have a ,length converter. http-request set-header TESTsize "%[hdr(TEST),length]" http-request del-header TEST server srv2 ${s1_addr}:${s1_port} } -start barrier b1 cond 4 client c1 -connect ${h1_fe_1_sock} { timeout 17 barrier b1 sync txreq -url "/1" rxresp expect resp.status == 200 } -start client c2 -connect ${h1_fe_1_sock} { timeout 17 barrier b1 sync txreq -url "/2" rxresp expect resp.status == 200 } -start client c3 -connect ${h1_fe_1_sock} { timeout 17 barrier b1 sync txreq -url "/3" rxresp expect resp.status == 200 } -start client c4 -connect ${h1_fe_1_sock} { timeout 17 barrier b1 sync txreq -url "/4" rxresp expect resp.status == 200 } -start client c1 -wait client c2 -wait client c3 -wait client c4 -wait # allow a little time to close connections. delay 1 haproxy h1 -cli { send "show info" expect ~ "CurrConns: 0 *\\nCumConns: 41*\\nCumReq: 81" #send "show activity" #expect ~ "hoi" }
Re: enabling H2 slows down my webapp, how to use keep-alive on backend ssl connection?
Hi Lukas, Op 29-10-2018 om 16:39 schreef Lukas Tribus: Hi, On Sun, 28 Oct 2018 at 23:47, PiBa-NL wrote: Hi List, When i enable H2 'alpn h2,http/1.1' on haproxy bind line with offloading 'mode http'. The overall loading of a web-application i use takes longer than without. (Tried with 1.9-dev5 and previous versions) The webapp loads around 25 objects of css/js/images on a page, and when using H1 it uses 4 keep-alive connections to retrieve all objects. However when enabling H2 on the frontend the connection to the webserver (which itself is also made with SSL encryption) is made for every single requested object i suspect this is the main reason for the slowdown, it now needs to perform the ssl handshake on the backend 25 times. Is this by (current) design? Is it planned/possible this will be changed before 1.9 release? Yes and yes, this is what will be fixed be the native HTTP representation (codenamed HTX), hopefully this is something we will be able to play with in 1.9-dev6. Regards, Lukas I wasn't sure if the H1 keep-alive connection on backend was supposed to work already (coming through a H2 frontend). Ill give it another try with dev6 or above. Thanks for your confirmation that this part is still a work in progress :). Regards, PiBa-NL (Pieter)
enabling H2 slows down my webapp, how to use keep-alive on backend ssl connection?
Hi List, When i enable H2 'alpn h2,http/1.1' on haproxy bind line with offloading 'mode http'. The overall loading of a web-application i use takes longer than without. (Tried with 1.9-dev5 and previous versions) The webapp loads around 25 objects of css/js/images on a page, and when using H1 it uses 4 keep-alive connections to retrieve all objects. However when enabling H2 on the frontend the connection to the webserver (which itself is also made with SSL encryption) is made for every single requested object i suspect this is the main reason for the slowdown, it now needs to perform the ssl handshake on the backend 25 times. Is this by (current) design? Is it planned/possible this will be changed before 1.9 release? Or is it likely my configuration / conclusion is wrong? I've added a little vtc trying to simulate the behavior, it currently fails on " s4 0.2 HTTP rx failed (fd:10 read: Connection reset by peer)" while that is where the s4 server expects a second request over its keep-alive connection. (assuming i wrote the test correctly..) While it 'should' fail on the s3 server. Regards, PiBa-NL (Pieter) # h2 with h1 backend connection reuse check # the c3 > h1 > s3 test works (wrongly?) because haproxy breaks connection to the server, and creates a new one.. # the c4 > h1 > s4 test fails because haproxy breaks connection to the server, while it should keep the connection alive. varnishtest "h2 with h1 backend connection reuse check" feature ignore_unknown_macro server s1 { rxreq txresp -gziplen 200 rxreq txresp -gziplen 200 } -start server s2 { stream 0 { rxsettings txsettings -ack } -run stream 1 { rxreq txresp -bodylen 200 } -run stream 3 { rxreq txresp -bodylen 200 } -run } -start server s3 -repeat 2 { rxreq txresp -gziplen 200 } -start server s4 { rxreq txresp -gziplen 200 rxreq txresp -gziplen 200 } -start haproxy h1 -W -conf { global #nbthread 3 log :1514 local0 stats socket /tmp/haproxy.socket level admin defaults mode http #option dontlog-normal log global option httplog timeout connect 3s timeout client 40s timeout server 40s frontend fe1 bind "fd@${fe1}" default_backend b1 backend b1 option http-keep-alive server srv1 ${s1_addr}:${s1_port} frontend fe3 bind "fd@${fe3}" proto h2 default_backend b3 backend b3 server srv3 ${s3_addr}:${s3_port} frontend fe4 bind "fd@${fe4}" proto h2 default_backend b4 backend b4 option http-keep-alive server srv4 ${s4_addr}:${s4_port} } -start client c1 -connect ${h1_fe1_sock} { txreq -url "/1" rxresp expect resp.status == 200 txreq -url "/2" rxresp expect resp.status == 200 } -start client c1 -wait client c2 -connect ${s2_sock} { stream 0 { txsettings -hdrtbl 0 rxsettings } -run stream 1 { txreq -req GET -url /3 rxresp } -run stream 3 { txreq -req GET -url /4 rxresp } -run } -start client c2 -wait client c3 -connect ${h1_fe3_sock} { stream 0 { txsettings -hdrtbl 0 rxsettings } -run stream 1 { txreq -req GET -url /3 rxresp } -run stream 3 { txreq -req GET -url /4 rxresp } -run } -start client c3 -wait client c4 -connect ${h1_fe4_sock} { stream 0 { txsettings -hdrtbl 0 rxsettings } -run stream 1 { txreq -req GET -url /3 rxresp } -run stream 3 { txreq -req GET -url /4 rxresp } -run } -start client c4 -wait server s1 -wait server s2 -wait server s3 -wait server s4 -wait
Re: 'http-response cache-store icons if { path_beg /icons }' produces crashes and/or random behavior
Hi Willy, Op 28-10-2018 om 20:21 schreef Willy Tarreau: Hi Pieter, On Sun, Oct 28, 2018 at 12:49:44AM +0200, PiBa-NL wrote: Hello Chad, List, Thanks for the nice article https://www.haproxy.com/blog/introduction-to-haproxy-acls/ However one of the examples that shows how to use cache-store seems flawed.. Attached ive made a little varnishtest, that: - fails to run success-full when repeated 100 times with the path_beg acl on 1.8.14 (some requests are send twice to the s1 server, which stops listening after 1..) but its about 6% of runs that fails.. There is a bug in this config, which is reported by an haproxy warning : Agreed the configuration is 'wrong', and haproxy says it will 'never' match, but well results above show it did match in 94% of runs, so it was all of the following: user-issue, documentation-issue and a software-issue. Documentation and software are both fixed now, thanks. And at least the user will get consistent results ;). *** h10.0 debug|[WARNING] 300/201711 (22373) : parsing [/tmp/vtc.22366.5fa5c991/h1/cfg:37] : acl 'WeCanSafelyCacheThatFile' will never match because it only involves keywords that are incompatible with 'backend http-response header rule' *** h10.0 debug|[WARNING] 300/201711 (22373) : parsing [/tmp/vtc.22366.5fa5c991/h1/cfg:38] : acl 'WeCanSafelyCacheThatFile' will never match because it only involves keywords that are incompatible with 'backend http-response header rule' Indeed, the ACL references the "path" sample fetch function in the response, which is not available. - produces core dumps with 1.9-dev4-1ff7633 I've just managed to reproduce it and fix it (thanks for your report and the reproducer). I *suspect* 1.8 and older are not safer, but that the way the buffers work there make it dereference a wrong (but existing) memory area, thus it doesn't crash. Thanks for the quick fix, with latest 1.9 code the test 'fails' consistently for the right reason. Using the var(txn.path) instead it succeeds on both versions. Indeed, it's expected since the path is lost once the request leaves. Thanks! Willy Regards, PiBa-NL (Pieter)
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Op 19-10-2018 om 11:51 schreef Frederic Lecaille: The idea of the script sounds good to me. About the script itself it is a nice work which could be a good start. Thanks. Just a few details below. Note that "cat $file | grep something" may be shortened by "grep something $file". It would also be interesting to avoid creating temporary files as most as possible (at least testflist.lst is not necessary I think). TARGET=$(haproxy -vv | grep "TARGET = " | sed 's/.*= //') OPTIONS=$(haproxy -vv | grep "OPTIONS = " | sed 's/.*= //') may be shortened by these lines: { read TARGET; read OPTIONS; } << EOF $(./haproxy -vv |grep 'TARGET\|OPTIONS' | sed 's/.* = //') EOF Thanks, I've changed this. which is portable, or something similar. sed 's/.*=//'| sed 's/,.*//' may be shortened by sed -e 's/.*=//' -e 's/,.*//' Thanks, I've changed this as well. Also note, there are some cases where options are enabled without appearing in OPTIONS variable value. For instance if you compile haproxy like that: $ make TARGET=linux2628 the support for the thread is enabled without appearing in OPTIONS variable value. I am not sure this is an issue at this time. That could become an issue. but should be easy to solve adding a 'fake' option perhaps to check against.. Or adding a separate check perhaps I'm not sure yet. Perhaps we could could only one line for the required options and excluded targets like that: #EXCLUDED_TARGETS=freebsd,dos,windows ;) #REQUIRED_OPTIONS=OPENSSL,LUA,ZLIB Added this option, but then i like the option of excluding single targets and having some comment behind it explaining the reason.. But i guess if one would want to know some comments above the setting could also 'explain' why that target is currently not 'valid' to run the test on. Should i remove the settings for the 'single' option/target.? New 'version' of the script attached. It now supports a set of parameters to modify its behavior a little. And also checking for a 'version requirement'. So a H2 test doesn't have to fail on 1.7. Should i 'ask' to delete old test result.? Or would there be a other better way to keep previous results separated from the current run? If you could give it another look i would appreciate that. Are there any things that need to be added/changed about it before considering to add it to haproxy sources branch? Regards, PiBa-NL (Pieter) #!/usr/bin/env sh if [ "$1" == "--help" ]; then cat << EOF ### run-regtests.sh ### Running run-regtests.sh --help shows this information about how to use it Run without parameters to run all tests in the current folder (including subfolders) run-regtests.sh Provide paths to run tests from (including subfolders): run-regtests.sh ./tests1 ./tests2 Parameters: --j , To run varnishtest with multiple jobs / threads for a faster overall result run-regtests.sh ./fasttest --j 16 --v, to run verbose run-regtests.sh --v, disables the default varnishtest 'quiet' parameter --varnishtestparams , passes custom ARGS to varnishtest run-regtests.sh --varnishtestparams "-n 10" --f, force deleting old /tmp/*.vtc results without asking Including text below into a .vtc file will check for its requirements related to haproxy's target and compilation options # Below targets are not capable of completing this test succesfully #EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd #EXCLUDE_TARGETS=dos,freebsd,windows # Below option is required to complete this test succesfully #REQUIRE_OPTION=OPENSSL, this test needs OPENSSL compiled in. #REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA # To define a range of versions that a test can run with: #REQUIRE_VERSION=0.0 #REQUIRE_VERSION_BELOW=99.9 EOF return fi _startswith() { _str="$1" _sub="$2" echo "$_str" | grep "^$_sub" >/dev/null 2>&1 } _findtests() { #find "$1" -name "*.vtc" | while read i; do IFS=' ' set -f for i in $( find "$1" -name "*.vtc" ); do #echo "TESTcheck '$i'" skiptest= require_version="$(grep "#REQUIRE_VERSION=" "$i" | sed -e 's/.*=//')" require_version_below="$(grep "#REQUIRE_VERSION_BELOW=" "$i" | sed -e 's/.*=//')" require_options="$(grep "#REQUIRE_OPTIONS=" "$i" | sed -e 's/.*=//')" exclude_targets=",$(grep "#EXCLUDE_TARGETS=" "$i" | sed -e 's/.*=//')," if [ -n "$r
'http-response cache-store icons if { path_beg /icons }' produces crashes and/or random behavior
Hello Chad, List, Thanks for the nice article https://www.haproxy.com/blog/introduction-to-haproxy-acls/ However one of the examples that shows how to use cache-store seems flawed.. Attached ive made a little varnishtest, that: - fails to run success-full when repeated 100 times with the path_beg acl on 1.8.14 (some requests are send twice to the s1 server, which stops listening after 1..) but its about 6% of runs that fails.. - produces core dumps with 1.9-dev4-1ff7633 Using the var(txn.path) instead it succeeds on both versions. I think its important to 'fix' the article (and perhaps include the cache section declaration also), and perhaps investigate why haproxy does seem to try and process the fetch it warns about on startup that it would/should never match.. Regards, PiBa-NL (Pieter) # Checks that basic cache is working varnishtest "Checks that basic cache is working" feature ignore_unknown_macro server s1 { rxreq txresp -bodylen 50 } -start syslog Slg_1 -level notice { recv expect ~ "Proxy fe1 started" recv expect ~ "Proxy b1 started" recv info expect ~ "fe1 b1/srv1" recv info expect ~ "fe1 b1/" recv info expect ~ "fe1 b1/" recv info expect ~ "fe1 b1/" } -start haproxy h1 -conf { global #nbthread 3 log ${Slg_1_addr}:${Slg_1_port} local0 #log :1514 local0 #nokqueue stats socket /tmp/haproxy.socket level admin #log /tmp/log local0 defaults mode http #option dontlog-normal log global option httplog timeout connect 3s timeout client 4s timeout server 4s cache icons total-max-size 10 max-age 60 frontend fe1 bind "fd@${fe_1}" default_backend b1 backend b1 http-request set-var(txn.MyPath) path #acl WeCanSafelyCacheThatFile var(txn.MyPath) -m beg /icons/ acl WeCanSafelyCacheThatFile path_beg /icons/ http-request cache-use icons if WeCanSafelyCacheThatFile http-response cache-store icons if WeCanSafelyCacheThatFile http-response add-header CacheResponse TRUE if WeCanSafelyCacheThatFile server srv1 ${s1_addr}:${s1_port} } -start client c1 -connect ${h1_fe_1_sock} { timeout 5 txreq -url /icons/ rxresp expect resp.status == 200 txreq -url /icons/ rxresp expect resp.status == 200 } -repeat 2 -run syslog Slg_1 -wait
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Do you have a little time to take a look at (the idea of) the script? Thanks in advance :). Op 3-10-2018 om 21:52 schreef PiBa-NL: Hi Frederic, Made a little script (attached).. What do you think ? Could this work for current 'requirements'? Regards, PiBa-NL (Pieter)
Re: Logging real rather than load balancer IP
Hi Mark, Op 17-10-2018 om 23:36 schreef Mark Holmes: Question: We have some web apps which are behind an haproxy load balancer, with TLS being terminated on the server rather than at the balancer (so using tcp mode). The web server logs are recording the source IP as that of the load balancer as expected. I now have a requirement to pass the ‘real’ IP address through to the web application and also record it in the webserver logs. Currently, with other applications where TLS is terminated at the balancer and we are using http mode to connect to the backend web servers I use X-FORWARDED-FOR to pass through the ‘real’ IP address but obviously that won’t help me when using TCP mode. I read some stuff about using the PROXY protocol, but I’m running IIS 8.5 and as far as I can tell it doesn’t support PROXY. Am I correct? My other option appears to be to switch to transparent proxying. I have verified the kernel I’m using is compiled with TPROXY support as is haproxy itself. Before I go down this road – is transparent proxying the correct/best option here? Thanks in advance for any advice Mark There are 3 options to let a webserver know the client-IP. -forwardfor (only works with 'mode http' and needs webserver to know how to use that header) -proxyprotocol (needs server to support it, and know how to use it.) -TPROXY (needs routing for reply traffic through haproxy) As you can see each has its own dis-advantage's.. And well with the first 2 already ruled out, the 3rd is your only option.. (that i know of anyhow..) Regards, PiBa-NL (Pieter)
Re: [PATCH] REGTEST/MINOR: loadtest: add a test for connection counters
Hi William, Op 12-10-2018 om 10:53 schreef William Lallemand: The attached patch should fix the issue. The patch works for me, thanks. Regards, PiBa-NL (Pieter)
Re: [PATCH] REGTEST/MINOR: loadtest: add a test for connection counters
Hi Willy, William, Op 2-10-2018 om 3:56 schreef Willy Tarreau: it's important to cut this into pieces to figure what it's showing. A little update on this issue split in 2 pieces. -Connection and request counters to low when ran as regtest from varnishtest (bug?) It turns out that starting haproxy from varnishtest, and using -W master-worker mode, actually creates 2 processes that are handling traffic. That explains that a large part of connections isn't seen by the other haproxy instance and stats showing to low amounts of connections. Bisecting it seems to fail on this commit: b3f2be3 , perhaps @William can you take a look at it? Not really sure when this occurs in a 'real' environment, it doesn't seem to happen when manually running haproxy -W, but still its strange that when varnisttest is calling haproxy this occurs. -Request counter to high (possibly a improvement request?) The http_end_txn_clean_session(..) is called which increments the counter on a finished http response, and i was testing with 2 different methods for 1.8 and 1.9 due to missing length converter i used in my 1.9 test, which makes the comparison unfair. Sorry i didn't realize this earlier i thought it did 'more or less' the same, that seems to have been 'less'. Together with that i found the numbers odd/unexpected i assumed a bigger problem that it actually seems to be, perhaps its not even that bad, haproxy is 'preparing' for a second request over the keep-alive connection if i understand correctly. Which eventually doesn't happen, but is counted. Maybe that is a point that can be improved in a future version if time permits.? Or would it even be expected to behave like that? Regards, PiBa-NL (Pieter)
Re: [PATCH] REGTEST/MINOR: loadtest: add a test for connection counters
Hi Willy, Op 2-10-2018 om 3:56 schreef Willy Tarreau: it's important to cut this into pieces to figure what it's showing. Okay, the good thing i suppose we already have a reproduction sort-off.. What would be the best way to try and get to the bottom of this? Add debug output to haproxy code? Get some kind of trace? Anything in particular that would be of interest? Regards, PiBa-NL (Pieter)