Re: Debugging PH state
On 2023-03-13 15:50, Christopher Faulet wrote: Le 3/6/23 à 08:50, Christian Ruppert a écrit : On 2023-03-03 16:37, Christopher Faulet wrote: Le 3/3/23 à 15:40, Christian Ruppert a écrit : So I can reproduce it. I captured the response, extracted the data which includes to entire header + payload. I put that into a file like foo.bin and I start a "nc" that just serves that one response on every request. It works fine, no 500, no PH. Even with the same request so I really don't get it. I really wonder if that's a HAProxy bug. Hi Christian, It is indeed possible. Could you share your configuration ? Is there any special in the response ? Hi Christopher, I can share the config off-list. I don't see anything special, neither in the request nor the response. Do you know where the 500/PH is being set in the source by chance? Perhaps I can add some additional debugging there. If it's a invalid response, I'd guess that Apache would already block that from the actual backend (Apache is in front of it). The Response has a small header, small body. XML. Too bad I cannot get that 500 on every request/response :( Hi Christian, Sorry for the delay, just back from vacation. So sure, you can send me your configuration off-list. If you have a simple reproducer, share it too. It will be easier to understand what happens. In the code, you can find rewrite errors by grepping on "failed_rewrites". It a frontend/backend/listener/server counter. A rewrite error may happen every time haproxy tries to alter the request or the response. Hi Christopher, no problem! I've been busy with other stuff as well. The customer decided to not dig into it any further, since it happens sometimes only and only in some specific cases. I can still try to get a copy of the relevant parts of the config anyway, perhaps also including one of those responses. But as I said, I simulated it via "nc" and replied with the same response that failed but anything was fine. So I don't have anything to trigger that one for sure :/ -- Regards, Christian Ruppert
Re: Debugging PH state
Le 3/6/23 à 08:50, Christian Ruppert a écrit : On 2023-03-03 16:37, Christopher Faulet wrote: Le 3/3/23 à 15:40, Christian Ruppert a écrit : So I can reproduce it. I captured the response, extracted the data which includes to entire header + payload. I put that into a file like foo.bin and I start a "nc" that just serves that one response on every request. It works fine, no 500, no PH. Even with the same request so I really don't get it. I really wonder if that's a HAProxy bug. Hi Christian, It is indeed possible. Could you share your configuration ? Is there any special in the response ? Hi Christopher, I can share the config off-list. I don't see anything special, neither in the request nor the response. Do you know where the 500/PH is being set in the source by chance? Perhaps I can add some additional debugging there. If it's a invalid response, I'd guess that Apache would already block that from the actual backend (Apache is in front of it). The Response has a small header, small body. XML. Too bad I cannot get that 500 on every request/response :( Hi Christian, Sorry for the delay, just back from vacation. So sure, you can send me your configuration off-list. If you have a simple reproducer, share it too. It will be easier to understand what happens. In the code, you can find rewrite errors by grepping on "failed_rewrites". It a frontend/backend/listener/server counter. A rewrite error may happen every time haproxy tries to alter the request or the response. -- Christopher Faulet
Re: Debugging PH state
On 2023-03-03 16:37, Christopher Faulet wrote: Le 3/3/23 à 15:40, Christian Ruppert a écrit : So I can reproduce it. I captured the response, extracted the data which includes to entire header + payload. I put that into a file like foo.bin and I start a "nc" that just serves that one response on every request. It works fine, no 500, no PH. Even with the same request so I really don't get it. I really wonder if that's a HAProxy bug. Hi Christian, It is indeed possible. Could you share your configuration ? Is there any special in the response ? Hi Christopher, I can share the config off-list. I don't see anything special, neither in the request nor the response. Do you know where the 500/PH is being set in the source by chance? Perhaps I can add some additional debugging there. If it's a invalid response, I'd guess that Apache would already block that from the actual backend (Apache is in front of it). The Response has a small header, small body. XML. Too bad I cannot get that 500 on every request/response :( -- Regards, Christian Ruppert
Re: Debugging PH state
Le 3/3/23 à 15:40, Christian Ruppert a écrit : So I can reproduce it. I captured the response, extracted the data which includes to entire header + payload. I put that into a file like foo.bin and I start a "nc" that just serves that one response on every request. It works fine, no 500, no PH. Even with the same request so I really don't get it. I really wonder if that's a HAProxy bug. Hi Christian, It is indeed possible. Could you share your configuration ? Is there any special in the response ? -- Christopher Faulet
Re: Debugging PH state
So I can reproduce it. I captured the response, extracted the data which includes to entire header + payload. I put that into a file like foo.bin and I start a "nc" that just serves that one response on every request. It works fine, no 500, no PH. Even with the same request so I really don't get it. I really wonder if that's a HAProxy bug. -- Regards, Christian Ruppert
Debugging PH state
Hi list, we have a case where a response results into a 500 (PH): Mar 1 14:44:05.000 n095211 haproxy[2427573]: 1.2.3.4:22917 [01/Mar/2023:14:44:05.108] genfrontend_somefoo_stage~ genbackend_49140-somefoo_stage/localhost 1/0/0/-1/533 500 20378 - - PH-- 24/1/0/0/0 0/0 {stage.somedom|someUA|https://stage.somedom/customizing/index.xhtml?cid=4||https://stage.somedom||somcapture||id-ecPublicKey} "POST /customizing/index.xhtml?cid=4 HTTP/1.1" On the backend: 192.168.95.211 - - [01/Mar/2023:14:44:05 +0100] "POST /customizing/index.xhtml?cid=4 HTTP/1.1" 200 27643 "https://stage.somedom/customizing/index.xhtml?cid=4; "someUA" So PH is stated as following: PH The proxy blocked the server's response, because it was invalid, incomplete, dangerous (cache control), or matched a security filter. In any case, an HTTP 502 error is sent to the client. One possible cause for this error is an invalid syntax in an HTTP header name containing unauthorized characters. It is also possible but quite rare, that the proxy blocked a chunked-encoding request from the client due to an invalid syntax, before the server responded. In this case, an HTTP 400 error is sent to the client and reported in the logs. Finally, it may be due to an HTTP header rewrite failure on the response. In this case, an HTTP 500 error is sent (see "tune.maxrewrite" and "http-response strict-mode" for more inforomation). In this case, the only relevant part seems to be: Finally, it may be due to an HTTP header rewrite failure on the response. In this case, an HTTP 500 error is sent (see "tune.maxrewrite" and "http-response strict-mode" for more inforomation). tune.maxrewrite, hm, ok, it's set to 4096 in this case already. Also the response header is really small and even below 1000 bytes so that shouldn't be the problem, shouldn't it? The response body is also pretty small compared to others but also the body shouldn't matter in this case all all, shouldn't it? It's gzipped, coming from an Apache, compressed it's between ~17-30kb, uncompressed it's around 70-80kb. Disabling strict mode is no option because we don't want to ignore actual errors when processing / rewriting stuff internally in HAProxy. I captured some of those responses but I don't see anything wrong or weird at all to be honest. Another problem is, we cannot reproduce it specifically. It only occurs sometimes, in some cases. Any ideas? HAProxy version 2.7.2-7e295dd 2023/01/20 - https://haproxy.org/ Status: stable branch - will stop receiving fixes around Q1 2024. Known bugs: http://www.haproxy.org/bugs/bugs-2.7.2.html Running on: Linux 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 Build options : TARGET = linux-glibc CPU = generic CC = cc CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment -DMAX_SESS_STKCTR=12 OPTIONS = USE_PCRE= USE_PCRE_JIT= USE_PCRE2=1 USE_PCRE2_JIT= USE_LIBCRYPT=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB= USE_SLZ=1 USE_NS= USE_SYSTEMD=1 USE_PROMEX=1 DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC +RT +SHM_OPEN +SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=4). Built with OpenSSL version : OpenSSL 1.1.1n 15 Mar 2022 Running on OpenSSL version : OpenSSL 1.1.1n 15 Mar 2022 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 Built with Lua version : Lua 5.3.3 Built with the Prometheus exporter as a service Support for malloc_trim() is enabled. Built with libslz for stateless compression. Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with PCRE2 version : 10.36 2020-12-04 PCRE2 library supports JIT : no (USE_PCRE2_JIT not set) Encrypted password support via crypt(3): yes Built with gcc compiler version 10.2.1 20210110 Available