I have a strange problem with OCSP updating. On one server everything
works. That server is in my basement, running Ubuntu 22.04.
Another system, Ubuntu 20.04 in AWS using exactly the same certificates
and exactly the same crt-list file is failing to do an OCSP update.
Jul 8 21:15:38 - haproxy[4075] -:- [08/Jul/2023:21:15:38.446]
<OCSP-UPDATE> /etc/ssl/certs/local/elyograg_org.wildcards.combined.pem 2
"HTTP error" 1 0
Here's the very weird part. It seems that haproxy is sending the OCSP
request to localhost, not the http://r3.o.lencr.org URL that it SHOULD
be sending it to. Right before the above log entry is this one:
Jul 8 21:15:38 - haproxy[4075] 127.0.0.1:57696
[08/Jul/2023:21:15:38.447] web80 web80/<NOSRV> 0/-1/-1/-1/0 302 230 - -
LR-- 1/1/0/0/0 0/0 "GET
/MFMwUTBPME0wSzAJBgUrDgMCGgUABBRI2smg%2ByvTLU%2Fw3mjS9We3NfmzxAQUFC6zF7dYVsuuUAlA5h%2BvnYsUwsYCEgOq9K0xVAXkgj8X4cNGeMutQw%3D%3D
HTTP/1.1"
Both systems are using a local bind9 caching resolver, which does work.
If I manually make the OCSP request with curl on the system, I get a
response.
Both servers are running haproxy 2.8.1 compiled from source with quictls
branch openssl-3.1.0+quic+locks. It was compiled locally on each
server. I don't know if it makes any difference ... the one that works
was compiled with make -j12 (24 CPU cores) and the one that doesn't was
compiled with make -j3 (2 CPU cores).
Below is haproxy -vv from both servers, first the one that works:
HAProxy version 2.8.1 2023/07/03 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2
2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.1.html
Running on: Linux 6.1.0-1015-oem #15-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun
16 09:51:49 UTC 2023 x86_64
Build options :
TARGET = linux-glibc
CPU = native
CC = cc
CFLAGS = -O2 -march=native -g -Wall -Wextra -Wundef
-Wdeclaration-after-statement -Wfatal-errors -Wtype-limits
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond
-Wnull-dereference -fwrapv -Wno-address-of-packed-member
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
-Wno-missing-field-initializers -Wno-cast-function-type
-Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_QUIC=1
USE_PCRE2_JIT=1
DEBUG =
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY
+CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE
-LIBATOMIC +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH
-MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL
-OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL
-PROCCTL -PROMEX -PTHREAD_EMULATION +QUIC +RT +SHM_OPEN -SLZ +SSL
-STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY
-WURFL +ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256,
default=48).
Built with OpenSSL version : OpenSSL 3.1.0+quic 14 Mar 2023
Running on OpenSSL version : OpenSSL 3.1.0+quic 14 Mar 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.39 2021-10-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.3.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : none
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
HAProxy version 2.8.1 2023/07/03 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2
2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.1.html
Running on: Linux 5.15.0-1039-aws #44~20.04.1-Ubuntu SMP Thu Jun 22
12:21:12 UTC 2023 x86_64
Build options :
TARGET = linux-glibc
CPU = native
CC = cc
CFLAGS = -O2 -march=native -g -Wall -Wextra -Wundef
-Wdeclaration-after-statement -Wfatal-errors -Wtype-limits
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond
-Wnull-dereference -fwrapv -Wno-address-of-packed-member
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
-Wno-missing-field-initializers -Wno-cast-function-type
-Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_QUIC=1
USE_PCRE2_JIT=1
DEBUG =
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY
+CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE
-LIBATOMIC +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH
-MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL
-OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL
-PROCCTL -PROMEX -PTHREAD_EMULATION +QUIC +RT +SHM_OPEN -SLZ +SSL
-STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY
-WURFL +ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256,
default=2).
Built with OpenSSL version : OpenSSL 3.1.0+quic 14 Mar 2023
Running on OpenSSL version : OpenSSL 3.1.0+quic 14 Mar 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.4.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : none
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
