The haproxy -vv output is at the end of this message.
I got the built-in OCSP udpating mechanism working. Works beautifully.
Today I discovered that once an hour when the OCSP gets updated, haproxy
stops all its proxies and starts them back up. syslog:
Sep 27 15:00:01 - haproxy[3520801] Proxy web80 stopped (cumulated conns:
FE: 42, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy web stopped (cumulated conns:
FE: 1403, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_deny stopped (cumulated
conns: FE: 0, BE: 122).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_raspi1_81 stopped (cumulated
conns: FE: 0, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_raspi2_81 stopped (cumulated
conns: FE: 0, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_raspi3_81 stopped (cumulated
conns: FE: 0, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_smeagol_81 stopped
(cumulated conns: FE: 0, BE: 700).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_plex_32400_tls stopped
(cumulated conns: FE: 0, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_gitlab_8881 stopped
(cumulated conns: FE: 0, BE: 235).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_gitlab2_8881 stopped
(cumulated conns: FE: 0, BE: 180).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_artifactory_8082 stopped
(cumulated conns: FE: 0, BE: 0).
Sep 27 15:00:01 - haproxy[3520801] Proxy be_zabbix_81 stopped (cumulated
conns: FE: 0, BE: 969).
Sep 27 15:00:01 - haproxy[3545799] -:- [27/Sep/2023:15:00:01.668]
<OCSP-UPDATE> /etc/ssl/certs/local/REDACTED_org.wildcards.combined
.pem 1 "Update successful" 0 1
Sep 27 15:00:01 - haproxy[3545799] -:- [27/Sep/2023:15:00:01.795]
<OCSP-UPDATE> /etc/ssl/certs/local/REDACTED2.com.wildcards.combined.p
em 1 "Update successful" 0 1
Sep 27 15:00:01 - haproxy[3520801] -:- [27/Sep/2023:15:00:01.944]
<OCSP-UPDATE> /etc/ssl/certs/local/REDACTED_org.wildcards.combined
.pem 1 "Update successful" 0 2
Sep 27 15:00:02 - haproxy[3520801] -:- [27/Sep/2023:15:00:01.998]
<OCSP-UPDATE> /etc/ssl/certs/local/REDACTED2.com.wildcards.combined.p
em 1 "Update successful" 0 2
The really irritating effect is that once an hour, my Zabbix server
records an event saying haproxy has been restarted:
https://imgur.com/a/WPkKoFa
(imgur will claim the image has mature content. it doesn't.)
It looks like the only thing that resets back to zero on the stats page
is the uptime in the "status" column for each backend. That's good
news, but I would hope for none of the data to be reset.
I have one big concern, which may be unfounded: I'm worried that the
proxies going down will mean that in-flight connections will be
terminated. I'm guessing that the work for seamless reloads will ensure
that doesn't happen, I just want to be sure.
Not knowing a lot about how haproxy is architected, I do not know if
there is some reason that the backends have to be cycled. Seems like
only frontends that listen with TLS would need that. I would hope it
would be possible to even avoid that ... maybe have OCSP data be copied
from a certain memory location every time a frontend needs it, and when
OCSP gets updated, overwrite the data in that memory location in a
thread-safe way. I know a fair amount about thread safety in Java, but
nothing about it in C.
Final questions for today:
1) Can the OCSP update interval be changed? I don't recall exactly what
the validity for a LetsEncrypt OCSP response is, but I know it was at
least 24 hours, and I think it might have even been as long as a week. I
would like to increase the interval to 8-12 hours if I can.
2) There are two certs being used in my setup, and haproxy logs updates
for both of them twice. I would have hoped for that to only happen
once. I'm a bit mystified by the fact that it is done twice. I would
have expected either one time or four times ... I have one frontend that
listens with TLS, with four bind lines all using exactly the same
certificate list. (one TCP, and three UDP)
-------------
HAProxy version 2.8.3-0499db-3 2023/09/14 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2
2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.3.html
Running on: Linux 6.1.0-1022-oem #22-Ubuntu SMP PREEMPT_DYNAMIC Wed Sep
6 08:19:34 UTC 2023 x86_64
Build options :
TARGET = linux-glibc
CPU = native
CC = cc
CFLAGS = -O2 -march=native -g -Wall -Wextra -Wundef
-Wdeclaration-after-statement -Wfatal-errors -Wtype-limits
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond
-Wnull-dereference -fwrapv -Wno-address-of-packed-member
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
-Wno-missing-field-initializers -Wno-cast-function-type
-Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_QUIC=1
USE_PCRE2_JIT=1
DEBUG =
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY
+CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE
-LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH
-MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL
-OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL
-PROCCTL -PROMEX -PTHREAD_EMULATION +QUIC +RT +SHM_OPEN -SLZ +SSL
-STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY
-WURFL +ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256,
default=48).
Built with OpenSSL version : OpenSSL 3.1.2+quic 1 Aug 2023
Running on OpenSSL version : OpenSSL 3.1.2+quic 1 Aug 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.39 2021-10-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.4.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : none
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
