Hi all, Looking for some help with a networking issue we've been debugging for several days. We use haproxy to TCP load-balance between Kafka Connectors and a Percona MySQL cluster. In this set-up, the connectors (i.e., Java JDBC) maintain long-running connections and read the database binlogs. This includes regular polling.
We have seen connection issues starting with haproxy 2.4 and persisting through 2.8 which result in the following errors in MySQL: 2023-07-31T17:25:45.745607Z 3364649 [Note] Got an error reading communication packets As you can see, this doesn't include a host or user and is happening early in the connection. The host cache shows handshake errors here regularly accumulating. We were unable to see errors on the haproxy side with tcplog on and have been unable to get useful information from tcpdump, netstat, etc. We are aware FE/BE connection closure behavior changed in 2.4. The 2.4 option of idle-close-on-response seemed like a possible solution but isn't compatible with mode tcp, so we're not sure what's happening here or next steps for debugging. Appreciate any help or guidance here. We're running haproxy in Kubernetes using the official container, and are also not seeing any issues with current haproxy versions with our other (Python) applications. A simplified version of our config: global daemon maxconn 25000 defaults balance roundrobin option dontlognull option redispatch timeout http-request 5s timeout queue 1m timeout connect 4s timeout client 50s timeout server 30s timeout http-keep-alive 10s timeout check 10s retries 3 frontend main_writer bind :3306 mode tcp timeout client 30s timeout client-fin 30s default_backend main_writer backend main_writer mode tcp balance leastconn option httpchk timeout server 30s timeout tunnel 3h server db1 <ip>:3306 check port 9200 on-marked-down shutdown-sessions weight 100 inter 3s rise 1 fall 2 server db2 <ip>:3306 check port 9200 on-marked-down shutdown-sessions weight 100 backup server db3 <ip>:3306 check port 9200 on-marked-down shutdown-sessions weight 100 backup *David GreenwaldSenior Site Reliability engineerdavid.greenw...@discogsinc.com <david.greenw...@discogsinc.com>* -- The contents of this communication are confidential. If you are not the intended recipient, please immediately notify the sender by reply email and delete this message and its attachments, if any.