I tested enabling HTTP/2 on the frontend for some of our sites today and
immediately started getting a flurry of failures. Browsers (at least
Chrome) showed a lot of SPDY protocol errors and the HAProxy logs had a lot
of lines ending in
https_domain_redacted/<NOSRV> -1/-1/-1/-1/100 400 187 - - PR-- 49/2/0/0/0
0/0
There were no useful or interesting errors logged to syslog. No sign of any
resources being exhausted (conntrack seems fine, etc). The times varied but
Ta was always low (usually around 100ms). I have not been able to reproduce
this issue in a staging environment, so it may be something "real browsers"
do that doesn't show up with h2load et al.
Turning off HTTP/2 (setting "alpn http/1.1") completely solves the problem.
The following timeouts are set on all of the affected frontends:
retries 3
timeout client 9s
timeout connect 3s
timeout http-keep-alive 5m
tcp-request inspect-delay 4s
option http-server-close
Additionally, we set maxconn to a very high value (20480).
Backends generally have timeout server set to a largeish value (90-300
seconds, depending on the backend).
Anything jump out at anyone?
--
James Brown
Systems & Network Engineer
EasyPost
