Hi Luke. Am 21.01.2019 um 10:30 schrieb Luke Seelenbinder: > Hi all, > > One more bug (or configuration hole) from our transition to 1.9.x using > end-to-end h2 connections. > > After enabling h2 backends (technically `server … alpn h2,http/1.1`), we > began seeing a high number of backend /server/ connection resets. A > reasonable number of client-side connection resets due to timeouts, etc., is > normal, but the server connection resets were new. > > I believe the root cause is that our backend servers are NGINX servers, which > by default have a 1000 request limit per h2 connection > (https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_requests). > As far as I can tell there's no way to set this to unlimited. That resulted > in NGINX resetting the HAProxy backend connections and thus resulted in user > requests being dropped or returning 404s (oddly enough; though this may be as > a result of the outstanding bug related to header manipulation and HTX mode).
Do you have such a info in the nginx log? "http2 flood detected" It's the message from this lines https://trac.nginx.org/nginx/browser/nginx/src/http/v2/ngx_http_v2.c#L4517 > This wouldn't be a problem if one of the following were true: > > - HAProxy could limit the number of times it reused a connection Can you try to set some timeout values for `timeout http-keep-alive` https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#timeout%20http-keep-alive I assume that this timeout could be helpful because of this block in the doc https://cbonte.github.io/haproxy-dconv/1.9/configuration.html ``` - KAL : keep alive ("option http-keep-alive") which is the default mode : all requests and responses are processed, and connections remain open but idle between responses and new requests. ``` and this code part https://github.com/haproxy/haproxy/blob/v1.9.0/src/backend.c#L1164 > - HAProxy could retry a failed request due to backend server connection reset > (possibly coming in 2.0 with L7 retries?) Mind you to create a issue for that if there isn't one already? > - NGINX could set that limit to unlimited. Isn't `unsigned int` not enought ? How many idle connections do you have for how long time? > Our http-reuse is set to aggressive, but that doesn't make much difference, I > don't think, since safe would result in the same behavior (the connection is > reusable…but only for a limited number of requests). > > We've worked around this by only using h/1.1 on the backends, which isn't a > big problem for us, but I thought I would raise the issue, since I'm sure a > lot of folks are using haproxy <-> nginx pairings, and this is a bit of a > subtle result of that in full h2 mode. Can you try to increase the max-requests to 200000 in nginx The `max_requests` is defined as `ngx_uint_t` which is `unsigned int` I have found this in the nginx source. https://www.nginx.com/resources/wiki/extending/api/main/#ngx-uint-t https://trac.nginx.org/nginx/browser/nginx/src/http/v2/ngx_http_v2_module.h#L27 https://trac.nginx.org/nginx/browser/nginx/src/http/v2/ngx_http_v2_module.c#L85 > Thanks again for such great software—I've found it pretty fantastic to run in > production. :) Just for my curiosity, have you seen any changes for your solution with the htx /H2 e2e? > Best, > Luke Best regards Aleks > — > Luke Seelenbinder > Stadia Maps | Founder > stadiamaps.com >
