Hi Willy, On Wed, Sep 13, 2023 at 3:25 PM Willy Tarreau <w...@1wt.eu> wrote: > > However, if the request on frontend does not have END_STREAM, then the > > backend also stays open. > > Till now this is expected since the client is expected to upload the > message's body, so a timeout might fire on the frontend side, but > nothing should close on the backend side. > > > When the backend sends a response (H2 HEADERS > > frame) with END_STREAM, the H2 stream is updated to "half-closed > > (remote)" but it is never properly considered closed. > > As annoying as it can be, this is also expected in order not to break > uploads. For example if you have an H2->H1 gateway after haproxy, > interrupting the upload before the end would require to break the > connection, which can be particularly expensive (i.e. need to close > and re-establish a new one for next request) and even be abused for > denial of services.
Thank you for the explanation of why it's set up the way it is. > > Instead, HTTP 502 is sent to the original requestor, and the session > > disconnect state is reported as SH--. > > I *suspect* that what is happening is that during the body forwarding > of the request (even if no data is being uploaded), we're seeing a > closed stream on the backend side and refraining from going further. > There might be a problem around this area. The response-before-completion > is often tricky to handle correctly, that's what I usually call the > "redirect-on-post" because that's the same principle as a client > uploading data (e.g. post on a webmail) and the server redirecting to > the login page due to an expired session. I guess you're facing a > combination of timing and events that causes the abortion of all the > request/response processing. Timing is only relevant to ensure frontend client's END_STREAM does not get delivered. If the frontend client is sending data (upload) then that is more likely to occur. Some very unfortunate network delays can also cause such a situation. In my testing, I can consistently reproduce this by simply closing the client stream with one second of delay. --- I set up a small PoC repository at https://github.com/sigv/grpcopen with a server and a client. There is a Ping endpoint, which works fine (the frontend client is first to close). There is also a Foobar endpoint, which is intentionally mangled, to ensure the backend server returns a gRPC error before the frontend client closes its side. Currently `client/main.go` sends both requests; to only observe the failing request, `ping(stream, ctx)` invocation can be removed from the end of the file. The only requirement to run it is to have Go available locally. The README file in the repository covers how to download and unpack Go, if you want to run this on a fresh VM. It also includes a minimal HAProxy configuration that I can reproduce the issue with. Hopefully having a sample application locally makes it easier for you to look at the raw traffic, and trace HAProxy itself. Please let me know if I can help out in any other way! Best regards, Valters Jansons
