On Tue, Sep 22, 2026 at 08:10:19PM +0900, ygkat wrote:
> Hi,
> We've been chasing a slow QUIC connection/memory leak on a production
> HTTP/3 edge (running 3.2.21) and traced it down to a specific ordering
> issue in the mux_quic / quic_conn interaction. Confirmed the same code
> path is still present in v3.2.23 and on current haproxy-3.2 branch HEAD,
> so writing it up here.
> If a client's first post-handshake flight on a QUIC connection contains
> RESET_STREAM and STOP_SENDING on stream 0 without ever completing an
> HTTP/3 HEADERS frame, and this becomes visible to HAProxy before the
> mux's very first qcc_io_cb() run for that connection, the resulting
> quic_conn is never freed. On one of our production nodes this had
> accumulated to about 1.22M stuck connections over 55 days, consuming
> around 80GB in the quic_conn_rxbuf pool alone (a fixed 64KiB buffer held
> per connection for its whole lifetime). timeout client and timeout
> http-request were both configured with finite values (65s and 10s) on
> the affected frontend, which rules out the more obvious "no timeout
> configured" explanation.
> The mechanism, as far as we can tell: qcc_recv_reset_stream() creates
> the qcs via qcc_get_qcs()->h3_attach()->qcs_wait_http_req(), which sets
> nb_hreq=1. It then calls qcs_close_remote() but never calls
> qcc_refresh_timeout() anywhere in the function, unlike its sibling
> qcc_recv_stop_sending() which does (guarded by !qcc->nb_hreq). Since the
> local side of the stream never closes, qcs_is_completed() never becomes
> true and nb_hreq stays at 1 forever. If the connection-level error is
> already visible by the time qcc_io_cb() runs for the first time,
> qcc_io_send() bails out before qcc_app_init() ever runs, so total stays
> 0 for that pass and qcc_io_cb()'s own "if (total) qcc_refresh_timeout()"
> is skipped too. qcc->task gets a finite expire computed once at
> qmux_init() time but qmux_init() never calls task_queue() on it, and
> since qcc_refresh_timeout() is the only place that does, the task simply
> never gets inserted into the scheduler's wait queue. Meanwhile
> qcc_is_dead() checks nb_hreq before the transport-error flags:
> static inline int qcc_is_dead(const struct qcc *qcc)
> {
> /* Maintain connection if there is still request streams active.
> */
> if (qcc->nb_hreq)
> return 0;
> if (qcc->flags & (QC_CF_ERR_CONN|QC_CF_ERRL_DONE) ||
> !qcc->task) {
> return 1;
> }
> return 0;
> }
> so with nb_hreq stuck at 1 this always returns 0 even once
> QC_CF_ERR_CONN is set. And quic_conn_release() refuses to run at all
> while the mux is still considered alive (BUG_ON(qc->mux_state ==
> QC_MUX_READY) in src/quic_conn.c), which with the above never changes
> for these connections.
> We attached gdb (read-only, using a symbol-bearing debug build as the
> symbol source against the running production binary) and confirmed the
> whole chain field for field on a sample of the stuck connections: all of
> them sitting in the per-thread quic_conns_clo list rather than the
> active quic_conns list, qc->idle_timer_task == NULL, qc->mux_state ==
> QC_MUX_READY, qcc->nb_sc == 0, qcc->nb_hreq == 1 (checked 2000
> connections from the closing list, 2000/2000 had nb_hreq == 1, so this
> isn't a rare edge case), qcc->flags == QC_CF_ERR_CONN only, qcc->app_st
> == QCC_APP_ST_NULL (qcc_app_init() never ran, which is also why there
> was exactly one qcs per connection instead of the usual control stream
> plus request stream), qcc->task non-NULL but not present in either the
> wait queue or run queue, and the single qcs on each connection sitting
> at id 0, state QC_SS_HREM, flags RECV_RESET|HREQ_RECV|TO_RESET|
> SIZE_KNOWN, with rx.offset_max == 0 (zero bytes of stream data ever
> arrived).
> We reproduced the same state locally, though we deliberately avoided
> building a client that forces the datagram-coalescing timing on the
> wire, since that would double as a working remote resource-exhaustion
> trigger against any QUIC-enabled HAProxy. Instead we forced the
> equivalent ordering from the server side in an instrumented debug build,
> gating qcc_io_cb()'s first pass behind a short delay while qcc->app_st
> was still QCC_APP_ST_NULL, and drove it with an ordinary, unmodified
> aioquic client that just cancels stream 0 via the library's normal
> reset/stop_sending API before ever sending a complete request. That
> reproduced the identical state (matched production on 14 of 16 checked
> fields directly; the remaining 2 we could only infer since they're
> behind a struct that's private to h3.c and not visible from mux_quic.c
> instrumentation). We suspect, based on qc->idle_start and qcs->start
> being identical on the production samples, that on the wire this
> ordering happens naturally whenever a client's abort frames land in the
> same UDP datagram as its Handshake-epoch Finished message, but we
> haven't confirmed that with an actual packet capture.
> The fix we validated against the reproduced state is to stop letting
> nb_hreq override an already-confirmed transport error in qcc_is_dead():
> static inline int qcc_is_dead(const struct qcc *qcc)
> {
> + /* A connection already dead at transport level can never
> + * complete any request, so nb_hreq must not keep it alive.
> + */
> + if (qcc->flags & (QC_CF_ERR_CONN|QC_CF_ERRL_DONE))
> + return 1;
> +
> /* Maintain connection if there is still request streams active.
> */
> if (qcc->nb_hreq)
> return 0;
> if (qcc->flags & (QC_CF_ERR_CONN|QC_CF_ERRL_DONE) ||
> !qcc->task) {
> return 1;
> }
> return 0;
> }
> With just this, the connection is reclaimed on the very next tasklet
> pass, no timer involved, and a normal completed H3 GET tears down
> identically with or without the patch. We also tried adding a
> qcc_refresh_timeout() call inside qcc_recv_reset_stream() itself,
> mirroring what qcc_recv_stop_sending() already does. That works too,
> but only reclaims the connection once timeout http-request/timeout
> client elapses, and does nothing if neither is configured, so we'd
> suggest the qcc_is_dead() change as the primary fix. The
> qcc_recv_reset_stream() addition still seems worth doing on its own
> merits though, since it brings it in line with its sibling function
> regardless of this specific bug.
> Happy to share the instrumented diff or the gdb session in more detail
> if useful. We don't have a packet capture confirming the exact trigger
> in the wild yet, production traffic is TLS end to end and we didn't have
> key logging enabled at the time, so the client-side behavior producing
> this is still inferred rather than directly observed.
Hi,
Thanks for your analysis. I'm not sure to agree yet with the proposed
fix as this could have unwanted consequences, most notably with the
backend support (which the proposed patch does not support here).
However, a localized fix in qcc_recv_reset_stream() is at least
mandatory here. I'm working on this issue to ensure there is no similar
problem via different code paths.
--
Amaury Denoyelle
