shashank created CAMEL-26574:
--------------------------------
Summary: camel-netty - Client mode consumer: stop leaves the
connection and the reconnect running with a shared worker group, and the
reconnect blocks the event loop
Key: CAMEL-26574
URL: https://issues.apache.org/jira/browse/CAMEL-26574
Project: Camel
Issue Type: Bug
Components: camel-netty
Reporter: shashank
Assignee: shashank
Found with a TLA+ model of {{ClientModeTCPNettyServerBootstrapFactory}} (the
lifecycle thread starting and stopping the consumer, the event loops running
the close listener and the reconnect task, connect outcomes and the server
closing the connection chosen nondeterministically; a second model with two
consumers on one shared group of two event loops), then reproduced on the real
classes against an embedded Netty server on localhost.
A consumer with {{clientMode=true}} connects to a server and, with
{{reconnect=true}} (the default), connects again when the connection is lost.
Several things go wrong:
*1. A stop does not close the connection when the worker group is shared.*
{{stopServerBootstrap()}} never closes the channel; it only shuts down a worker
group the factory created itself. With {{workerGroup=#myGroup}} (the documented
way to share event loops between routes) the stopped consumer keeps its
connection: whatever the server sends after the stop is read and dropped (the
server sees it as delivered). The close listener and the scheduled reconnect
stay active too:
* a reconnect scheduled before the stop (the server was down at start, or had
just closed the connection) connects after the stop;
* when the server closes the leftover connection after the stop, the stopped
consumer connects again;
* a route restart opens a second connection while the first one stays open, and
both feed the route.
TLA+ trace (main): start while the server is down (reconnect scheduled), stop:
after the stop the reconnect is still scheduled ({{StopNoReconnect}} violated
in 5 steps); start, stop, start: two open connections ({{OneConnection}}
violated).
*2. The reconnect waits on an event loop.* The reconnect task runs on an event
loop and calls {{openChannel}}, which waits for the connect with
{{awaitUninterruptibly()}}.
* When the new channel gets the same event loop ({{workerCount=1}}, or a shared
group whose round robin returns the loop of the task), Netty throws
{{BlockingOperationException}}. The task logs "Will attempt again in ...
millis. This exception is ignored." but schedules nothing; the connect goes on
without the close listener, so after the next connection loss the consumer
never reconnects.
* When two reconnect tasks on a shared group of two loops (two consumers whose
server restarted, or a stopped consumer's leftover reconnect and the restarted
one) each connect a channel on the loop of the other, both loops wait for a
connect that only the other loop can complete. The connect timeout is scheduled
on the same blocked loops, so this never ends: every channel of the shared
group (other routes, producers) hangs, and a route start waiting for its
connect on one of these loops never returns (nor does the shutdown of the
context, which waits for that start).
TLA+ trace (two consumers, shared group of two loops): both connections drop;
the reconnect of consumer 2 (loop 2) connects a channel on loop 1 and waits;
the reconnect of consumer 1 (loop 1) connects a channel on loop 2 and waits:
{{NoLoopDeadlock}} violated in 5 steps. Another trace: a reconnect whose
channel lands on its own loop ends the reconnects of that consumer
({{ChainsKept}} violated in 3 steps).
Reproduced on main (1b12eea752e6) with an embedded Netty server on localhost:
* {{workerGroup=#shared}}, route stopped: the server side of the connection is
still open 10 s later; with {{reconnect=true}}, the server closing that
connection after the stop is followed by a new connection from the stopped
consumer; after a stop and a start two connections are open and messages on
both reach the route.
* {{workerCount=1}}: the consumer connects once and never again after the
server closes the connection ({{BlockingOperationException}} in the log).
* Two client consumers on one shared group of two loops, the server closing
both connections at once five times: fails 20 of 20 runs.
* A hook-free loop of 200 stop/start cycles (shared group, the server closing
every connection after 0-3 ms): a connection after the stop in 105 of the first
110 cycles, then the start of the next cycle hung on the event loop deadlock
(the thread dump shows both loops in {{openChannel}} ->
{{awaitUninterruptibly}}). With the change: 0 of 200, no hang.
*Proposal*
* {{stopServerBootstrap}} cancels the scheduled reconnect and closes the
channel and a connect in progress, and waits for the close (up to
{{shutdownTimeout}}, as the server mode does, not when called on the event loop
of the channel).
* The reconnect does not wait on the event loop: the connect outcome is handled
by a listener (track the channel and its close listener, or schedule the next
reconnect).
* The reconnect state (channel, connect in progress, scheduled task, stopping
flag, a generation per start) is guarded by one lock; the close listener, the
reconnect task and the connect listener act only if the consumer is not
stopping and belongs to the current start, so a stop racing a reconnect leaves
nothing behind and a restart cannot run two reconnect chains.
* No option or default changes. The code is the same on camel-4.23.x, 4.18.x
and 4.14.x.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
