[
https://issues.apache.org/jira/browse/CAMEL-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-26570 started by shashank.
----------------------------------------
> camel-milo - the loss report of a consumer started without a connection runs
> on the request pool and can deadlock the connection
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26570
> URL: https://issues.apache.org/jira/browse/CAMEL-26570
> Project: Camel
> Issue Type: Bug
> Components: camel-milo
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> Found with a TLA+ model of the camel-milo client connection, then reproduced
> with the embedded Milo server of the tests.
> Since CAMEL-25835 a consumer that starts while its connection is not up (the
> server cannot be reached, or Camel is rebuilding the connection) reports its
> nodes with {{Bad_NoCommunication}}. {{SubscriptionManager.registerItems}}
> hands that report to the {{MiloClient}} pool of the connection (CAMEL-25713),
> and {{MiloClientConsumer}} runs the route synchronously on that thread. The
> same pool completes the producer requests ({{outcomeOf}}:
> {{whenCompleteAsync(..., executor)}}) and has as many threads as there are
> processors: one in a container limited to one CPU.
> When the route waits for a request of the same connection (an awaited write,
> read or method call: {{defaultAwaitWrites=true}} or the {{CamelMiloAwait}}
> header) that is sent once the connection is up, its outcome is queued behind
> the thread that waits for it. With as many such consumers as the pool has
> threads:
> * their exchanges wait for ever, and so does every other awaited request of
> the connection;
> * the values stop for every consumer of the connection: the waiting report
> holds the lock of its node ({{Subscription.report}} is synchronized), and the
> delivery queue of the shared subscription waits behind the first value of
> that node;
> * stopping the context waits for its timeouts and for the interrupt of the
> waiting threads.
> The request reaches the connection after it is up when the error handler
> delivers it again (while there is no connection it fails at once with "No
> connected"), or when the route is slower than the connection attempt:
> {code:java}
> errorHandler(defaultErrorHandler().maximumRedeliveries(600).redeliveryDelay(100));
> from("milo-client:opc.tcp://localhost:4840?node=RAW(ns=2;s=valve1)")
> .setBody(constant(new Variant("ack")))
>
> .to("milo-client:opc.tcp://localhost:4840?node=RAW(ns=2;s=valve1Ack)&defaultAwaitWrites=true");
> {code}
> Not affected: requests that are not awaited (the default), routes that do not
> wait for a request after the connection came up, and virtual threads with
> more than one processor.
> Short trace (model, one thread):
> {noformat}
> reconnect thread: performConnect (outside the lock)
> consumer: registerItems: connected == null -> executor.execute(report)
> reconnect thread: putSubscriptions, connected = C1
> executor thread: report -> route -> write on C1, waits for its outcome
> Milo thread: answer -> whenCompleteAsync(executor): queued behind the
> waiting thread
> {noformat}
> On main (1b12eea752e6), with one consumer route like the above per processor
> and the server reachable a moment after the routes started, none of the
> writes completed in 30 s; with one consumer fewer they all completed. A
> manager with a one-thread request pool whose consumer writes from the report
> once the connection is up times out the same way.
> *Proposal*: deliver that report on Milo's shared executor
> ({{Stack.sharedExecutor()}}, the cached pool the clients deliver the values
> on), not on the request pool. It keeps generation 0, so it is still dropped
> once a connection has delivered a value. The model holds with this change.
> Affects main only (4.24.0-SNAPSHOT: CAMEL-25835 and CAMEL-25713 are not
> released).
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
