[
https://issues.apache.org/jira/browse/CAMEL-26535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-26535 started by shashank.
----------------------------------------
> camel-kamelet - adding a route that uses a Kamelet can deadlock with a toD
> creating the route of another Kamelet (model lock vs Kamelet component lock)
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26535
> URL: https://issues.apache.org/jira/browse/CAMEL-26535
> Project: Camel
> Issue Type: Bug
> Components: camel-core, camel-kamelet
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> Found with a TLA+ model of the route-management locks of camel-core and
> camel-kamelet (the lock of the model, the lock of the Kamelet component, the
> route lock of the CamelContext), then reproduced on the real classes.
> Adding a route that uses a Kamelet while the CamelContext runs can deadlock
> for good with an exchange that sends to a new Kamelet endpoint (such as
> {{toD("kamelet:b?x=${header.x}")}}). The two threads take the same two locks
> in opposite orders:
> * adding routes ({{addRoutes}}, a route reload, a route created from a
> template at runtime with {{addRouteFromTemplate}} or templated routes, rest
> definitions added at runtime): {{DefaultModel.addRouteDefinitions}} holds the
> lock of the model while it creates and starts the new route; creating the
> route resolves its {{kamelet:a}} endpoint, whose {{doInit}} creates the route
> of the Kamelet in
> {{KameletComponent.LifecycleHandler.createRouteForEndpoint}}, which takes the
> lock of the Kamelet component. Order: model, then Kamelet component.
> * the toD resolves the new endpoint with no lock held, so
> {{createRouteForEndpoint}} takes the lock of the Kamelet component first, and
> {{addRouteFromKamelet}} then adds the route with {{addRouteDefinitions}},
> which takes the lock of the model. Order: Kamelet component, then model.
> Short trace of the model:
> {noformat}
> 1 toD thread : createRouteForEndpoint takes the Kamelet component lock
> 2 add thread : addRouteDefinitions takes the model lock
> 3 add thread : reifying the new route -> kamelet:a doInit -> wants the
> Kamelet component lock
> toD thread : addRouteFromKamelet -> addRouteDefinitions -> wants the model
> lock -> deadlock
> {noformat}
> Both locks are taken with {{lock()}}, so both threads hang, and so does
> everything that needs the lock of the model afterwards (adding and removing
> routes, every later Kamelet route). A test with two hooks of the public API
> ({{ModelLifecycleStrategy}} and a route template converter) forces the
> interleaving and {{ThreadMXBean.findDeadlockedThreads()}} reports the two
> threads each waiting for the {{ReentrantLock}} the other holds. Without
> hooks, adding 200 routes with new Kamelet endpoints while another thread
> sends 200 messages through toD to new Kamelet endpoints hung in each of 9
> runs.
> *Proposal:* the Kamelet component creates the route of a Kamelet holding the
> lock of the model instead of a lock of its own (one lock order). Every path
> through {{createRouteForEndpoint}} that creates a route already takes the
> lock of the model inside it, so Kamelet route creation stays serialized. The
> model does not expose its lock, so {{Model}} gets a new method
> {{getModelLock()}} (upgrade guide entry for custom implementations of
> {{Model}}).
> Related, not part of this: in the same model, {{stopRoute}} holds the route
> lock of the CamelContext while the shutdown strategy waits for the in-flight
> exchanges, and an in-flight exchange of that route that sends to a new
> Kamelet endpoint needs that lock to start the Kamelet's route, so it waits
> until the shutdown times out and the stop is forced (bounded by the shutdown
> timeout).
> Affected: main, 4.23.x, 4.22.x, 4.18.x, 4.14.x (the same two lock orders; on
> 4.18.x and 4.14.x {{DefaultModel.addRouteDefinitions}} is {{synchronized}}).
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
