shashank created CAMEL-26535:
--------------------------------
Summary: camel-kamelet - adding a route that uses a Kamelet can
deadlock with a toD creating the route of another Kamelet (model lock vs
Kamelet component lock)
Key: CAMEL-26535
URL: https://issues.apache.org/jira/browse/CAMEL-26535
Project: Camel
Issue Type: Bug
Components: camel-kamelet, camel-core
Reporter: shashank
Assignee: shashank
Found with a TLA+ model of the route-management locks of camel-core and
camel-kamelet (the lock of the model, the lock of the Kamelet component, the
route lock of the CamelContext), then reproduced on the real classes.
Adding a route that uses a Kamelet while the CamelContext runs can deadlock for
good with an exchange that sends to a new Kamelet endpoint (such as
{{toD("kamelet:b?x=${header.x}")}}). The two threads take the same two locks in
opposite orders:
* adding routes ({{addRoutes}}, a route reload, a route created from a template
at runtime with {{addRouteFromTemplate}} or templated routes, rest definitions
added at runtime): {{DefaultModel.addRouteDefinitions}} holds the lock of the
model while it creates and starts the new route; creating the route resolves
its {{kamelet:a}} endpoint, whose {{doInit}} creates the route of the Kamelet
in {{KameletComponent.LifecycleHandler.createRouteForEndpoint}}, which takes
the lock of the Kamelet component. Order: model, then Kamelet component.
* the toD resolves the new endpoint with no lock held, so
{{createRouteForEndpoint}} takes the lock of the Kamelet component first, and
{{addRouteFromKamelet}} then adds the route with {{addRouteDefinitions}}, which
takes the lock of the model. Order: Kamelet component, then model.
Short trace of the model:
{noformat}
1 toD thread : createRouteForEndpoint takes the Kamelet component lock
2 add thread : addRouteDefinitions takes the model lock
3 add thread : reifying the new route -> kamelet:a doInit -> wants the Kamelet
component lock
toD thread : addRouteFromKamelet -> addRouteDefinitions -> wants the model
lock -> deadlock
{noformat}
Both locks are taken with {{lock()}}, so both threads hang, and so does
everything that needs the lock of the model afterwards (adding and removing
routes, every later Kamelet route). A test with two hooks of the public API
({{ModelLifecycleStrategy}} and a route template converter) forces the
interleaving and {{ThreadMXBean.findDeadlockedThreads()}} reports the two
threads each waiting for the {{ReentrantLock}} the other holds. Without hooks,
adding 200 routes with new Kamelet endpoints while another thread sends 200
messages through toD to new Kamelet endpoints hung in each of 9 runs.
*Proposal:* the Kamelet component creates the route of a Kamelet holding the
lock of the model instead of a lock of its own (one lock order). Every path
through {{createRouteForEndpoint}} that creates a route already takes the lock
of the model inside it, so Kamelet route creation stays serialized. The model
does not expose its lock, so {{Model}} gets a new method {{getModelLock()}}
(upgrade guide entry for custom implementations of {{Model}}).
Related, not part of this: in the same model, {{stopRoute}} holds the route
lock of the CamelContext while the shutdown strategy waits for the in-flight
exchanges, and an in-flight exchange of that route that sends to a new Kamelet
endpoint needs that lock to start the Kamelet's route, so it waits until the
shutdown times out and the stop is forced (bounded by the shutdown timeout).
Affected: main, 4.23.x, 4.22.x, 4.18.x, 4.14.x (the same two lock orders; on
4.18.x and 4.14.x {{DefaultModel.addRouteDefinitions}} is {{synchronized}}).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
