[
https://issues.apache.org/jira/browse/CAMEL-25093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shashank reassigned CAMEL-25093:
--------------------------------
Assignee: shashank
> Thread-Safety: Concurrent updateRoutes() Against a Single CamelContext Causes
> ConcurrentModificationException
> -------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25093
> URL: https://issues.apache.org/jira/browse/CAMEL-25093
> Project: Camel
> Issue Type: Improvement
> Components: camel-core, camel-xml-io
> Affects Versions: 4.14.2
> Reporter: Randheer Chauhan
> Assignee: shashank
> Priority: Minor
> Attachments: analyse-the-downstream-calls-abundant-brook.md,
> concurrency-issues-camelcontext-route-bean-registration.md
>
>
> h2. Background
> * We use {{{}camel-spring-boot{}}}, which provides a single {{CamelContext}}
> within the Spring Boot application.
> * During startup, necessary Camel components and beans are registered into
> this single {{{}CamelContext{}}}.
> * After the application is running, dynamically-loaded content is deployed
> into it. Each deployment unit contains a Camel XML DSL file with one or more
> route definitions and bean definitions used by those routes.
> * To register these routes and beans into the running {{{}CamelContext{}}},
> the following {{camel-support}} API is used:
> {code:java}
> PluginHelper.getRoutesLoader(camelContext).updateRoutes(resource) {code}
> * The number of deployment units can range from a few hundred to several
> thousand per application instance.
> * Once all units are deployed and their routes and beans are registered, the
> application is marked as ready.
> ----
> h2. Problem Statement
> * When the above API is invoked *sequentially* for each XML DSL
> {{{}Resource{}}}, it works correctly and registers all routes and beans as
> expected.
> * When the API is invoked *concurrently from multiple threads* — each thread
> loading a different XML DSL resource into the same {{CamelContext}} — it
> fails with {{ConcurrentModificationException}} originating from multiple
> Camel-internal paths.
> * The stacktraces below capture the distinct failure patterns observed. All
> application-specific frames have been removed; only {{org.apache.camel}} and
> {{java.base}} frames are retained.
> ----
> h2. Steps to Reproduce
> # Start a Spring Boot application with {{camel-spring-boot}} (single
> {{{}CamelContext{}}}).
> # After context startup, prepare multiple independent XML DSL resource files
> — each containing its own route definitions and bean definitions.
> # Submit all resources for loading concurrently via a thread pool, calling
> {{PluginHelper.getRoutesLoader(camelContext).updateRoutes(resource)}} from
> each thread.
> # Observe {{ConcurrentModificationException}} failures from the
> Camel-internal paths listed below.
> The failures do not occur when the same resources are loaded sequentially
> (one {{updateRoutes}} call at a time on the same thread).
> ----
> h2. Observed Failures
> h3. Failure 1 — {{ArrayList}} iterator race in {{XmlRoutesBuilderLoader}}
> The anonymous {{configureCamel}} callback in {{XmlRoutesBuilderLoader}}
> iterates a shared {{ArrayList}} at line 195 while another concurrent thread
> structurally modifies it. The iterator's fail-fast {{checkForComodification}}
> detects the change and throws.
> *Camel classes implicated:*
> * {{XmlRoutesBuilderLoader}} — anonymous configure callback iterates a
> shared {{ArrayList}} (line 195)
> * {{RouteBuilder.checkInitialized}} — reads shared state (line 908)
> * {{DefaultRoutesLoader.updateRoutes}} — invoked concurrently from multiple
> threads (line 302)
>
> {code:java}
> java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification
> at java.util.ArrayList$Itr.next
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader$1.configureCamel(XmlRoutesBuilderLoader.java:195)
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader$1.configure(XmlRoutesBuilderLoader.java:129)
> at
> org.apache.camel.builder.RouteBuilder.checkInitialized(RouteBuilder.java:908)
> at
> org.apache.camel.builder.RouteBuilder.configureRoutes(RouteBuilder.java:832)
> at
> org.apache.camel.builder.RouteBuilder.updateRoutesToCamelContext(RouteBuilder.java:786)
> at
> org.apache.camel.impl.engine.DefaultRoutesLoader.updateRoutes(DefaultRoutesLoader.java:302)
> at org.apache.camel.spi.RoutesLoader.updateRoutes(RoutesLoader.java:111)
> ... [caller: route activation on parallel deployment thread]
> at java.util.concurrent.CompletableFuture$AsyncSupply.run
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> at java.lang.Thread.run {code}
> ----
> h3. Failure 2 — {{ArrayList.removeIf}} race in {{DefaultModel.addCustomBean}}
> Multiple concurrent threads each call
> {{XmlRoutesBuilderLoader.registerBeanDefinition}} → {{bindBean}} →
> {{DefaultCamelContext.addCustomBean}} → {{{}DefaultModel.addCustomBean{}}}.
> Inside {{{}addCustomBean{}}}, {{ArrayList.removeIf}} is called on a shared,
> unsynchronized list while another thread is simultaneously modifying that
> same list.
> *Camel classes implicated:*
> * {{DefaultModel.addCustomBean}} (line 855) — invokes {{ArrayList.removeIf}}
> on an unsynchronized shared list
> * {{DefaultCamelContext.addCustomBean}} (line 554) — delegates directly to
> {{DefaultModel}}
> * {{XmlRoutesBuilderLoader.bindBean}} (line 435) — calls {{addCustomBean}}
> without any external lock
> * {{XmlRoutesBuilderLoader.registerBeanDefinition}} (line 415) — entry point
> from each parallel thread
> {code:java}
> java.lang.RuntimeException: Error creating bean: <beanName> of type:
> <beanType>
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader.registerBeanDefinition(XmlRoutesBuilderLoader.java:423)
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader$1.configureCamel(XmlRoutesBuilderLoader.java:197)
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader$1.configure(XmlRoutesBuilderLoader.java:129)
> at
> org.apache.camel.builder.RouteBuilder.checkInitialized(RouteBuilder.java:908)
> at
> org.apache.camel.builder.RouteBuilder.configureRoutes(RouteBuilder.java:832)
> at
> org.apache.camel.builder.RouteBuilder.updateRoutesToCamelContext(RouteBuilder.java:786)
> at
> org.apache.camel.impl.engine.DefaultRoutesLoader.updateRoutes(DefaultRoutesLoader.java:302)
> at org.apache.camel.spi.RoutesLoader.updateRoutes(RoutesLoader.java:111)
> ... [caller: route activation on parallel deployment thread]
> at java.util.concurrent.CompletableFuture$AsyncSupply.run
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> at java.lang.Thread.run
> Caused by: java.util.ConcurrentModificationException
> at java.util.ArrayList.removeIf
> at java.util.ArrayList.removeIf
> at org.apache.camel.impl.DefaultModel.addCustomBean(DefaultModel.java:855)
> at
> org.apache.camel.impl.DefaultCamelContext.addCustomBean(DefaultCamelContext.java:554)
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader.bindBean(XmlRoutesBuilderLoader.java:435)
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader.registerBeanDefinition(XmlRoutesBuilderLoader.java:415)
> {code}
> ----
> h3. Failure 3 — {{HashMap.computeIfAbsent}} race in {{SimpleRegistry.bind}}
> Two concurrent threads call {{SimpleRegistry.bind}} simultaneously.
> {{SimpleRegistry}} is backed by a plain {{{}HashMap{}}}. Concurrent
> {{HashMap.computeIfAbsent}} calls from two threads cause a structural
> modification that the second thread detects as a concurrent modification.
> *Camel classes implicated:*
> * {{SimpleRegistry.bind}} (line 106) — uses {{HashMap.computeIfAbsent}} with
> no synchronization
> * {{DefaultRegistry.bind}} (line 202) — delegates to {{SimpleRegistry}}
> * {{XmlRoutesBuilderLoader.bindBean}} (line 431) — called concurrently per
> resource from each parallel thread
> {code:java}
> java.lang.RuntimeException: Error creating bean: <beanName> of type:
> <beanType>
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader.registerBeanDefinition(XmlRoutesBuilderLoader.java:423)
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader$1.configureCamel(XmlRoutesBuilderLoader.java:197)
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader$1.configure(XmlRoutesBuilderLoader.java:129)
> at
> org.apache.camel.builder.RouteBuilder.checkInitialized(RouteBuilder.java:908)
> at
> org.apache.camel.builder.RouteBuilder.configureRoutes(RouteBuilder.java:832)
> at
> org.apache.camel.builder.RouteBuilder.updateRoutesToCamelContext(RouteBuilder.java:786)
> at
> org.apache.camel.impl.engine.DefaultRoutesLoader.updateRoutes(DefaultRoutesLoader.java:302)
> at org.apache.camel.spi.RoutesLoader.updateRoutes(RoutesLoader.java:111)
> ... [caller: route activation on parallel deployment thread]
> at java.util.concurrent.CompletableFuture$AsyncSupply.run
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> at java.lang.Thread.run
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap.computeIfAbsent
> at org.apache.camel.support.SimpleRegistry.bind(SimpleRegistry.java:106)
> at org.apache.camel.support.DefaultRegistry.bind(DefaultRegistry.java:202)
> at org.apache.camel.spi.Registry.bind(Registry.java:57)
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader.bindBean(XmlRoutesBuilderLoader.java:431)
> at
> org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader.registerBeanDefinition(XmlRoutesBuilderLoader.java:415)
> {code}
> ----
> h2. Camel Classes Observed in Failures
> The following classes appear directly in the failure stacktraces. This is not
> an exhaustive list — a full audit of the route-loading and bean-registration
> path may reveal additional classes with similar thread-safety gaps.
> ||Camel Class||Unsafe Operation||Pattern||
> |{{org.apache.camel.dsl.xml.io.XmlRoutesBuilderLoader}}|{{{}registerBeanDefinition{}}},
> {{{}bindBean{}}}, anonymous {{configureCamel}} — race entry point|1, 2, 3|
> |{{org.apache.camel.impl.DefaultModel}}|{{addCustomBean}} —
> {{ArrayList.removeIf}} on shared unsynchronized list|2|
> |{{org.apache.camel.impl.DefaultCamelContext}}|{{addCustomBean}} — delegates
> to {{DefaultModel}} without locking|2|
> |{{org.apache.camel.support.SimpleRegistry}}|{{bind}} — plain
> {{HashMap.computeIfAbsent}} with no synchronization|3|
> |{{org.apache.camel.support.DefaultRegistry}}|{{bind}} — delegates to
> {{SimpleRegistry}}|3|
> |{{org.apache.camel.builder.RouteBuilder}}|{{{}checkInitialized{}}},
> {{{}configureRoutes{}}}, {{updateRoutesToCamelContext}} — shared state read
> under concurrent modification|1|
> |{{org.apache.camel.impl.engine.DefaultRoutesLoader}}|{{updateRoutes}} —
> invoked concurrently per resource, shares {{RouteBuilder}} state|1, 2, 3|
> |{{org.apache.camel.spi.RoutesLoader}}|{{updateRoutes}} — {{RoutesLoader}}
> SPI entry point for all patterns|1, 2, 3|
> ----
> h2. Expected Solution
> With a single {{CamelContext}} hosted in a Spring Boot application, the
> internal Camel APIs, SPIs, and classes involved in route and bean
> registration should be enhanced to support concurrent invocations of
> {{RoutesLoader.updateRoutes}} from multiple threads — each loading a
> different XML DSL resource. Based on the observed failures, the changes
> needed include at minimum:
> * *{{DefaultModel}} / {{DefaultCamelContext}}* — synchronize
> {{addCustomBean}} or replace the underlying {{ArrayList}} with a thread-safe
> equivalent so concurrent {{removeIf}} calls do not race.
> * *{{SimpleRegistry}}* — replace the backing {{HashMap}} with
> {{ConcurrentHashMap}} so concurrent {{bind}} calls from multiple threads are
> safe.
> * *{{XmlRoutesBuilderLoader}}* — ensure each {{updateRoutes}} invocation
> works with an independent, thread-locally-scoped builder instance; no mutable
> state should be shared across concurrent calls for different resources.
> * *{{RoutesLoader}} SPI* — document the thread-safety contract of
> {{updateRoutes}} so that callers have a clear expectation.
> A broader audit of the full route-loading and bean-registration call path is
> likely needed to surface any additional unsynchronized shared state not
> captured in these stacktraces.
>
> Attached code analysis reports that was done on the camel codebase.
> [^concurrency-issues-camelcontext-route-bean-registration.md]
> [^analyse-the-downstream-calls-abundant-brook.md]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
