Vladimir Dobos created CAMEL-22364:
--------------------------------------
Summary: ConcurrentModificationException on message header map
when using multiple camel-jms calls after each other in route
Key: CAMEL-22364
URL: https://issues.apache.org/jira/browse/CAMEL-22364
Project: Camel
Issue Type: Bug
Components: camel-jms
Affects Versions: 4.14.0, 4.13.0, 4.12.0, 4.11.0, 4.10.6, 4.9.0, 4.8.8,
4.6.0, 4.5.0, 4.4.5
Reporter: Vladimir Dobos
Attachments: access_history.txt, camel.log
When calling multiple camel-jms endpoints in sequence in one route, there is a
random possibility of ConcurrentModificationException on header map during
route execution when using more than 1 thread to run the route (see stacktrace
on line 12 in attached log file camel.log) for details.
I wrote simple program to simulate the issue
(https://github.com/vdobos-tr/TestJmsConcurrentModification), program contains
both client (RageCallMQ.java using jmh, run by jmh.sh) and backend (CamelMain,
executed usually from my IDE) part, client uses JMH to tax the camel-jms
endpoints and cause the exception. We use IBM MQ as a message broker (added
example compose in directory mq-compose, requires path modification to run).
Probability and time until the exception happens depends heavily on hardware
and general setup of the host machine. Our customers can easily replicate the
issue in their test environments running 20 consumers and simple route
(from(jms, in only).process().to(jms, inout).process(...).to(jms, out only))
whereas we had to run 50-75 threads and up to 6 orchestrations to get one
exception within 30 minutes of execution (from(jms, in only).process().[to(jms,
inout).process(...)<repeat 6 times>].to(jms, out only)). As a race condition,
unfortunatelly, the timing when the exception occuring is very random,
sometimes it can occur within first minute, sometimes it can take 30+ minutes
:(.
Exception occurs regardless if CachingConnectionFactory is used and regardles
of asyncConsumer setting. ConcurrentModificationException is caused by parallel
acces of two threads to headers map on message (I used custom HeadersMapFactory
and custom map implementation with some Proxy classes to trace the issue, see
package cz.trask.bi.mq.camel.debugmap in application)
I was able to trace the issue to change in camel 4.4.0
(https://issues.apache.org/jira/browse/CAMEL-20338, see file
access_history.txt, lines 78 to 200 and line 202 created by custom Headers Map
that shows thread acces causing the issue).
As far as I understand the problem, it is posible for doSend(...) on line 260
in JmsProducer to not finish (maybe OS schduler parking threads ?) until next
thread processing next part of route received response from backend and is
already sending next request (or final response) in route (header map iteration
on line 368 of JmsBinfding, where the stacktrace originates), during this
iteration original doSend(...) on line on line 260 in JmsProducer resumes and
hits lines 260-262, added by CAMEL-20338, causing
ConcurrentModificationException on next loop iteration in JmsBinding, line 368.
I tested the issue on camel 4.4.0, 4.8.3 - the version of apache camel we and
our customers are currently using, 4.10.6, 4.13.0 and also newly relased 4.14.0
and it occurs on all of them.
Apache camel releases before canel 4.4.0 (4.3.0, 4.2.0, 4.1.0, 4.0.0) do not
manifest the issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
