shashank created CAMEL-26537:
--------------------------------
Summary: camel-core - TimerListenerManager stops for good when a
listener is added or removed while it runs
Key: CAMEL-26537
URL: https://issues.apache.org/jira/browse/CAMEL-26537
Project: Camel
Issue Type: Bug
Components: camel-core
Reporter: shashank
Assignee: shashank
Found with an interleaving test of {{TimerListenerManager}}.
{{TimerListenerManager}} keeps its listeners in a {{LinkedHashSet}}. The timer
thread iterates the set in {{run()}}, while {{addTimerListener}} and
{{removeTimerListener}} change it with no lock, from other threads or from a
listener running on the timer thread.
If the set changes while the timer is in a listener that is not the last one,
the iterator throws a {{ConcurrentModificationException}}. The {{try}} is
around each listener call, not around the loop, so {{run()}} throws, and
{{scheduleAtFixedRate}} never runs it again. The timer stops for good, and
nothing is logged.
{noformat}
1 timer thread : run() -> iterator.next() = A, A.onTimer() is running (B is
next)
2 A itself, or another thread : addTimerListener(C) (or
removeTimerListener(B))
3 timer thread : A.onTimer() returns -> iterator.next() ->
ConcurrentModificationException
{noformat}
*The trigger that needs no race.* Two things combine:
* A vault refresh task (AWS Secrets Manager, HashiCorp Vault, Spring Cloud
Config, {{camel.vault.*.refreshEnabled}}) runs on the timer thread through
{{DefaultPeriodTaskScheduler}}, and reloads the routes from its {{run()}}.
* A restarted {{ServiceBusConsumer}} schedules its lock renewer in {{doStart}}
(default {{PEEK_LOCK}} and {{maxAutoLockRenewDuration}}).
The refresh task is scheduled before any route starts, and the renewer from the
consumer's first start is never removed, so the refresh task is never the last
listener. The first secret rotation therefore stops the timer. After that, no
lock is renewed (long exchanges are delivered again) and no later rotation is
picked up. This was traced in the code and reproduced with a test. It was not
run against a real vault and Service Bus.
*Other triggers:*
* a period task scheduled from another thread (a Service Bus route started at
runtime);
* the JMX load statistics ({{loadStatisticsEnabled}}), whose listeners are
added and removed when routes are added, removed or reloaded;
* two threads adding at the same time can lose listeners, because
{{LinkedHashSet}} is not thread safe.
*Tests on main:*
* a latch holds the timer thread in the first listener while another listener
is added (or the next one removed);
* a period task schedules another task from its own run.
In both, the timer never calls a listener again.
*Proposal:* keep the listeners in a {{CopyOnWriteArraySet}}, as
{{DefaultBackOffTimer}} does. It keeps the insertion order, and the timer
iterates a snapshot.
Affected: main, 4.23.x, 4.22.x, 4.18.x. 4.14.x has the same code, but only the
JMX statistics are affected there (it has no Service Bus renewer and schedules
no period task after startup).
Related, not part of this (follow-ups):
* {{PeriodTaskScheduler}} cannot remove a task, so {{ServiceBusConsumer}}
leaves its cancelled renewer behind on each restart.
* {{run()}} catches only {{Exception}}. An {{Error}} thrown by a listener still
stops the timer, with nothing logged.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
