[
https://issues.apache.org/jira/browse/CAMEL-26537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-26537 started by shashank.
----------------------------------------
> camel-core - TimerListenerManager stops for good when a listener is added or
> removed while it runs
> --------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26537
> URL: https://issues.apache.org/jira/browse/CAMEL-26537
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> Found with an interleaving test of {{TimerListenerManager}}.
> {{TimerListenerManager}} keeps its listeners in a {{LinkedHashSet}}. The
> timer thread iterates the set in {{run()}}, while {{addTimerListener}} and
> {{removeTimerListener}} change it with no lock, from other threads or from a
> listener running on the timer thread.
> If the set changes while the timer is in a listener that is not the last one,
> the iterator throws a {{ConcurrentModificationException}}. The {{try}} is
> around each listener call, not around the loop, so {{run()}} throws, and
> {{scheduleAtFixedRate}} never runs it again. The timer stops for good, and
> nothing is logged.
> {noformat}
> 1 timer thread : run() -> iterator.next() = A, A.onTimer() is running (B is
> next)
> 2 A itself, or another thread : addTimerListener(C) (or
> removeTimerListener(B))
> 3 timer thread : A.onTimer() returns -> iterator.next() ->
> ConcurrentModificationException
> {noformat}
> *The trigger that needs no race.* Two things combine:
> * A vault refresh task (AWS Secrets Manager, HashiCorp Vault, Spring Cloud
> Config, {{camel.vault.*.refreshEnabled}}) runs on the timer thread through
> {{DefaultPeriodTaskScheduler}}, and reloads the routes from its {{run()}}.
> * A restarted {{ServiceBusConsumer}} schedules its lock renewer in
> {{doStart}} (default {{PEEK_LOCK}} and {{maxAutoLockRenewDuration}}).
> The refresh task is scheduled before any route starts, and the renewer from
> the consumer's first start is never removed, so the refresh task is never the
> last listener. The first secret rotation therefore stops the timer. After
> that, no lock is renewed (long exchanges are delivered again) and no later
> rotation is picked up. This was traced in the code and reproduced with a
> test. It was not run against a real vault and Service Bus.
> *Other triggers:*
> * a period task scheduled from another thread (a Service Bus route started at
> runtime);
> * the JMX load statistics ({{loadStatisticsEnabled}}), whose listeners are
> added and removed when routes are added, removed or reloaded;
> * two threads adding at the same time can lose listeners, because
> {{LinkedHashSet}} is not thread safe.
> *Tests on main:*
> * a latch holds the timer thread in the first listener while another listener
> is added (or the next one removed);
> * a period task schedules another task from its own run.
> In both, the timer never calls a listener again.
> *Proposal:* keep the listeners in a {{CopyOnWriteArraySet}}, as
> {{DefaultBackOffTimer}} does. It keeps the insertion order, and the timer
> iterates a snapshot.
> Affected: main, 4.23.x, 4.22.x, 4.18.x. 4.14.x has the same code, but only
> the JMX statistics are affected there (it has no Service Bus renewer and
> schedules no period task after startup).
> Related, not part of this (follow-ups):
> * {{PeriodTaskScheduler}} cannot remove a task, so {{ServiceBusConsumer}}
> leaves its cancelled renewer behind on each restart.
> * {{run()}} catches only {{Exception}}. An {{Error}} thrown by a listener
> still stops the timer, with nothing logged.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
