Christian Schneider created SLING-13395:
-------------------------------------------
Summary: DistributionSubscriber imports the same package twice
when it is restarted during a topology change
Key: SLING-13395
URL: https://issues.apache.org/jira/browse/SLING-13395
Project: Sling
Issue Type: Bug
Components: Content Distribution
Reporter: Christian Schneider
Fix For: Content Distribution Journal Core 0.6.0
On a clustered instance, every topology change restarts all journal subscribers
on the leader, even when the leader does not change. If a package import is
still running at that moment, the restarted subscriber imports the same package
again in parallel.
How it happens:
* OnlyOnLeader unregisters its service on every TOPOLOGY_CHANGING event and
registers it again on TOPOLOGY_CHANGED. DistributionSubscriber has a static
reference to OnlyOnLeader (SLING-12639), so it is deactivated and activated
again on every topology change.
* DistributionSubscriber.deactivate() does not interrupt the poller thread
(SLING-9340) and does not wait for it, so an import that is already running
continues.
* The new subscriber instance starts at the last committed offset + 1, which is
the offset still being imported by the old thread. The same package is then
imported twice in parallel.
* Both imports write the BookKeeper state for the same subscriber, so one of
them can fail with an OakState0001 conflict on the subscriber store.
The effect is significant when imports are slow (for example, an import that
takes several minutes because of an expensive package builder) and the topology
changes frequently (for example, instances joining and leaving the cluster).
Several overlapping imports of the same offset were observed, each running for
minutes.
Proposed fix:
* OnlyOnLeader: don't unregister on TOPOLOGY_CHANGING; only update the
registration on TOPOLOGY_INIT/TOPOLOGY_CHANGED, and unregister only when the
local instance is no longer the leader.
* DistributionSubscriber.deactivate(): wake up a pending retry back-off and
wait, bounded (30 s), for an import that is still running before returning. No
locking between subscriber instances, to avoid hanging processing; a duplicate
import remains possible when an import takes longer than the wait or when
leadership really changes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
