[
https://issues.apache.org/jira/browse/SLING-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Schneider reassigned SLING-13395:
-------------------------------------------
Assignee: Christian Schneider
> DistributionSubscriber imports the same package twice when it is restarted
> during a topology change
> ---------------------------------------------------------------------------------------------------
>
> Key: SLING-13395
> URL: https://issues.apache.org/jira/browse/SLING-13395
> Project: Sling
> Issue Type: Bug
> Components: Content Distribution
> Reporter: Christian Schneider
> Assignee: Christian Schneider
> Priority: Major
> Fix For: Content Distribution Journal Core 0.6.0
>
>
> On a clustered instance, every topology change restarts all journal
> subscribers on the leader, even when the leader does not change. If a package
> import is still running at that moment, the restarted subscriber imports the
> same package again in parallel.
> How it happens:
> * OnlyOnLeader unregisters its service on every TOPOLOGY_CHANGING event and
> registers it again on TOPOLOGY_CHANGED. DistributionSubscriber has a static
> reference to OnlyOnLeader (SLING-12639), so it is deactivated and activated
> again on every topology change.
> * DistributionSubscriber.deactivate() does not interrupt the poller thread
> (SLING-9340) and does not wait for it, so an import that is already running
> continues.
> * The new subscriber instance starts at the last committed offset + 1, which
> is the offset still being imported by the old thread. The same package is
> then imported twice in parallel.
> * Both imports write the BookKeeper state for the same subscriber, so one of
> them can fail with an OakState0001 conflict on the subscriber store.
> The effect is significant when imports are slow (for example, an import that
> takes several minutes because of an expensive package builder) and the
> topology changes frequently (for example, instances joining and leaving the
> cluster). Several overlapping imports of the same offset were observed, each
> running for minutes.
> Proposed fix:
> * OnlyOnLeader: don't unregister on TOPOLOGY_CHANGING; only update the
> registration on TOPOLOGY_INIT/TOPOLOGY_CHANGED, and unregister only when the
> local instance is no longer the leader.
> * DistributionSubscriber.deactivate(): wake up a pending retry back-off and
> wait, bounded (30 s), for an import that is still running before returning.
> No locking between subscriber instances, to avoid hanging processing; a
> duplicate import remains possible when an import takes longer than the wait
> or when leadership really changes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
