[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15663155#comment-15663155 ] Stefan Egli commented on SLING-5435: [~ianeboston], there has been no changes to the API as of yet (ie no new property introduced for this). Are you saying this is no longer a requirement? (I'm not aware this is needed by anyone atm) > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126162#comment-15126162 ] Carsten Ziegeler commented on SLING-5435: - [~egli] ok yes, maybe marking the listener with a property is better than introducing a new event type. For the API, for now we could simply state that the API only handles one case (sync'ed). Adding a new method looks too confusing to me , given the little benefit. I don't really expect client code to be based on the API - it's more an informative API > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[
https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126058#comment-15126058
]
Stefan Egli commented on SLING-5435:
[~cziegeler], re
bq. It might happen that the old leader did a change, which the new leader
doesn't see yet and therefore the new leader might try a similar operation
That was the original thinking behind doing the sync, yes.
A certain class of TopologyEventListeners (such as for example job handling)
can be written in a way that is resilient to repository delays. Namely by
forcing a 'repository sync' for a particular node by making a change that
forces a conflict even if old leader changes are delayed.
However, I think that this is not possible for all cases - ignoring the fact
that when writing a TopologyEventListener you might not be aware of these
subleties.
Additionally though, there might be 'derivative cases' - such as for example
the Sling Scheduler - which only check the 'leader' flag to then behave
leader-like or not. What that 'leader-like' implies is unclear and can perhaps
not always be 'guarded' via a repository-synched flag (which tries to result in
a conflict in these delay-cases).
I think it helps the default implementor of TopologyEventListener writing
leader-failover-stable code when it receives the TOPOLOGY_CHANGED only after a
repo-sync. Leaving this to each and every listener implementor might result in
duplication of code as well as less stable code..
[~marett], re
bq. For those use cases, with the given discovery API and implementations, it
is already possible to avoid the delay by polling the DiscoveryService instead
of handling the TopologyEventListener events. However, I believe it is a rather
discouraged thing.
I disagree. You cannot poll DiscoveryService to get {{isCurrent}} true *before*
the TopologyEventListeners get informed. In fact, these two things are coupled:
{{TopologyView.isCurrent()}} and the {{TOPOLOGY_CHANGED}} event are
synchronized. That is, {{isCurrent()}} only returns true once discovery starts
sending out {{TOPOLOGY_CHANGED}} events.
bq. LEADER_CHANGED
Some comments that come to mind:
* Alternative suggestion for the name: {{TOPOLOGY_CHANGED_UNSYNCHED}}
* If we introduce this event, it would have to be kept backwards-compatible, ie
after this event you'd also must get a {{TOPOLOGY_CHANGED}} event.
* I still see the risk of breaking client code with this, as some might do
things like "{{if (event.getType() != TOPOLOGY_CHANGING){}}}" or similar - in
which case the new event type might result in execution where that was not
intended..
* We should think about how we could keep the API symmetric: currently there
are two fully equivalent variants: polling via
{{DiscoveryService.getTopology()}} or push via {{TopologyEventListener}}. The
new event is so far only available via push.
* If we do it the other way, by making this a property of
{{TopologyEventListener}} ("{{unsynchronized=true}}"), we make it more
backwards compatible, as only those listeners are affected that set this
property. However it would still result in an asymmetric API and I think we
should do something about it in both cases.
* Perhaps one possibility for providing this info in the poll variant could be:
{{getUnsynchronizedTopology()}}. (sounds somewhat ugly though..)
Overall I tend to think that going via a listener property introduces less
friction, but a new event type would certainly also be possible...
> Decouple processes that depend on cluster leader elections from the cluster
> leader elections.
> -
>
> Key: SLING-5435
> URL: https://issues.apache.org/jira/browse/SLING-5435
> Project: Sling
> Issue Type: Improvement
> Components: General
>Reporter: Ian Boston
>
> Currently there are many processes in Sling that must complete before a Sling
> Discovery cluster leader election is declared complete. These processes
> include things like transferring all Jobs from the old leader to the new
> leader and waiting for the data to appear visible on the new leader. This
> introduces an additional overhead to the leader election process which
> introduces a higher than desirable timeout for elections and heartbeat. This
> higher than desirable timeout precludes the use of more efficient election
> and distributed consensus algorithms as implemented in Etcd, Zookeeper or
> implementations of RAFT.
> If the election could be declared complete leaving individual components to
> manage their own post election operations (ie decoupling those processes from
> the election), then faster election or alternative Discovery implementations
> such as the one implemented on etcd could be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#63
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125927#comment-15125927 ] Timothee Maret commented on SLING-5435: --- Using a broken example was probably not making things clearer :-) I could help with a patch (API) unless someone wants to tackle this. For discovery implementations to support that new API, I would open separate issues. > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124895#comment-15124895 ] Carsten Ziegeler commented on SLING-5435: - Ok, sorry for my slow working brain, I think I understand the problem now as well :) It might happen that the old leader did a change, which the new leader doesn't see yet and therefore the new leader might try a similar operation. Which then either results in conflicts or duplicate processing etc. Adding a new event type like LEADER_CHANGED sounds like a good solution to me > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[
https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124861#comment-15124861
]
Timothee Maret commented on SLING-5435:
---
Assuming modifying the API is ok, we may also consider keeping the
{{TopologyEventListener}} as is, but add an event type such as
{code}
/**
* The topology has elected a new leader.
*/
LEADER_CHANGED
{code}
which would be sent before a {{TOPOLOGY_CHANGED}} event (but does not wait on
the repo sync).
> Decouple processes that depend on cluster leader elections from the cluster
> leader elections.
> -
>
> Key: SLING-5435
> URL: https://issues.apache.org/jira/browse/SLING-5435
> Project: Sling
> Issue Type: Improvement
> Components: General
>Reporter: Ian Boston
>
> Currently there are many processes in Sling that must complete before a Sling
> Discovery cluster leader election is declared complete. These processes
> include things like transferring all Jobs from the old leader to the new
> leader and waiting for the data to appear visible on the new leader. This
> introduces an additional overhead to the leader election process which
> introduces a higher than desirable timeout for elections and heartbeat. This
> higher than desirable timeout precludes the use of more efficient election
> and distributed consensus algorithms as implemented in Etcd, Zookeeper or
> implementations of RAFT.
> If the election could be declared complete leaving individual components to
> manage their own post election operations (ie decoupling those processes from
> the election), then faster election or alternative Discovery implementations
> such as the one implemented on etcd could be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124848#comment-15124848 ] Timothee Maret commented on SLING-5435: --- bq. Ok, it seems we agree that you're example works similar whether the leader election waits for the repo sync or not Not exactly equivalent. With the current implementation the "SomeImporter" must wait on the repo-sync before importing even though it does not need to. bq. do we have an example where the code is acting on a change in leader election and needs a repo sync? I think this category contains the TopologyEventListener in Sling, which AFAIU do need to wait on the previous leader change before proceeding. The existing topology event support fits those use case. The other category (acting on a change in leader election *and does not* need a repo sync) contains the use cases I shared as well as the example code above. The existing topology event support impose to wait on the repository sync where there is no need for it. > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124836#comment-15124836 ] Carsten Ziegeler commented on SLING-5435: - Ok, it seems we agree that you're example works similar whether the leader election waits for the repo sync or not - from the point of your code there is nothing you would need to change. Sorry to insist on it, but do we have an example where the code is acting on a change in leader election *and* needs a repo sync? > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123629#comment-15123629 ] Timothee Maret commented on SLING-5435: --- bq. The leader is already elected in 2. I guess you assume there is a delay between the leader election and sending the event, which then happens in 5. Exactly, I don't assume it, it does exists. bq. But there is always some delay between 2 and 5 +1 with the current implementation, the delay is somewhat related to the load and may be counted in hour(s). What I propose is to reduce that delay to a more constant time which may be a couple of CPU cycles. bq. So the above situation can happen in both cases True, in one case much more likely than in the other bq. Once the changed event is sent, you might want to start the import Yes, that's the proper solution to not lose the import. But even if you won't lose the import invocation, why did you wait on the repository ? > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123620#comment-15123620 ] Carsten Ziegeler commented on SLING-5435: - I'm sorry but I don't fully understand the above example. Why is "else" executed in 4? The leader is already elected in 2. I guess you assume there is a delay between the leader election and sending the event, which then happens in 5. But there is always some delay between 2 and 5, regardless whether discovery waits for repository sync or not. So the above situation can happen in both cases. Once the changed event is sent, you might want to start the import > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[
https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123600#comment-15123600
]
Timothee Maret commented on SLING-5435:
---
One example from the list of use case I shared above.
{code}
/*
* Pattern: Only one instance (the topology leader) does the actual import.
*/
class SomeImporter implements Importer, TopologyEventListener {
volatile boolean leader;
public void handleTopologyEvent(final TopologyEvent event) {
if ( event.getType() == TopologyEvent.Type.TOPOLOGY_CHANGED
|| event.getType() == TopologyEvent.Type.TOPOLOGY_INIT) {
this.leader = event.getNewView().getLocalInstance().isLeader();
}
}
/**
* Invoked by the framework.
*/
publiv void import() {
if (leader) {
// do actually import data from a data source into the
repository.
} else {
// do nothing as you are not the leader.
}
}
}
{code}
The import method does not need to wait on the repository replication before
running.
The import method should not need to wait on the repository replication before
running, in order to avoid the following sequence
{code}
# note: import method is invoked every day by the framework
1. The topology leader leaves the topology (as a result of a crash for instance)
2. The topology elects a new leader
3. The framework invokes the import method
4. The import method runs the "else" branch on every machine even though a
leader exists
5. The discovery implementation decides to send the TopologyEventListener
events (it things the changes from the previous leader are visible on every
other instances)
# result: the import method did not execute the if(true) branch, even though it
had everything in place to do so.
{code}
> Decouple processes that depend on cluster leader elections from the cluster
> leader elections.
> -
>
> Key: SLING-5435
> URL: https://issues.apache.org/jira/browse/SLING-5435
> Project: Sling
> Issue Type: Improvement
> Components: General
>Reporter: Ian Boston
>
> Currently there are many processes in Sling that must complete before a Sling
> Discovery cluster leader election is declared complete. These processes
> include things like transferring all Jobs from the old leader to the new
> leader and waiting for the data to appear visible on the new leader. This
> introduces an additional overhead to the leader election process which
> introduces a higher than desirable timeout for elections and heartbeat. This
> higher than desirable timeout precludes the use of more efficient election
> and distributed consensus algorithms as implemented in Etcd, Zookeeper or
> implementations of RAFT.
> If the election could be declared complete leaving individual components to
> manage their own post election operations (ie decoupling those processes from
> the election), then faster election or alternative Discovery implementations
> such as the one implemented on etcd could be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123553#comment-15123553 ] Carsten Ziegeler commented on SLING-5435: - Ok, I thanks, I think I understand now what this is about. However, just to refresh my memory, can someone please come up with an example of existing code where this problem could occur? > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[
https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123511#comment-15123511
]
Ian Boston commented on SLING-5435:
---
[~marett] Thank you for the clarification, no need to open a second issue
SLING-5423 covers the raft impl.
imho, now that the "processes" that are waiting on the repository have been
identified, or can be identified, this issue should remain open and assuming
Sling wants to have faster, more reliable elections we should do something
about the {{TopologyEventListener}} area.
[~cziegeler] wdyt ?
> Decouple processes that depend on cluster leader elections from the cluster
> leader elections.
> -
>
> Key: SLING-5435
> URL: https://issues.apache.org/jira/browse/SLING-5435
> Project: Sling
> Issue Type: Improvement
> Components: General
>Reporter: Ian Boston
>
> Currently there are many processes in Sling that must complete before a Sling
> Discovery cluster leader election is declared complete. These processes
> include things like transferring all Jobs from the old leader to the new
> leader and waiting for the data to appear visible on the new leader. This
> introduces an additional overhead to the leader election process which
> introduces a higher than desirable timeout for elections and heartbeat. This
> higher than desirable timeout precludes the use of more efficient election
> and distributed consensus algorithms as implemented in Etcd, Zookeeper or
> implementations of RAFT.
> If the election could be declared complete leaving individual components to
> manage their own post election operations (ie decoupling those processes from
> the election), then faster election or alternative Discovery implementations
> such as the one implemented on etcd could be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[
https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123504#comment-15123504
]
Timothee Maret commented on SLING-5435:
---
bq. If those "processes" don't exist, then it sounds like there is nothing to
stop a faster leader election implementation that is not slowed down by the
latency required to ensure a repository reaches a consistent state.
As I wrote in my previous comment, I think that all consumers of the
{{TopologyEventListener}} currently in Sling do make a legit case of waiting on
the repository. However, my point is that not all {{TopologyEventListener}}
need to wait on the repository. I have shared a list of use cases as well.
Thus, instead of imposing the repository wait, we could allow to configure a
listener so that it does not have to wait on some repository sync. The
implementation proposal has been discussed above as well.
I propose to keep the focus of this issue on that goal. Alternatively, I would
open a separate one.
bq. If that's really the case, then this issue can be closed and replaced by a
new issue titled something like "Implement leader election using RAFT over the
network" in the same way that systems like etcd perform elections.
I have already opened SLING-5423 to track that and I am working on it.
> Decouple processes that depend on cluster leader elections from the cluster
> leader elections.
> -
>
> Key: SLING-5435
> URL: https://issues.apache.org/jira/browse/SLING-5435
> Project: Sling
> Issue Type: Improvement
> Components: General
>Reporter: Ian Boston
>
> Currently there are many processes in Sling that must complete before a Sling
> Discovery cluster leader election is declared complete. These processes
> include things like transferring all Jobs from the old leader to the new
> leader and waiting for the data to appear visible on the new leader. This
> introduces an additional overhead to the leader election process which
> introduces a higher than desirable timeout for elections and heartbeat. This
> higher than desirable timeout precludes the use of more efficient election
> and distributed consensus algorithms as implemented in Etcd, Zookeeper or
> implementations of RAFT.
> If the election could be declared complete leaving individual components to
> manage their own post election operations (ie decoupling those processes from
> the election), then faster election or alternative Discovery implementations
> such as the one implemented on etcd could be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123485#comment-15123485 ] Ian Boston commented on SLING-5435: --- I wrote the initial description, so need to clarify. The aim of the issue is to make it possible to reduce heartbeat timeouts and improve the speed and reliability of a leader election, from the current level to a level seen in other applications that perform leader elections (eg MongoDB, etcd, etc). Most of these applications are using some variant of RAFT, but any widely tested and proven algorithm would stand the best chance of achieving this. Most, if not all, perform the leader election via network communication between the candidates. None, afaik rely on a centralised store to perform leader election, as that would itself need to be elected (as MongoDB does) IIUC (or rather I was told) that the reason a leader election could not use an algorithm like RAFT over network for leader election as opposed to the current via repository discovery mechanism was because it was too fast for some "processes". Those "processes" needed the repository to arrive at a consistent state before they could accept a new leader. If those "processes" don't exist, then it sounds like there is nothing to stop a faster leader election implementation that is not slowed down by the latency required to ensure a repository reaches a consistent state. If that's really the case, then this issue can be closed and replaced by a new issue titled something like "Implement leader election using RAFT over the network" in the same way that systems like etcd perform elections. I have put the word processes in " as I don't know if they are single threads, or conceptual tasks that represent a sequence of operations. > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123444#comment-15123444 ] Carsten Ziegeler commented on SLING-5435: - The initial description of this issue says: "Currently there are many processes in Sling that must complete before a Sling Discovery cluster leader election is declared complete. These processes include things like transferring all Jobs from the old leader to the new leader and waiting for the data to appear visible on the new leader" So far I fail to see these - looking at you're reply it seems we're talking about something different now > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[
https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123436#comment-15123436
]
Timothee Maret commented on SLING-5435:
---
I don't think there is such process in Sling. Indeed, the Sling components
handling topology events are located in the {{org.apache.sling.event.impl}}
package and AFAIU those components deal with data stored in the repository and
attempt to process data with a guaranteed single execution. For those use
cases, the repository sync is likely helping toward that goal.
However, topology handlers in application code, for instance in Adobe AEM or
services based on Adobe AEM, may not need that repository sync.
I can't share pointers here because it is proprietary code, but I can give you
some potential use cases such as data importers (from an external source to the
repository), data validation tasks and notification services (e.g. sends email)
which typically only one leader instance executes.
> Decouple processes that depend on cluster leader elections from the cluster
> leader elections.
> -
>
> Key: SLING-5435
> URL: https://issues.apache.org/jira/browse/SLING-5435
> Project: Sling
> Issue Type: Improvement
> Components: General
>Reporter: Ian Boston
>
> Currently there are many processes in Sling that must complete before a Sling
> Discovery cluster leader election is declared complete. These processes
> include things like transferring all Jobs from the old leader to the new
> leader and waiting for the data to appear visible on the new leader. This
> introduces an additional overhead to the leader election process which
> introduces a higher than desirable timeout for elections and heartbeat. This
> higher than desirable timeout precludes the use of more efficient election
> and distributed consensus algorithms as implemented in Etcd, Zookeeper or
> implementations of RAFT.
> If the election could be declared complete leaving individual components to
> manage their own post election operations (ie decoupling those processes from
> the election), then faster election or alternative Discovery implementations
> such as the one implemented on etcd could be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121714#comment-15121714 ] Carsten Ziegeler commented on SLING-5435: - Can we list all these processes first before we talk about a solution? > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121624#comment-15121624 ] Stefan Egli commented on SLING-5435: I would prefer to put that into the API - esp if we can keep the resulting/required version bump on a dot dot level (1.0.4) > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[
https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121611#comment-15121611
]
Timothee Maret commented on SLING-5435:
---
One way to achieve this in a backward compatible way would be to allow
specifying the different notification modes (repository-sync,
no-repository-sync) as an optional OSGI property set on a services implementing
the {{TopologyEventListener}} API (default or no config means repository-sync).
IMO the specification could either be in the API (a small backward compatible
change of the API would be required) or in the common implementation
{{org.apache.sling.discovery.commons.providers.base.AsyncEventSender}}. Since
there's a common bundle that can be reused by all discovery implementation, I
think that *may* be acceptable.
> Decouple processes that depend on cluster leader elections from the cluster
> leader elections.
> -
>
> Key: SLING-5435
> URL: https://issues.apache.org/jira/browse/SLING-5435
> Project: Sling
> Issue Type: Improvement
> Components: General
>Reporter: Ian Boston
>
> Currently there are many processes in Sling that must complete before a Sling
> Discovery cluster leader election is declared complete. These processes
> include things like transferring all Jobs from the old leader to the new
> leader and waiting for the data to appear visible on the new leader. This
> introduces an additional overhead to the leader election process which
> introduces a higher than desirable timeout for elections and heartbeat. This
> higher than desirable timeout precludes the use of more efficient election
> and distributed consensus algorithms as implemented in Etcd, Zookeeper or
> implementations of RAFT.
> If the election could be declared complete leaving individual components to
> manage their own post election operations (ie decoupling those processes from
> the election), then faster election or alternative Discovery implementations
> such as the one implemented on etcd could be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121372#comment-15121372 ] Stefan Egli commented on SLING-5435: I'm wondering how this could be achieved though - without changing the discovery API > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121122#comment-15121122 ] Timothee Maret commented on SLING-5435: --- bq. Perhaps what might be useful is to group the TopologyEventListeners into those that want synchronization and those that explicitly don't want it? [~egli] +1 It would avoid API consumers to explicitly invoke a synchronisation mechanism should they require it. It could be done without breaking the existing API consumers which rely on repository synchronisation and let components that don't require this sync to smoothly migrate to the alternative support (without sync). > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[
https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15115218#comment-15115218
]
Stefan Egli commented on SLING-5435:
[~ianeboston], parts of this concern has already been worked upon. During
discussions around discovery.oak and discovery.etcd it became clear that both
of these had the problem of propagating the 'leader change' information faster
than changes propagate in the repository. Which opened up the problem of
getting notified about a leader change before the last changes of a perhaps
crashed/shutdown instance have been seen by all other remaining instances (this
is just one example, another is threading with the {{TopologyEvent}} itself).
This lead to the conclusion that such 'fast leader detection mechanisms'
require additional synchronization with the repository.
For the new discovery.oak this has been implemented as a separate (spi)
interface called {{ClusterSyncService}} which can be enabled/disabled via
configuration. So you can already run discovery.oak with a fast leader detector
without synchronization - except that the application then has to deal with the
missing synchronization one way or another.
Sounds like what might be missing is some kind of generic support for the case
where this synchronization is disabled from the discovery mechanism. Perhaps
what might be useful is to group the {{TopologyEventListeners}} into those that
want synchronization and those that explicitly don't want it?
> Decouple processes that depend on cluster leader elections from the cluster
> leader elections.
> -
>
> Key: SLING-5435
> URL: https://issues.apache.org/jira/browse/SLING-5435
> Project: Sling
> Issue Type: Improvement
> Components: General
>Reporter: Ian Boston
>
> Currently there are many processes in Sling that must complete before a Sling
> Discovery cluster leader election is declared complete. These processes
> include things like transferring all Jobs from the old leader to the new
> leader and waiting for the data to appear visible on the new leader. This
> introduces an additional overhead to the leader election process which
> introduces a higher than desirable timeout for elections and heartbeat. This
> higher than desirable timeout precludes the use of more efficient election
> and distributed consensus algorithms as implemented in Etcd, Zookeeper or
> implementations of RAFT.
> If the election could be declared complete leaving individual components to
> manage their own post election operations (ie decoupling those processes from
> the election), then faster election or alternative Discovery implementations
> such as the one implemented on etcd could be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
