[jira] [Updated] (KUDU-2149) New failure detector implementation can lead to election stacking
[ https://issues.apache.org/jira/browse/KUDU-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adar Dembo updated KUDU-2149: - Resolution: Fixed Fix Version/s: 1.6.0 Status: Resolved (was: In Review) Fixed in commit edd41cb40fbad206e2c356983baba8fbc57199b5. > New failure detector implementation can lead to election stacking > - > > Key: KUDU-2149 > URL: https://issues.apache.org/jira/browse/KUDU-2149 > Project: Kudu > Issue Type: Bug > Components: consensus >Affects Versions: 1.5.0 >Reporter: Adar Dembo >Assignee: Adar Dembo >Priority: Critical > Fix For: 1.6.0 > > > A new failure detector (FD) implementation was merged in commit 21b0f3d and > is part of Kudu 1.5. One of the key changes is that the detection logic runs > on a reactor thread rather than on a dedicated per-replica thread. But, > because reactor threads are shared, the election started in the event of a > failure must be thunked to the Raft thread pool (starting an election means > casting a vote, which generally means performing IO, which is verboten on a > reactor thread). > By thunking, the FD immediately rearms; the previous implementation did not > do this. If there's a lot of outstanding IO (i.e. during an election storm > across thousands of tablets), it's possible for the FD to fire again while > the first election task is still waiting to cast its vote. The new election > task will try to acquire the consensus lock and block on it (it's held by the > first election task). And so on. When the original IO finally completes, all > of the follow-on elections will get unblocked at the same time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KUDU-2149) New failure detector implementation can lead to election stacking
[ https://issues.apache.org/jira/browse/KUDU-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adar Dembo updated KUDU-2149: - Code Review: https://gerrit.cloudera.org/#/c/8107 > New failure detector implementation can lead to election stacking > - > > Key: KUDU-2149 > URL: https://issues.apache.org/jira/browse/KUDU-2149 > Project: Kudu > Issue Type: Bug > Components: consensus >Affects Versions: 1.5.0 >Reporter: Adar Dembo >Assignee: Adar Dembo >Priority: Critical > > A new failure detector (FD) implementation was merged in commit 21b0f3d and > is part of Kudu 1.5. One of the key changes is that the detection logic runs > on a reactor thread rather than on a dedicated per-replica thread. But, > because reactor threads are shared, the election started in the event of a > failure must be thunked to the Raft thread pool (starting an election means > casting a vote, which generally means performing IO, which is verboten on a > reactor thread). > By thunking, the FD immediately rearms; the previous implementation did not > do this. If there's a lot of outstanding IO (i.e. during an election storm > across thousands of tablets), it's possible for the FD to fire again while > the first election task is still waiting to cast its vote. The new election > task will try to acquire the consensus lock and block on it (it's held by the > first election task). And so on. When the original IO finally completes, all > of the follow-on elections will get unblocked at the same time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KUDU-2149) New failure detector implementation can lead to election stacking
[ https://issues.apache.org/jira/browse/KUDU-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adar Dembo updated KUDU-2149: - Status: In Review (was: In Progress) > New failure detector implementation can lead to election stacking > - > > Key: KUDU-2149 > URL: https://issues.apache.org/jira/browse/KUDU-2149 > Project: Kudu > Issue Type: Bug > Components: consensus >Affects Versions: 1.5.0 >Reporter: Adar Dembo >Assignee: Adar Dembo >Priority: Critical > > A new failure detector (FD) implementation was merged in commit 21b0f3d and > is part of Kudu 1.5. One of the key changes is that the detection logic runs > on a reactor thread rather than on a dedicated per-replica thread. But, > because reactor threads are shared, the election started in the event of a > failure must be thunked to the Raft thread pool (starting an election means > casting a vote, which generally means performing IO, which is verboten on a > reactor thread). > By thunking, the FD immediately rearms; the previous implementation did not > do this. If there's a lot of outstanding IO (i.e. during an election storm > across thousands of tablets), it's possible for the FD to fire again while > the first election task is still waiting to cast its vote. The new election > task will try to acquire the consensus lock and block on it (it's held by the > first election task). And so on. When the original IO finally completes, all > of the follow-on elections will get unblocked at the same time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
