[ https://issues.apache.org/jira/browse/AURORA-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115938#comment-16115938 ]
Stephan Erb edited comment on AURORA-1944 at 8/6/17 9:11 PM: ------------------------------------------------------------- On a first glance, there appear to be two problems and we should probably fix both of them: * When ZK is down at startup, {{CuratorServiceGroupMonitor.start}} fails with {{Failed to begin monitoring service group.}}. The start is not re-tried in any form but instead a shutdown is triggered. * The scheduler shutdown fails with {{State transition from CONSTRUCTED to STOPPED is not allowed.}} With this patch applied, unit tests, e2e tests, and scheduler start with ZK down seem all to work on first sight in the vagrant environment: {code} diff --git a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java index eba56be..4551b44 100644 --- a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java +++ b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java @@ -69,10 +69,7 @@ class CuratorServiceGroupMonitor implements ServiceGroupMonitor { @Override public void start() throws MonitorException { try { - // NB: This blocks on an initial group population to emulate legacy ServerSetMonitor behavior; - // asynchronous population is an option using NORMAL or POST_INITIALIZED_EVENT StartModes - // though. - groupCache.start(PathChildrenCache.StartMode.BUILD_INITIAL_CACHE); + groupCache.start(PathChildrenCache.StartMode.NORMAL); } catch (Exception e) { throw new MonitorException("Failed to begin monitoring service group.", e); } {code} [~jsirois] what do you think? was (Author: stephanerb): On a first glance, there appear to be two problems and we should probably fix both of them: * When ZK is down at startup, {{CuratorServiceGroupMonitor.start}} fails with {{Failed to begin monitoring service group.}}. The start is not re-tried in any form but instead a shutdown is triggered. * The scheduler shutdown fails with {{State transition from CONSTRUCTED to STOPPED is not allowed.}} With this patch applied, unit tests, e2e tests, and scheduler start with ZK down seem all to work on first sight: {code} diff --git a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java index eba56be..4551b44 100644 --- a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java +++ b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java @@ -69,10 +69,7 @@ class CuratorServiceGroupMonitor implements ServiceGroupMonitor { @Override public void start() throws MonitorException { try { - // NB: This blocks on an initial group population to emulate legacy ServerSetMonitor behavior; - // asynchronous population is an option using NORMAL or POST_INITIALIZED_EVENT StartModes - // though. - groupCache.start(PathChildrenCache.StartMode.BUILD_INITIAL_CACHE); + groupCache.start(PathChildrenCache.StartMode.NORMAL); } catch (Exception e) { throw new MonitorException("Failed to begin monitoring service group.", e); } {code} [~jsirois] what do you think? > Aurora is unable to elect leader after losing ZK for an extended period of > time > ------------------------------------------------------------------------------- > > Key: AURORA-1944 > URL: https://issues.apache.org/jira/browse/AURORA-1944 > Project: Aurora > Issue Type: Bug > Components: Scheduler > Environment: Running on 0.17.0 > Reporter: Renan DelValle > Attachments: aurora-0.log, aurora-1.log, aurora-2.log > > > Using Apache Curator as the Zookeeper library causes an issue where Aurora is > unable to elect a leader if Zookeeper loses quorum for an extended period of > time. > Scheduler seems to crash around: > {{W0802 14:01:14.436 [TaskEventBatchWorker, SchedulerLifecycle] Failed to > leave leadership: > org.apache.aurora.common.zookeeper.SingletonService$LeaveException: Failed to > abdicate leadership of group at /aurora/scheduler}} > When the init system brings the scheduler back up, it is unable to elect a > leader if ZK is still down. > Specifically, the redirect monitor fails: > {{E0802 14:09:37.063 [RedirectMonitor STARTING, > GuavaUtils$LifecycleShutdownListener] Service: RedirectMonitor [FAILED] > failed unexpectedly. Triggering shutdown.}} > Leading to every scheduler showing the following: > {{W0802 14:16:34.646 [qtp576711849-43, LeaderRedirect] No serviceGroupMonitor > in host set, will not redirect despite not being leader.}} > Once the scheduler enters this state, it is unable to snap out of it until it > is manually restarted. -- This message was sent by Atlassian JIRA (v6.4.14#64029)