This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
The following commit(s) were added to refs/heads/master by this push:
new 44ac52f07 Fix supervisor crash on transient ZK connection loss (#9153)
44ac52f07 is described below
commit 44ac52f07ae9d78e412f911351d133487c9b5665
Author: Jonas <[email protected]>
AuthorDate: Wed Nov 4 08:13:47 2026 +0100
Fix supervisor crash on transient ZK connection loss (#9153)
* Fix supervisor crash on transient ZK connection loss
Root cause: A transient ZooKeeper connection loss during the supervisor
heartbeat cycle caused the process to die in 629ms via
DefaultUncaughtExceptionHandler. The Curator RetryLoop blindly slept
between retries, racing against the ZK client's SendThread reconnection
instead of yielding to it.
Fix (two layers):
1. ConnectionAwareRetryPolicy (new): A RetryPolicy wrapper that tracks
the ZK ConnectionState via a ConnectionStateListener. When the
connection is SUSPENDED or LOST, it calls
CuratorFramework.blockUntilConnected(sessionTimeout) to yield to the
SendThread's failover, instead of blind sleep+retry. On reconnection,
the operation retries immediately on the new connection. Uses
storm.zookeeper.session.timeout as the wait bound (no magic numbers).
2. SupervisorHeartbeat safety net: Wrap the heartbeat body in
try/catch so that a total ZK outage (beyond session timeout) skips
the cycle instead of killing the process. A missed heartbeat is
harmless: nimbus.supervisor.timeout.secs (30s) allows 6 missed beats.
Wired up in CuratorUtils.newCurator() via AtomicReference late binding
(the CuratorFramework doesn't exist until builder.build() returns).
* Address PR review: test fix, unit tests, retry bound, jitter, scoped catch
- Fix CuratorUtilsTest ClassCastException (unwrap
ConnectionAwareRetryPolicy)
- Add @VisibleForTesting getDelegate() accessor
- Add ConnectionAwareRetryPolicyTest (8 cases) and SupervisorHeartbeatTest
(2 cases)
- Add delegate budget check before blocking on SUSPENDED/LOST
- Add RECONNECT_JITTER_MS constant (100ms) to avoid thundering herd
- Scope SupervisorHeartbeat catch to ZK write only (not
getNumaMap/buildSupervisorInfo)
- Fix heartbeat comment (liveness via ephemeral ZK nodes, not heartbeat
timestamps)
* Scope ReadClusterState ZK catch to profiler lookup
Per upstream review, don't drop the throw for the whole run() - that would
swallow persistent local failures (mkSlot, port conflicts). The only ZK
read in
run() is the profiler request lookup; assignments come from the local
backend.
Scope the catch to getProfileActions so a transient ZK error skips the cycle
(requests persist in ZK, retried in 10s) while persistent failures still
fail
fast. Adds ReadClusterStateTest.
* Add a failing test for ConnectionAwareRetryPolicy on the ZooKeeper
EventThread
Curator also consults the retry policy from the ZooKeeper EventThread when
it retries a background operation
(CuratorFrameworkImpl.checkBackgroundRetry).
Blocking that thread in blockUntilConnected keeps Curator from processing
SyncConnected, so after the session timeout it injects a session expiration
on a connection that had already recovered.
The test stops a real ZooKeeper server, restarts it after 500 ms and calls
the policy once it has seen SUSPENDED. Off the EventThread the retry is
allowed after the reconnection. On the EventThread the call blocks for the
whole session timeout, returns false and the client goes to LOST, so that
case fails with the current implementation.
* Scope PR to supervisor: remove ConnectionAwareRetryPolicy, narrow
heartbeat catch
Per maintainer feedback (GGraziadei, rzo1):
- Remove ConnectionAwareRetryPolicy and CuratorUtils changes (move to
follow-up PR)
- The policy blocked the ZK EventThread via blockUntilConnected, causing
session
expiry on background operations (Nimbus LeaderLatch, etc.)
- Narrow SupervisorHeartbeat catch to KeeperException causes only; rethrow
non-ZK failures so persistent misconfigurations still surface
- Update tests to verify ZK failures are swallowed and non-ZK failures
propagate
The two scoped catches (SupervisorHeartbeat + ReadClusterState) are
sufficient
to fix the supervisor crash on transient ZK connection loss.
* Restore CuratorUtils and its test to master's formatting
The retry policy was moved out of this PR, so these two files carry no
change. They had picked up the formatting from before the checkstyle
reformat in #9184.
* Only swallow transient ZooKeeper failures in the supervisor heartbeat
The heartbeat now only continues on connection loss, session expiry and
operation timeout. Persistent ZooKeeper errors such as NoAuth or
InvalidACL are rethrown, so a supervisor with wrong ZooKeeper credentials
or ACLs still fails at launch instead of running without registering.
Applies the review suggestions from GGraziadei and reiabreu.
---------
Co-authored-by: Jonas Krauss <[email protected]>
Co-authored-by: Gianluca Graziadei <[email protected]>
Co-authored-by: Richard Zowalla <[email protected]>
---
.../storm/daemon/supervisor/ReadClusterState.java | 11 +-
.../supervisor/timer/SupervisorHeartbeat.java | 20 +++-
.../daemon/supervisor/ReadClusterStateTest.java | 86 ++++++++++++++
.../supervisor/timer/SupervisorHeartbeatTest.java | 130 +++++++++++++++++++++
4 files changed, 245 insertions(+), 2 deletions(-)
diff --git
a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/ReadClusterState.java
b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/ReadClusterState.java
index 0186230d0..cf308e893 100644
---
a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/ReadClusterState.java
+++
b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/ReadClusterState.java
@@ -14,6 +14,7 @@ package org.apache.storm.daemon.supervisor;
import java.util.ArrayList;
import java.util.Collection;
+import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
@@ -140,7 +141,15 @@ public class ReadClusterState implements Runnable,
AutoCloseable {
//Something odd happened try again later
return;
}
- Map<String, List<ProfileRequest>> topoIdToProfilerActions =
getProfileActions(stormClusterState, stormIds);
+ Map<String, List<ProfileRequest>> topoIdToProfilerActions;
+ try {
+ topoIdToProfilerActions = getProfileActions(stormClusterState,
stormIds);
+ } catch (Exception e) {
+ // Assignments come from the local backend; profiler requests
are the only ZooKeeper read here.
+ // They stay in ZooKeeper, so skip them for this cycle instead
of halting the supervisor.
+ LOG.warn("Failed to read profiler requests, will retry next
cycle", e);
+ topoIdToProfilerActions = Collections.emptyMap();
+ }
HashSet<Integer> assignedPorts = new HashSet<>();
LOG.debug("Synchronizing supervisor");
diff --git
a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeat.java
b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeat.java
index b613f36ee..488397d06 100644
---
a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeat.java
+++
b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeat.java
@@ -29,8 +29,10 @@ import org.apache.storm.generated.SupervisorInfo;
import
org.apache.storm.scheduler.resource.normalization.NormalizedResourceOffer;
import org.apache.storm.scheduler.resource.normalization.NormalizedResources;
import org.apache.storm.shade.org.apache.commons.collections4.ListUtils;
+import org.apache.storm.shade.org.apache.zookeeper.KeeperException;
import org.apache.storm.utils.ObjectReader;
import org.apache.storm.utils.Time;
+import org.apache.storm.utils.Utils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -162,7 +164,23 @@ public class SupervisorHeartbeat implements Runnable {
Map<String, Object> validatedNumaMap =
SupervisorUtils.getNumaMap(conf);
Map<String, SupervisorInfo> supervisorInfoList =
buildSupervisorInfo(conf, supervisor, validatedNumaMap);
for (Map.Entry<String, SupervisorInfo> supervisorInfoEntry :
supervisorInfoList.entrySet()) {
-
stormClusterState.supervisorHeartbeat(supervisorInfoEntry.getKey(),
supervisorInfoEntry.getValue());
+ try {
+
stormClusterState.supervisorHeartbeat(supervisorInfoEntry.getKey(),
supervisorInfoEntry.getValue());
+ } catch (Exception e) {
+ if (!isTransientZkFailure(e)) {
+ throw e;
+ }
+ // A transient ZK failure: the ephemeral node is recreated on
the next heartbeat.
+ // Local and persistent ZK errors (e.g. bad ACLs) are rethrown
so they surface.
+ LOG.warn("Supervisor ZK heartbeat failed for {}, will retry
next cycle",
+ supervisorInfoEntry.getKey(), e);
+ }
}
}
+
+ private static boolean isTransientZkFailure(Throwable t) {
+ return
Utils.exceptionCauseIsInstanceOf(KeeperException.ConnectionLossException.class,
t)
+ ||
Utils.exceptionCauseIsInstanceOf(KeeperException.SessionExpiredException.class,
t)
+ ||
Utils.exceptionCauseIsInstanceOf(KeeperException.OperationTimeoutException.class,
t);
+ }
}
diff --git
a/storm-server/src/test/java/org/apache/storm/daemon/supervisor/ReadClusterStateTest.java
b/storm-server/src/test/java/org/apache/storm/daemon/supervisor/ReadClusterStateTest.java
new file mode 100644
index 000000000..950818258
--- /dev/null
+++
b/storm-server/src/test/java/org/apache/storm/daemon/supervisor/ReadClusterStateTest.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership. The
ASF licenses this file to you under the Apache License, Version
+ * 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+package org.apache.storm.daemon.supervisor;
+
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.storm.Config;
+import org.apache.storm.DaemonConfig;
+import org.apache.storm.cluster.IStormClusterState;
+import org.apache.storm.scheduler.ISupervisor;
+import org.apache.storm.shade.org.apache.zookeeper.KeeperException;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class ReadClusterStateTest {
+
+ private static final String TOPOLOGY_ID = "topo-1";
+
+ @TempDir
+ Path localDir;
+
+ private final IStormClusterState clusterState =
mock(IStormClusterState.class);
+ private final ISupervisor iSupervisor = mock(ISupervisor.class);
+
+ private ReadClusterState newReadClusterState() throws Exception {
+ Map<String, Object> conf = new HashMap<>();
+ conf.put(Config.STORM_CLUSTER_MODE, "local");
+ conf.put(Config.STORM_LOCAL_DIR, localDir.toString());
+ conf.put(DaemonConfig.SUPERVISOR_SLOTS_PORTS, Collections.emptyList());
+
+ Supervisor supervisor = mock(Supervisor.class);
+ when(supervisor.getConf()).thenReturn(conf);
+ when(supervisor.getStormClusterState()).thenReturn(clusterState);
+ when(supervisor.getAssignmentId()).thenReturn("supervisor-1");
+ when(supervisor.getiSupervisor()).thenReturn(iSupervisor);
+ when(supervisor.getHostName()).thenReturn("host-1");
+ when(supervisor.getCurrAssignment()).thenReturn(new
AtomicReference<>(new HashMap<>()));
+ return new ReadClusterState(supervisor);
+ }
+
+ @Test
+ public void profilerRequestFailureDoesNotHaltAssignmentSync() throws
Exception {
+
when(clusterState.assignments(null)).thenReturn(Collections.singletonList(TOPOLOGY_ID));
+
when(clusterState.assignmentsInfo()).thenReturn(Collections.emptyMap());
+ // What ClientZookeeper surfaces after Curator gives up, as in the
reported supervisor crash.
+
when(clusterState.getTopologyProfileRequests(TOPOLOGY_ID)).thenThrow(new
RuntimeException(
+ new KeeperException.SessionExpiredException()));
+
+ ReadClusterState readClusterState = newReadClusterState();
+
+ assertDoesNotThrow(readClusterState::run);
+ verify(iSupervisor).assigned(any());
+ }
+
+ @Test
+ public void assignmentReadFailureStillPropagates() throws Exception {
+ when(clusterState.assignments(null)).thenThrow(new
IllegalStateException("broken local backend"));
+
+ ReadClusterState readClusterState = newReadClusterState();
+
+ assertThrows(RuntimeException.class, readClusterState::run);
+ verify(clusterState, never()).getTopologyProfileRequests(anyString());
+ }
+}
diff --git
a/storm-server/src/test/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeatTest.java
b/storm-server/src/test/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeatTest.java
new file mode 100644
index 000000000..5fbe35668
--- /dev/null
+++
b/storm-server/src/test/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeatTest.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership. The
ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.supervisor.timer;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.storm.cluster.IStormClusterState;
+import org.apache.storm.daemon.supervisor.Supervisor;
+import org.apache.storm.generated.SupervisorInfo;
+import org.apache.storm.scheduler.ISupervisor;
+import org.apache.storm.shade.org.apache.zookeeper.KeeperException;
+import org.apache.storm.utils.Utils;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class SupervisorHeartbeatTest {
+
+ private static final String SUPERVISOR_ID = "supervisor-1";
+
+ private Supervisor mockSupervisor(IStormClusterState clusterState) {
+ Supervisor supervisor = mock(Supervisor.class);
+ ISupervisor iSupervisor = mock(ISupervisor.class);
+ when(iSupervisor.getMetadata()).thenReturn(Arrays.asList(6700, 6701));
+
+ when(supervisor.getStormClusterState()).thenReturn(clusterState);
+ when(supervisor.getId()).thenReturn(SUPERVISOR_ID);
+ when(supervisor.getiSupervisor()).thenReturn(iSupervisor);
+ when(supervisor.getCurrAssignment()).thenReturn(new
AtomicReference<>(new HashMap<>()));
+ when(supervisor.getHostName()).thenReturn("host-1");
+ when(supervisor.getAssignmentId()).thenReturn(SUPERVISOR_ID);
+ when(supervisor.getThriftServerPort()).thenReturn(6627);
+ when(supervisor.getUpTime()).thenReturn(Utils.makeUptimeComputer());
+ when(supervisor.getStormVersion()).thenReturn("test-version");
+ return supervisor;
+ }
+
+ private Map<String, Object> baseConf() {
+ return new HashMap<>(Utils.readStormConfig());
+ }
+
+ @Test
+ public void
transientZkHeartbeatFailureIsSwallowedInsteadOfKillingTheProcess() {
+ IStormClusterState clusterState = mock(IStormClusterState.class);
+ doThrow(new RuntimeException(
+ new KeeperException.ConnectionLossException()))
+ .when(clusterState).supervisorHeartbeat(anyString(),
any(SupervisorInfo.class));
+
+ Supervisor supervisor = mockSupervisor(clusterState);
+ SupervisorHeartbeat heartbeat = new SupervisorHeartbeat(baseConf(),
supervisor);
+
+ assertDoesNotThrow(heartbeat::run);
+ verify(clusterState, atLeastOnce()).supervisorHeartbeat(anyString(),
any(SupervisorInfo.class));
+ }
+
+ @Test
+ public void otherTransientZkFailuresAreSwallowed() {
+ for (KeeperException failure : Arrays.asList(new
KeeperException.SessionExpiredException(),
+ new KeeperException.OperationTimeoutException())) {
+ IStormClusterState clusterState = mock(IStormClusterState.class);
+ doThrow(new RuntimeException(failure))
+ .when(clusterState).supervisorHeartbeat(anyString(),
any(SupervisorInfo.class));
+
+ SupervisorHeartbeat heartbeat = new
SupervisorHeartbeat(baseConf(), mockSupervisor(clusterState));
+
+ assertDoesNotThrow(heartbeat::run);
+ }
+ }
+
+ @Test
+ public void persistentZkFailureStillPropagates() {
+ for (KeeperException failure : Arrays.asList(new
KeeperException.NoAuthException(),
+ new KeeperException.InvalidACLException())) {
+ IStormClusterState clusterState = mock(IStormClusterState.class);
+ doThrow(new RuntimeException(failure))
+ .when(clusterState).supervisorHeartbeat(anyString(),
any(SupervisorInfo.class));
+
+ SupervisorHeartbeat heartbeat = new
SupervisorHeartbeat(baseConf(), mockSupervisor(clusterState));
+
+ assertThrows(RuntimeException.class, heartbeat::run);
+ }
+ }
+
+ @Test
+ public void nonZkHeartbeatFailureStillPropagates() {
+ IStormClusterState clusterState = mock(IStormClusterState.class);
+ doThrow(new IllegalStateException("broken local config"))
+ .when(clusterState).supervisorHeartbeat(anyString(),
any(SupervisorInfo.class));
+
+ Supervisor supervisor = mockSupervisor(clusterState);
+ SupervisorHeartbeat heartbeat = new SupervisorHeartbeat(baseConf(),
supervisor);
+
+ assertThrows(IllegalStateException.class, heartbeat::run);
+ }
+
+ @Test
+ public void successfulCycleSendsTheHeartbeat() {
+ IStormClusterState clusterState = mock(IStormClusterState.class);
+ Supervisor supervisor = mockSupervisor(clusterState);
+ SupervisorHeartbeat heartbeat = new SupervisorHeartbeat(baseConf(),
supervisor);
+
+ assertDoesNotThrow(heartbeat::run);
+ verify(clusterState).supervisorHeartbeat(eq(SUPERVISOR_ID),
any(SupervisorInfo.class));
+ }
+}