Hi, The actual issue is
"Client will not use DIGEST-MD5 as SASL mechanism, because FIPS mode is enabled." For this reason, auth will fail later. This was introduced in ZooKeeper 3.9.4 introduced ZOOKEEPER-4889 - "Fallback to DIGEST-MD5 auth mech should be disabled in FIPS mode" (https://www.mail-archive.com/[email protected]/msg09243.html). The client now refuses to create a DIGEST-MD5 SASL client when FIPS mode is on. Since 2.8.8. ships with a newer ZooKeeper client. The funny part is, that zookeeper.fips-mode defaults to true in ZooKeeper 3.9.x As a workaround, you can pass the system property to the daemon JVMs in storm.yaml (it must be a -D flag; per ZOOKEEPER-4973 (https://www.mail-archive.com/[email protected]/msg09733.html) setting it via config file doesn't work for the client side): nimbus.childopts: "-Xmx1024m -Dzookeeper.fips-mode=false" supervisor.childopts: "... -Dzookeeper.fips-mode=false" # plus ui.childopts / worker.childopts if those also do SASL to ZK The property name is not relocated by the shade plugin (it doesn't match the org.apache.zookeeper package pattern), so plain zookeeper.fips-mode works against the shaded client. A real fix would be migrating Storm↔ZK auth to Kerberos (GSSAPI) rather than keeping the override flag forever. Hope it helps Richard > Am 02.06.2026 um 17:40 schrieb GMTT via user <[email protected]>: > > Hello, > > We recently upgraded our Apache Storm Nimbus server to 2.8.8 from 2.8.0 and > the storm-nimbus service just continuously restart with the error below. We > are currently using SASL authentication using the DIGEST-MD5 mechanism > between our storm and zookeeper servers which is working fine in version > 2.8.0 but now we get this error in 2.8.8. > > 2026-06-02 17:26:29.157 o.a.s.v.ConfigValidation main [INFO] Will use [class > org.apache.storm.DaemonConfig, class org.apache.storm.Config] for validation > 2026-06-02 17:26:29.317 o.a.s.z.ClientZookeeper main [INFO] Starting ZK > Curator > 2026-06-02 17:26:29.317 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl main [INFO] > Starting > 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:zookeeper.version=3.9.5-293c895a8d966a3ecb92872be4a1daf87d725da2, > built on 2026-02-11 20:18 UTC > 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:host.name=clrv0000294762.ic.ing.net > 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:java.version=17.0.19 > 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:java.vendor=Red Hat, Inc. > 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:java.home=/usr/lib/jvm/java-17-openjdk-17.0.19.0.10-2.el9.x86_64 > 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:java.class.path=/var/opt/storm/apache-storm-2.8.8/*:/var/opt/storm/apache-storm-2.8.8/lib/storm-shaded-deps-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/clojure-1.12.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-util-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-handler-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-marshalling-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-exec-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-jvm-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jspecify-1.0.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/slf4j-api-2.0.17.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-core-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-annotations-2.21.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-dataformat-smile-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/hadoop-auth-3.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-lang3-3.20.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jline-2.14.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-collections4-4.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/amqp-client-5.28.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-core-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-server-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/j2objc-annotations-3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-asn1-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.activation-1.2.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/javax.annotation-api-1.3.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-io-2.22.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/httpcore-4.4.16.jar:/var/opt/storm/apache-storm-2.8.8/lib/guava-33.6.0-jre.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-pkix-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-buffer-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-logging-1.3.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-server-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/audience-annotations-0.12.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.activation-api-1.2.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/zookeeper-jute-3.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/nimbus-jose-jwt-10.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-codec-1.22.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.xml.bind-api-4.0.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-slf4j2-impl-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/zookeeper-3.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-databind-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-fileupload-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/curator-framework-5.9.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-core-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-io-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/chill-java-0.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/hadoop-shaded-guava-1.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-clojure-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-util-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-graphite-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-core-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-util-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/curator-client-5.9.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-over-slf4j-2.0.17.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-client-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/reflectasm-1.11.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-transport-native-unix-common-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/rocksdbjni-10.10.1.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/snakeyaml-2.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/spec.alpha-0.5.238.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-compression-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/error_prone_annotations-2.49.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-config-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/minlog-1.3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/kryo-5.6.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/objenesis-3.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcprov-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-security-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcpkix-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-protobuf-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-transport-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.servlet-api-6.1.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcutil-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/json-smart-2.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-session-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-core-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-common-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/asm-9.9.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/failureaccess-1.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/core.specs.alpha-0.4.74.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-http-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-base-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-api-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-resolver-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-ee10-servlet-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-ee10-servlets-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/httpclient-4.5.14.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-crypto-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/tools.logging-1.3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/accessors-smart-2.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/carbonite-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-jmx-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-compress-1.28.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-cli-1.11.0.jar:/var/opt/storm/apache-storm-2.8.8/extlib/*:/var/opt/storm/apache-storm-2.8.8/extlib-daemon/*:/var/opt/storm/current/conf > 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64 > 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:java.io.tmpdir=/var/storm-tmp > 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:java.compiler=<NA> > 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:os.name=Linux > 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:os.arch=amd64 > 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:os.version=5.14.0-611.55.1.el9_7.x86_64 > 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:user.name=gmttusr > 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:user.home=/home/gmttusr > 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:user.dir=/ > 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:os.memory.free=36MB > 2026-06-02 17:26:29.325 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:os.memory.max=1024MB > 2026-06-02 17:26:29.325 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client > environment:os.memory.total=60MB > 2026-06-02 17:26:29.327 o.a.s.s.o.a.z.ZooKeeper main [INFO] Initiating client > connection, connectString=clrv0000301358.ic.ing.net:2181 sessionTimeout=20000 > watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@23a9ba52 > 2026-06-02 17:26:29.332 o.a.s.s.o.a.z.c.X509Util main [INFO] Setting -D > jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated > TLS renegotiation > 2026-06-02 17:26:29.471 o.a.s.s.o.a.z.c.X509Util main [INFO] Default TLS > protocol is TLSv1.3, supported TLS protocols are [TLSv1.3, TLSv1.2, TLSv1.1, > TLSv1, SSLv3, SSLv2Hello] > 2026-06-02 17:26:29.478 o.a.s.s.o.a.z.ClientCnxnSocket main [INFO] > jute.maxbuffer value is 1048575 Bytes > 2026-06-02 17:26:29.486 o.a.s.s.o.a.z.ClientCnxn main [INFO] > zookeeper.request.timeout value is 0. feature enabled=false > 2026-06-02 17:26:29.494 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl main [INFO] > Default schema > 2026-06-02 17:26:29.503 o.a.s.s.o.a.z.Login > main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Client successfully > logged in. > 2026-06-02 17:26:29.506 o.a.s.s.o.a.z.c.ZooKeeperSaslClient > main-SendThread(clrv0000301358.ic.ing.net:2181) [WARN] Client will not use > DIGEST-MD5 as SASL mechanism, because FIPS mode is enabled. > 2026-06-02 17:26:29.506 o.a.s.s.o.a.z.ClientCnxn > main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Opening socket > connection to server clrv0000301358.ic.ing.net/10.159.128.170:2181. > 2026-06-02 17:26:29.507 o.a.s.s.o.a.z.ClientCnxn > main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] SASL config status: > Will attempt to SASL-authenticate using Login Context section 'Client' > 2026-06-02 17:26:29.510 o.a.s.s.o.a.z.ClientCnxn > main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Socket connection > established, initiating session, client: /10.158.64.7:48370, server: > clrv0000301358.ic.ing.net/10.159.128.170:2181 > 2026-06-02 17:26:29.515 o.a.s.s.o.a.z.ClientCnxn > main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Session establishment > complete on server clrv0000301358.ic.ing.net/10.159.128.170:2181, session id > = 0x100000051861675, negotiated timeout = 20000 > 2026-06-02 17:26:29.516 o.a.s.s.o.a.z.ClientCnxn > main-SendThread(clrv0000301358.ic.ing.net:2181) [ERROR] SASL authentication > with Zookeeper Quorum member failed. > javax.security.sasl.SaslException: saslClient failed to initialize properly: > it's null. > at > org.apache.storm.shade.org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:402) > at > org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1212) > 2026-06-02 17:26:29.518 o.a.s.s.o.a.c.f.s.ConnectionStateManager > main-EventThread [INFO] State change: CONNECTED > 2026-06-02 17:26:29.520 o.a.s.s.o.a.z.ClientCnxn main-EventThread [ERROR] > Error while calling watcher. > java.lang.NullPointerException: Cannot invoke > "org.apache.storm.shade.org.apache.zookeeper.Watcher$Event$KeeperState.ordinal()" > because "state" is null > at > org.apache.storm.shade.org.apache.curator.ConnectionState.checkState(ConnectionState.java:206) > at > org.apache.storm.shade.org.apache.curator.ConnectionState.process(ConnectionState.java:146) > at > org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:566) > at > org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:541) > 2026-06-02 17:26:29.521 o.a.s.s.o.a.z.ClientCnxn main-EventThread [INFO] > EventThread shut down for session: 0x100000051861675 > 2026-06-02 17:26:29.524 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl > Curator-Framework-0 [INFO] backgroundOperationsLoop exiting > 2026-06-02 17:26:29.528 o.a.s.u.Utils main [ERROR] Received error in thread > main.. terminating server... > java.lang.Error: > org.apache.storm.shade.org.apache.zookeeper.KeeperException$AuthFailedException: > KeeperErrorCode = AuthFailed for /storm > at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:668) > at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:672) > at > org.apache.storm.utils.Utils.lambda$createDefaultUncaughtExceptionHandler$2(Utils.java:1055) > at > java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1082) > at > java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1077) > at > java.base/java.lang.Thread.dispatchUncaughtException(Thread.java:2017) > Caused by: > org.apache.storm.shade.org.apache.zookeeper.KeeperException$AuthFailedException: > KeeperErrorCode = AuthFailed for /storm > at > org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:129) > at > org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:53) > at > org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1867) > at > org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:247) > at > org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:240) > at > org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:88) > at > org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForegroundStandard(ExistsBuilderImpl.java:240) > at > org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:235) > at > org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:202) > at > org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:35) > at > org.apache.storm.zookeeper.AclEnforcement.verifyAcls(AclEnforcement.java:84) > at org.apache.storm.daemon.nimbus.Nimbus.launch(Nimbus.java:1608) > at org.apache.storm.daemon.nimbus.Nimbus.main(Nimbus.java:1615) > > Best regards, > Norman > ----------------------------------------------------------------- > ATTENTION: > The information in this e-mail is confidential and only meant for the > intended recipient. If you are not the intended recipient, don't use or > disclose it in any way. Please let the sender know and delete the message > immediately. > -----------------------------------------------------------------
