After doing more testing and finding the nimbus logfile in the storm dir, I was able to narrow down the problem to a bug in Zookeeper 3.4, which is somehow still used by storm internally, and Java 14:
https://issues.apache.org/jira/browse/ZOOKEEPER-3779 Others seem to have the same problem: https://stackoverflow.com/questions/63804162/apache-storm-local-cluster-unable-to-canonicalize-address It would be helpful if anybody knew how to overcome this, either by using a higher zookeeper version in the shaded deps or by setting some config options so it goes away. If someone from the storm devs thinks it's worth raising an issue I am glad to contribute, if I can. Thanks Jonas Am Di., 24. Nov. 2020 um 17:34 Uhr schrieb Jonas Krauss <jkraus...@gmail.com >: > Hi all, > > we have recently tried to update zookeeper from 3.4.13 to 3.6.2 on our > production storm cluster without success. Therefore, we are now tinkering > with a test cluster, including just two nodes. > > The two nodes have the following setup: > > *OS* Ubuntu 20.04.1 LTS > *Zookeeper *3.6.2 > *Storm *2.2.0 > *Java *OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04) > *Python* 2.7.18 > *Python2* 2.7.18 > *Python3* 3.8.5 > > > We are following this guide *to the letter*: > http://storm.apache.org/releases/current/Setting-up-a-Storm-cluster.html > > First problem here: when following the guide and setting the mandatory > parameters, you will end up with a configuration where the UI does not work. > This is because according to the defaults.yaml, *both ui.port and > storm.exhibitor.port are set to 8080*. This is where I get my information > in regards to the default values from: > https://github.com/apache/storm/blob/master/conf/defaults.yaml > Left unconfigured, the UI will not start because it cannot bind to port > 8080. Instead, you will see the 404 page from jetty. > It took me quite a while to figure this, so I would be glad the person in > charge could amend the docs to reflect the necessity to set one of the two > to something different. > > Next problem, which I am not able to solve, but from searching the web I > understand many storm users are facing similar problems, is the dreaded > *NimbusLeaderNotFoundException: > Could not find leader nimbus from seed hosts* > Some examples: > > > https://stackoverflow.com/questions/36742451/apache-storm-could-not-find-leader-nimbus-from-seed-hosts > > https://stackoverflow.com/questions/58155057/how-to-fix-could-not-find-leader-nimbus-from-seed-hosts-192-168-23-165 > > https://stackoverflow.com/questions/37480155/nimbusleadernotfoundexception-in-apache-storm > > > I have double checked the ports are accessible on both nodes and that > everything else is running as expected. > > Looking at the outputs from nimbus and supervisor, there is no error > message (stderr is empty) and there is no useful logging in stdout, only > the java command which starts the two processes. Upon accessing the UI in > the browser, the process will load for a while, then, both nimbus and > supervisor will crash/be killed/? and the error message appears. I have run > the processes in supervision with supervisord and also have tried to start > them via bash, to get more meaningful logging, no luck here. > > It is almost impossible to debug without more meaningful logging. My best > guess is it has to do with that we > > Maybe I am missing something in the host configurations, but I am at a > loss here so I would appreciate any pointers to where to look next. > > Attaching storm.yaml and hosts configurations below. > > Regards > > Jonas > > *storm.yaml* > > storm.zookeeper.servers: > - "stormtest1.example.com" > - "stormtest2.example.com" > > nimbus.seeds: ["stormtest1.example.com", "stormtest2.example.com"] > > storm.local.dir: "/storm" > storm.zookeeper.root: "/zookeeper" > supervisor.slots.ports: > - 6700 > - 6701 > - 6702 > - 6703 > > ui.port: 8888 > > > */etc/hosts (node 1)* > > 127.0.1.1 storm-1 > 127.0.0.1 localhost > # Test Storm > xxx.xxx.xxx.xxx stormtest1.example.com storm-test-1 > xxx.xxx.xxx.xxx stormtest2.example.com storm-test-2 > > > > */etc/hosts (node 2)* > > 127.0.1.1 storm-2 > 127.0.0.1 localhost > # Test Storm > xxx.xxx.xxx.xxx stormtest1.example.com storm-test-1 > xxx.xxx.xxx.xxx stormtest2.example.com storm-test-2 > > >
