Just to let others know: I was able to get storm running with OpenJDK 15 and Zookeeper 3.6.2.
I had to manually build the storm shaded deps from the storm git repository and adapt the maven configuration so it includes all required dependencies and do the appropriate relocations in the pom.xml. Then, exclude the shaded deps from the individual storm project and replace the shaded deps on each worker on the storm cluster. It works nicely so far on our test cluster. I am not sure about dependencies in the storm project regarding Hadoop and Cassandra, as those seem to depend on Zookeeper 3.4.14. As we do not make use of these libraries, for us it is acceptable this way. However, if I have more time I will try to run the unit tests and see if the update goes through without breaking anything. If it does I could offer to do a pull request, in case this would be relevant to anyone here. Nice weekend everyone Jonas Am Mi., 25. Nov. 2020 um 12:03 Uhr schrieb Jonas Krauss <jkraus...@gmail.com >: > After doing more testing and finding the nimbus logfile in the storm dir, > I was able to narrow down the problem to a bug in Zookeeper 3.4, which is > somehow still used by storm internally, and Java 14: > > https://issues.apache.org/jira/browse/ZOOKEEPER-3779 > > Others seem to have the same problem: > > > https://stackoverflow.com/questions/63804162/apache-storm-local-cluster-unable-to-canonicalize-address > > It would be helpful if anybody knew how to overcome this, either by using > a higher zookeeper version in the shaded deps or by setting some config > options so it goes away. > > If someone from the storm devs thinks it's worth raising an issue I am > glad to contribute, if I can. > > Thanks > > Jonas > > > Am Di., 24. Nov. 2020 um 17:34 Uhr schrieb Jonas Krauss < > jkraus...@gmail.com>: > >> Hi all, >> >> we have recently tried to update zookeeper from 3.4.13 to 3.6.2 on our >> production storm cluster without success. Therefore, we are now tinkering >> with a test cluster, including just two nodes. >> >> The two nodes have the following setup: >> >> *OS* Ubuntu 20.04.1 LTS >> *Zookeeper *3.6.2 >> *Storm *2.2.0 >> *Java *OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04) >> *Python* 2.7.18 >> *Python2* 2.7.18 >> *Python3* 3.8.5 >> >> >> We are following this guide *to the letter*: >> http://storm.apache.org/releases/current/Setting-up-a-Storm-cluster.html >> >> First problem here: when following the guide and setting the mandatory >> parameters, you will end up with a configuration where the UI does not work. >> This is because according to the defaults.yaml, *both ui.port and >> storm.exhibitor.port are set to 8080*. This is where I get my >> information in regards to the default values from: >> https://github.com/apache/storm/blob/master/conf/defaults.yaml >> Left unconfigured, the UI will not start because it cannot bind to port >> 8080. Instead, you will see the 404 page from jetty. >> It took me quite a while to figure this, so I would be glad the person in >> charge could amend the docs to reflect the necessity to set one of the two >> to something different. >> >> Next problem, which I am not able to solve, but from searching the web I >> understand many storm users are facing similar problems, is the dreaded >> *NimbusLeaderNotFoundException: >> Could not find leader nimbus from seed hosts* >> Some examples: >> >> >> https://stackoverflow.com/questions/36742451/apache-storm-could-not-find-leader-nimbus-from-seed-hosts >> >> https://stackoverflow.com/questions/58155057/how-to-fix-could-not-find-leader-nimbus-from-seed-hosts-192-168-23-165 >> >> https://stackoverflow.com/questions/37480155/nimbusleadernotfoundexception-in-apache-storm >> >> >> I have double checked the ports are accessible on both nodes and that >> everything else is running as expected. >> >> Looking at the outputs from nimbus and supervisor, there is no error >> message (stderr is empty) and there is no useful logging in stdout, only >> the java command which starts the two processes. Upon accessing the UI in >> the browser, the process will load for a while, then, both nimbus and >> supervisor will crash/be killed/? and the error message appears. I have run >> the processes in supervision with supervisord and also have tried to start >> them via bash, to get more meaningful logging, no luck here. >> >> It is almost impossible to debug without more meaningful logging. My best >> guess is it has to do with that we >> >> Maybe I am missing something in the host configurations, but I am at a >> loss here so I would appreciate any pointers to where to look next. >> >> Attaching storm.yaml and hosts configurations below. >> >> Regards >> >> Jonas >> >> *storm.yaml* >> >> storm.zookeeper.servers: >> - "stormtest1.example.com" >> - "stormtest2.example.com" >> >> nimbus.seeds: ["stormtest1.example.com", "stormtest2.example.com"] >> >> storm.local.dir: "/storm" >> storm.zookeeper.root: "/zookeeper" >> supervisor.slots.ports: >> - 6700 >> - 6701 >> - 6702 >> - 6703 >> >> ui.port: 8888 >> >> >> */etc/hosts (node 1)* >> >> 127.0.1.1 storm-1 >> 127.0.0.1 localhost >> # Test Storm >> xxx.xxx.xxx.xxx stormtest1.example.com storm-test-1 >> xxx.xxx.xxx.xxx stormtest2.example.com storm-test-2 >> >> >> >> */etc/hosts (node 2)* >> >> 127.0.1.1 storm-2 >> 127.0.0.1 localhost >> # Test Storm >> xxx.xxx.xxx.xxx stormtest1.example.com storm-test-1 >> xxx.xxx.xxx.xxx stormtest2.example.com storm-test-2 >> >> >>
