Hi Shinhyung, I looked into this, it's a bug. Zookeeper 3.5.3-beta is ending up on the classpath, which was not the intent. I've raised https://issues.apache.org/jira/browse/STORM-2978 to track fixing the issue. For now, you can work around it by adding an explicit dependency on Zookeeper 3.4.6 to your pom.
2018-02-26 17:35 GMT+01:00 Shinhyung Yang <[email protected]>: > Dear Storm users and developers, > > I have been running the Yahoo streaming benchmarks on Storm 0.9.7. The > entire Cloud infrastructure includes Zookeeper 3.4.6 on 3 nodes, 5 Kafka > 2.10-0.8.2.1 nodes, 1 Storm 0.9.7 Nimbus node, 10 Storm 0.9.7 supervisor > nodes with 4 worker processes on each node, 10 kafka producer nodes, and 1 > redis 3.0.5 node. > > With the introduction of Storm 1.2.0, I decided to upgrade from 0.9.7 to > 1.2.0. Currently I'm testing Yahoo streaming benchmark's topology on the > new setup and I end up getting the following exceptions: > > # streaming-benchmarks/apache-storm-1.2.0/logs/workers- > artifacts/test-topo-1-1519628010/6700/worker.log > > 148 java.lang.RuntimeException: > org.apache.zookeeper.KeeperException$UnimplementedException: > KeeperErrorCode = Unimplemented for /ad-events/7183b5b2-4971-41a1- > b86d-0788f646bc64/partition_0 > 149 at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:84) > ~[stormjar.jar:?] > 150 at org.apache.storm.kafka.ZkState.writeJSON(ZkState.java:70) > ~[stormjar.jar:?] > 151 at > org.apache.storm.kafka.PartitionManager.commit(PartitionManager.java:361) > ~[stormjar.jar:?] > 152 at org.apache.storm.kafka.KafkaSpout.commit(KafkaSpout.java:257) > ~[stormjar.jar:?] > 153 at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:156) > ~[stormjar.jar:?] > 154 at > storm.benchmark.AdvertisingTopology$MyKafkaSpout.nextTuple(AdvertisingTopology.java:181) > ~[stormjar.jar:?] > 155 at > org.apache.storm.daemon.executor$fn__9610$fn__9625$fn__9656.invoke(executor.clj:654) > ~[storm-core-1.2.0.jar:1.2.0] > 156 at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) > [storm-core-1.2.0.jar:1.2.0] > 157 at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > 158 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74] > 159 Caused by: org.apache.zookeeper.KeeperException$UnimplementedException: > KeeperErrorCode = Unimplemented for /ad-events/7183b5b2-4971-41a1- > b86d-0788f646bc64/partition_0 > 160 at > org.apache.zookeeper.KeeperException.create(KeeperException.java:103) > ~[stormjar.jar:?] > 161 at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > ~[stormjar.jar:?] > 162 at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1525) > ~[stormjar.jar:?] > 163 at org.apache.curator.framework.imps.CreateBuilderImpl$17. > call(CreateBuilderImpl.java:1189) ~[stormjar.jar:?] > 164 at org.apache.curator.framework.imps.CreateBuilderImpl$17. > call(CreateBuilderImpl.java:1166) ~[stormjar.jar:?] > 165 at org.apache.curator.connection.StandardConnectionHandlingPoli > cy.callWithRetry(StandardConnectionHandlingPolicy.java:64) > ~[stormjar.jar:?] > 166 at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100) > ~[stormjar.jar:?] > 167 at org.apache.curator.framework.imps.CreateBuilderImpl. > pathInForeground(CreateBuilderImpl.java:1163) ~[stormjar.jar:?] > 168 at org.apache.curator.framework.imps.CreateBuilderImpl. > protectedPathInForeground(CreateBuilderImpl.java:603) ~[stormjar.jar:?] > 169 at org.apache.curator.framework.imps.CreateBuilderImpl. > forPath(CreateBuilderImpl.java:593) ~[stormjar.jar:?] > 170 at org.apache.curator.framework.imps.CreateBuilderImpl. > forPath(CreateBuilderImpl.java:47) ~[stormjar.jar:?] > 171 at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:79) > ~[stormjar.jar:?] > 172 ... 9 more > > For each instance of benchmark execution, more than one of the spout > instances will eventually generate exception messages in their log files > like above. > > Some people suggest that it's probably the compatibility issue between > Zookeeper and Curator: > > https://stackoverflow.com/a/35753999 > > Apparently this error can occur if you're using the wrong combination of > > Curator in combination with Zookeeper. From curator.apache.org : > >> Curator 2.x.x - compatible with both ZooKeeper 3.4.x and ZooKeeper 3.5.x > >> Curator 3.x.x - compatible only with ZooKeeper 3.5.x and includes > >> support for new features such as dynamic reconfiguration, etc. > > http://curator.apache.org > > Apache Curator is meant to be used with ZooKeeper 3.5+. However, it > > is also compatible with ZooKeeper 3.4.x. See Compatibility for details. > > http://curator.apache.org/zk-compatibility.html > > While ZooKeeper 3.5.x is still considered "beta" by the ZooKeeper > > development team, the reality is that it is used in production by many > users. > > However, ZooKeeper 3.4.x is also used in production. Prior to Apache > Curator 4.0, > > both versions of ZooKeeper were supported via two versions of Apache > Curator. > > Starting with Curator 4.0 both versions of ZooKeeper are supported via > > the same Curator libraries. > > > > ZooKeeper 3.5.x > > Curator 4.0 has a hard dependency on ZooKeeper 3.5.x > > If you are using ZooKeeper 3.5.x there's nothing additional to do - > > just use Curator 4.0 > > > > ZooKeeper 3.4.x > > Curator 4.0 supports ZooKeeper 3.4.x ensembles in a soft-compatibility > > mode. To use this mode you must exclude ZooKeeper when adding > > Curator to your dependency management tool. > > > Could it be possible that something went wrong when Curator 4.0.0 is > working against Zookeeper 3.4.6? (I'm really wondering about this > soft-compatibility mode introduced in Curator 4.0) > > Thank you for reading. > > Best regards, > Shinhyung Yang > > >
