Dear Storm users and developers, I have been running the Yahoo streaming benchmarks on Storm 0.9.7. The entire Cloud infrastructure includes Zookeeper 3.4.6 on 3 nodes, 5 Kafka 2.10-0.8.2.1 nodes, 1 Storm 0.9.7 Nimbus node, 10 Storm 0.9.7 supervisor nodes with 4 worker processes on each node, 10 kafka producer nodes, and 1 redis 3.0.5 node.
With the introduction of Storm 1.2.0, I decided to upgrade from 0.9.7 to 1.2.0. Currently I'm testing Yahoo streaming benchmark's topology on the new setup and I end up getting the following exceptions: # streaming-benchmarks/apache-storm-1.2.0/logs/workers-artifacts/test-topo-1-1519628010/6700/worker.log 148 java.lang.RuntimeException: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /ad-events/7183b5b2-4971-41a1-b86d-0788f646bc64/partition_0 149 at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:84) ~[stormjar.jar:?] 150 at org.apache.storm.kafka.ZkState.writeJSON(ZkState.java:70) ~[stormjar.jar:?] 151 at org.apache.storm.kafka.PartitionManager.commit(PartitionManager.java:361) ~[stormjar.jar:?] 152 at org.apache.storm.kafka.KafkaSpout.commit(KafkaSpout.java:257) ~[stormjar.jar:?] 153 at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:156) ~[stormjar.jar:?] 154 at storm.benchmark.AdvertisingTopology$MyKafkaSpout.nextTuple(AdvertisingTopology.java:181) ~[stormjar.jar:?] 155 at org.apache.storm.daemon.executor$fn__9610$fn__9625$fn__9656.invoke(executor.clj:654) ~[storm-core-1.2.0.jar:1.2.0] 156 at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.2.0.jar:1.2.0] 157 at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] 158 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74] 159 Caused by: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /ad-events/7183b5b2-4971-41a1-b86d-0788f646bc64/partition_0 160 at org.apache.zookeeper.KeeperException.create(KeeperException.java:103) ~[stormjar.jar:?] 161 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[stormjar.jar:?] 162 at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1525) ~[stormjar.jar:?] 163 at org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1189) ~[stormjar.jar:?] 164 at org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1166) ~[stormjar.jar:?] 165 at org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64) ~[stormjar.jar:?] 166 at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100) ~[stormjar.jar:?] 167 at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:1163) ~[stormjar.jar:?] 168 at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:603) ~[stormjar.jar:?] 169 at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:593) ~[stormjar.jar:?] 170 at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:47) ~[stormjar.jar:?] 171 at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:79) ~[stormjar.jar:?] 172 ... 9 more For each instance of benchmark execution, more than one of the spout instances will eventually generate exception messages in their log files like above. Some people suggest that it's probably the compatibility issue between Zookeeper and Curator: https://stackoverflow.com/a/35753999 > Apparently this error can occur if you're using the wrong combination of > Curator in combination with Zookeeper. From curator.apache.org : >> Curator 2.x.x - compatible with both ZooKeeper 3.4.x and ZooKeeper 3.5.x >> Curator 3.x.x - compatible only with ZooKeeper 3.5.x and includes >> support for new features such as dynamic reconfiguration, etc. http://curator.apache.org > Apache Curator is meant to be used with ZooKeeper 3.5+. However, it > is also compatible with ZooKeeper 3.4.x. See Compatibility for details. http://curator.apache.org/zk-compatibility.html > While ZooKeeper 3.5.x is still considered "beta" by the ZooKeeper > development team, the reality is that it is used in production by many users. > However, ZooKeeper 3.4.x is also used in production. Prior to Apache Curator 4.0, > both versions of ZooKeeper were supported via two versions of Apache Curator. > Starting with Curator 4.0 both versions of ZooKeeper are supported via > the same Curator libraries. > > ZooKeeper 3.5.x > Curator 4.0 has a hard dependency on ZooKeeper 3.5.x > If you are using ZooKeeper 3.5.x there's nothing additional to do - > just use Curator 4.0 > > ZooKeeper 3.4.x > Curator 4.0 supports ZooKeeper 3.4.x ensembles in a soft-compatibility > mode. To use this mode you must exclude ZooKeeper when adding > Curator to your dependency management tool. Could it be possible that something went wrong when Curator 4.0.0 is working against Zookeeper 3.4.6? (I'm really wondering about this soft-compatibility mode introduced in Curator 4.0) Thank you for reading. Best regards, Shinhyung Yang
