Unfortunately, storm-kafka-client 1.0.x is compatible with Kafka 0.9 and storm-kafka-client 1.1.0 (not released yet but we'll make it happen) is compatible with Kafka 0.10.
https://github.com/apache/storm/blob/1.0.x-branch/external/storm-kafka-client/README.md https://github.com/apache/storm/blob/1.x-branch/docs/storm-kafka-client.md You may want to participate release vote for 1.1.0 RC2 and make it happen soon. Thanks, Jungtaek Lim (HeartSaVioR) 2017년 2월 26일 (일) 오전 9:21, Erik Weathers <eweath...@groupon.com>님이 작성: > You must be doing *something* wrong. > > One thing that stands out to me is that you're using a different version > of the kafka-client (0.10.1.1) than storm-kafka-client has configured > (0.9.0.1): > > - https://github.com/apache/storm/blob/v1.0.3/pom.xml#L256-L257 > > If the problem isn't related to that, then I'd say you are somehow not > including kafka-clients jar into your uber jar. Can you "jar tf" your uber > jar and ensure it has the KafkaConsumer class in it? Then you could use > "jar xf" to get the .class file, and then use "javap" to decompile and look > for the KafkaConsumer.subscribe() method. > > - Erik > > On Sat, Feb 25, 2017 at 1:26 PM, pradeep s <sreekumar.prad...@gmail.com> > wrote: > > Hi, > Tried adding Kafka jar (kafka_2.11) in topology jar dependency as well as > in storm lib. Still its failing with same error. > KafkaConsumer class is loaded from kafka-clients-0.10.1.1.jar. > KafkaConsumer.subscribe call is happening from KafkaSpout which is loaded > from storm-kafka-client-1.0.3.jar. > > *KafkaSpout source code shows below*. Its passing a List of topics to > subscribe call. > > private void subscribeKafkaConsumer() { > > kafkaConsumer = new KafkaConsumer<>(kafkaSpoutConfig > .getKafkaProps(), > > kafkaSpoutConfig.getKeyDeserializer(), kafkaSpoutConfig > .getValueDeserializer()); > > > if (kafkaSpoutStreams instanceof KafkaSpoutStreamsNamedTopics) { > > * final List<String> topics = ((KafkaSpoutStreamsNamedTopics) > kafkaSpoutStreams).getTopics();* > > kafkaConsumer.subscribe(topics, new > KafkaSpoutConsumerRebalanceListener()); > > LOG.info("Kafka consumer subscribed topics {}", topics); > > } > > > Verified the worker logs and confirmed that its loading the proper kafka > version. > > 2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser > Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka version : 0.10.1.1 > > 2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser > Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka commitId : > f10ef2720b03b247 > > 2017-02-25 21:20:50.376 o.a.k.c.u.AppInfoParser > Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka version : 0.10.1.1 > > 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser > Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka commitId : > f10ef2720b03b247 > > 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser > Thread-73-merchSpout-executor[432 433] [INFO] Kafka version : 0.10.1.1 > > 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser > Thread-73-merchSpout-executor[432 433] [INFO] Kafka commitId : > f10ef2720b03b247 > > 2017-02-25 21:20:50.381 o.a.k.c.u.AppInfoParser > Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka version : 0.10.1.1 > > 2017-02-25 21:20:50.382 o.a.k.c.u.AppInfoParser > Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka commitId : > f10ef2720b03b247 > > > > Regards > > Pradeep S > > On Sat, Feb 25, 2017 at 9:06 AM, Kunal Gupta <kunal.gu...@kartrocket.com> > wrote: > > Please add Kafka maven repo in Storm lib directory > > On Sat, Feb 25, 2017 at 2:57 PM, pradeep s <sreekumar.prad...@gmail.com> > wrote: > > Hi, > I am using Storm 1.0.3 and kafka 0.10.1.1 and initialising KafkaSpout > using storm-kafka-client > https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md > Dependencies used are > > <dependency> > <groupId>org.apache.kafka</groupId> > <artifactId>kafka-clients</artifactId> > <version>0.10.1.1</version> > </dependency> > <dependency> > <groupId>org.apache.storm</groupId> > <artifactId>storm-kafka-client</artifactId> > <version>1.0.3</version> > </dependency> > > While starting the topology spout is failing by below error > java.lang.*NoSuchMethodError*: > org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/kafka/clients/consumer/ConsumerRebalanceListener;)V > at > org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:360) > ~[stormjar.jar:?] > at > org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:351) > ~[stormjar.jar:?] > at > org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:639) > ~[storm-core-1.0.3.jar:1.0.3] > at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) > [storm-core-1.0.3.jar:1.0.3] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111] > > Regards > Pradeep S > > > > > -- > *Thanks **Kunal* > *+91-9958189589 <+91%2099581%2089589>* > *Senior Software Engineer* > > > *Bigfoot Retail Solutions Pvt Ltd.Khasra No. 264, Westend Marg, > Saidulajab, Saket 110030http://www.kartrocket.com > <http://www.kartrocket.com/>* > > > >
