Hi,
Tried adding Kafka jar (kafka_2.11) in topology jar dependency as well as
in storm lib. Still its failing with same error.
KafkaConsumer class is loaded from kafka-clients-0.10.1.1.jar.
KafkaConsumer.subscribe call is happening from KafkaSpout which is loaded
from storm-kafka-client-1.0.3.jar.
*KafkaSpout source code shows below*. Its passing a List of topics to
subscribe call.
private void subscribeKafkaConsumer() {
kafkaConsumer = new KafkaConsumer<>(kafkaSpoutConfig
.getKafkaProps(),
kafkaSpoutConfig.getKeyDeserializer(), kafkaSpoutConfig
.getValueDeserializer());
if (kafkaSpoutStreams instanceof KafkaSpoutStreamsNamedTopics) {
* final List<String> topics = ((KafkaSpoutStreamsNamedTopics)
kafkaSpoutStreams).getTopics();*
kafkaConsumer.subscribe(topics, new
KafkaSpoutConsumerRebalanceListener());
LOG.info("Kafka consumer subscribed topics {}", topics);
}
Verified the worker logs and confirmed that its loading the proper kafka
version.
2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser
Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka version : 0.10.1.1
2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser
Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka commitId :
f10ef2720b03b247
2017-02-25 21:20:50.376 o.a.k.c.u.AppInfoParser
Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka version : 0.10.1.1
2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka commitId :
f10ef2720b03b247
2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
Thread-73-merchSpout-executor[432 433] [INFO] Kafka version : 0.10.1.1
2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
Thread-73-merchSpout-executor[432 433] [INFO] Kafka commitId :
f10ef2720b03b247
2017-02-25 21:20:50.381 o.a.k.c.u.AppInfoParser
Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka version : 0.10.1.1
2017-02-25 21:20:50.382 o.a.k.c.u.AppInfoParser
Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka commitId :
f10ef2720b03b247
Regards
Pradeep S
On Sat, Feb 25, 2017 at 9:06 AM, Kunal Gupta <kunal.gu...@kartrocket.com>
wrote:
> Please add Kafka maven repo in Storm lib directory
>
> On Sat, Feb 25, 2017 at 2:57 PM, pradeep s <sreekumar.prad...@gmail.com>
> wrote:
>
>> Hi,
>> I am using Storm 1.0.3 and kafka 0.10.1.1 and initialising KafkaSpout
>> using storm-kafka-client
>> https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md
>> Dependencies used are
>>
>> <dependency>
>> <groupId>org.apache.kafka</groupId>
>> <artifactId>kafka-clients</artifactId>
>> <version>0.10.1.1</version>
>> </dependency>
>> <dependency>
>> <groupId>org.apache.storm</groupId>
>> <artifactId>storm-kafka-client</artifactId>
>> <version>1.0.3</version>
>> </dependency>
>>
>> While starting the topology spout is failing by below error
>> java.lang.*NoSuchMethodError*: org.apache.kafka.clients.consu
>> mer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/
>> kafka/clients/consumer/ConsumerRebalanceListener;)V
>> at
>> org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:360)
>> ~[stormjar.jar:?]
>> at
>> org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:351)
>> ~[stormjar.jar:?]
>> at
>> org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:639)
>> ~[storm-core-1.0.3.jar:1.0.3]
>> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
>> [storm-core-1.0.3.jar:1.0.3]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
>>
>> Regards
>> Pradeep S
>>
>
>
>
> --
> *Thanks **Kunal*
> *+91-9958189589 <+91%2099581%2089589>*
> *Senior Software Engineer*
>
>
> *Bigfoot Retail Solutions Pvt Ltd.Khasra No. 264, Westend Marg,
> Saidulajab, Saket 110030http://www.kartrocket.com
> <http://www.kartrocket.com/>*
>
