Hi,
I have the same problem when running Kafka to Spark Streaming pipeline from
Java with explicitely specified message decoders. I had thought, that it was
related to Eclipse environment, as suggested here, but it's not the case. I
have coded an example based on class:
https://github.com/apache/spark/blob/branch-0.9/examples/src/main/java/org/apache/spark/streaming/examples/JavaKafkaWordCount.java
and have builded shaded uber jar with all the deps and tried to run it from
command line. When I use the createStream method from the example class like
this:
KafkaUtils.createStream(jssc, "zookeeper:port", "test", topicMap);
everything is working fine, but when I explicitely specify message decoder
classes used in this method with another overloaded createStream method:
KafkaUtils.createStream(jssc, String.class, String.class,
StringDecoder.class, StringDecoder.class, props, topicMap,
StorageLevels.MEMORY_AND_DISK_2);
the applications stops with an error:
14/06/10 22:28:06 ERROR kafka.KafkaReceiver: Error receiving data
java.lang.NoSuchMethodException:
java.lang.Object.(kafka.utils.VerifiableProperties)
at java.lang.Class.getConstructor0(Unknown Source)
at java.lang.Class.getConstructor(Unknown Source)
at
org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:108)
at
org.apache.spark.streaming.dstream.NetworkReceiver.start(NetworkInputDStream.scala:126)
I have tried Spark versions 0.9.0-incubating, 0.9.0 and 0.9.1, but the error
occurs everywhere. Kafka StringDecoder class has the constructor with
VerifiableProperties parameter and all required classes are in the same uber
jar, so it is strange that scala/java cannot find it with reflection api.
Maybe there is some problem with Manifest/ClassTag usage in KafkaUtils or
KafkaInputDStream classes, but I'm not a Scala expert and cannot be sure
about it. The problematic code is the same from version 0.9 to the current
one, so it's still there. Unit test from the Spark project is working fine
with every KafkaUtils method, because the test does not try to register the
kafka stream, only checks the interface.
Currently it is possible to use Kafka to Spark Streaming pipeline from Java
only with the default String message decoders, which makes this tool almost
useless (unless you are a great JSON fan).
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209p7347.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
