Hi, I have the same problem when running Kafka to Spark Streaming pipeline from Java with explicitely specified message decoders. I had thought, that it was related to Eclipse environment, as suggested here, but it's not the case. I have coded an example based on class:
https://github.com/apache/spark/blob/branch-0.9/examples/src/main/java/org/apache/spark/streaming/examples/JavaKafkaWordCount.java and have builded shaded uber jar with all the deps and tried to run it from command line. When I use the createStream method from the example class like this: KafkaUtils.createStream(jssc, "zookeeper:port", "test", topicMap); everything is working fine, but when I explicitely specify message decoder classes used in this method with another overloaded createStream method: KafkaUtils.createStream(jssc, String.class, String.class, StringDecoder.class, StringDecoder.class, props, topicMap, StorageLevels.MEMORY_AND_DISK_2); the applications stops with an error: 14/06/10 22:28:06 ERROR kafka.KafkaReceiver: Error receiving data java.lang.NoSuchMethodException: java.lang.Object.<init>(kafka.utils.VerifiableProperties) at java.lang.Class.getConstructor0(Unknown Source) at java.lang.Class.getConstructor(Unknown Source) at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:108) at org.apache.spark.streaming.dstream.NetworkReceiver.start(NetworkInputDStream.scala:126) I have tried Spark versions 0.9.0-incubating, 0.9.0 and 0.9.1, but the error occurs everywhere. Kafka StringDecoder class has the constructor with VerifiableProperties parameter and all required classes are in the same uber jar, so it is strange that scala/java cannot find it with reflection api. Maybe there is some problem with Manifest/ClassTag usage in KafkaUtils or KafkaInputDStream classes, but I'm not a Scala expert and cannot be sure about it. The problematic code is the same from version 0.9 to the current one, so it's still there. Unit test from the Spark project is working fine with every KafkaUtils method, because the test does not try to register the kafka stream, only checks the interface. Currently it is possible to use Kafka to Spark Streaming pipeline from Java only with the default String message decoders, which makes this tool almost useless (unless you are a great JSON fan). -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209p7347.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
