Re: NoSuchMethodError in KafkaReciever
To be honest I'm a scala newbie too. I just copied it from createStream.
I assume it's the canonical way to convert a java map (JMap) to a scala
map (Map)
On Mon, Jul 7, 2014 at 1:40 PM, mcampbell
wrote:
> xtrahotsauce wrote
> > I had this same problem as well. I ended up just adding the necessary
> > code
> > in KafkaUtil and compiling my own spark jar. Something like this for the
> > "raw" stream:
> >
> > def createRawStream(
> > jssc: JavaStreamingContext,
> > kafkaParams: JMap[String, String],
> > topics: JMap[String, JInt]
> >): JavaPairDStream[Array[Byte], Array[Byte]] = {
> > new KafkaInputDStream[Array[Byte], Array[Byte], DefaultDecoder,
> > DefaultDecoder](
> > jssc.ssc, kafkaParams.toMap,
> > Map(topics.mapValues(_.intValue()).toSeq: _*),
> > StorageLevel.MEMORY_AND_DISK_SER_2)
> > }
>
>
> I had this same problem, and this solution also worked for me so thanks for
> this!
>
> One question... what is this doing?
>
> > Map(topics.mapValues(_.intValue()).toSeq: _*),
>
> it appears to be converting the incoming Map[String, Integer] to a
> Map[String, Integer]. I'm not seeing the purpose of it... help? (I'm a
> bit of a scala newbie.)
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209p8953.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: NoSuchMethodError in KafkaReciever
xtrahotsauce wrote
> I had this same problem as well. I ended up just adding the necessary
> code
> in KafkaUtil and compiling my own spark jar. Something like this for the
> "raw" stream:
>
> def createRawStream(
> jssc: JavaStreamingContext,
> kafkaParams: JMap[String, String],
> topics: JMap[String, JInt]
>): JavaPairDStream[Array[Byte], Array[Byte]] = {
> new KafkaInputDStream[Array[Byte], Array[Byte], DefaultDecoder,
> DefaultDecoder](
> jssc.ssc, kafkaParams.toMap,
> Map(topics.mapValues(_.intValue()).toSeq: _*),
> StorageLevel.MEMORY_AND_DISK_SER_2)
> }
I had this same problem, and this solution also worked for me so thanks for
this!
One question... what is this doing?
> Map(topics.mapValues(_.intValue()).toSeq: _*),
it appears to be converting the incoming Map[String, Integer] to a
Map[String, Integer]. I'm not seeing the purpose of it... help? (I'm a
bit of a scala newbie.)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209p8953.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: NoSuchMethodError in KafkaReciever
The createRawStream method by xtrahotsauce, which operate on byte array could be proposed as a workaround for this bug until it is fixed. The message should be decoded in map/reduce phase then, but it's better than nothing. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209p7400.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: NoSuchMethodError in KafkaReciever
I added https://issues.apache.org/jira/browse/SPARK-2103 to track this. I also ran into it. I don't have a fix, but, somehow I think someone with more understanding of Scala and Manifest objects might see the easy fix. On Tue, Jun 10, 2014 at 5:15 PM, mpieck wrote: > Hi, > > I have the same problem when running Kafka to Spark Streaming pipeline from > Java with explicitely specified message decoders. I had thought, that it was > related to Eclipse environment, as suggested here, but it's not the case. I > have coded an example based on class: > > https://github.com/apache/spark/blob/branch-0.9/examples/src/main/java/org/apache/spark/streaming/examples/JavaKafkaWordCount.java > > and have builded shaded uber jar with all the deps and tried to run it from > command line. When I use the createStream method from the example class like > this: > > KafkaUtils.createStream(jssc, "zookeeper:port", "test", topicMap); > > everything is working fine, but when I explicitely specify message decoder > classes used in this method with another overloaded createStream method: > > KafkaUtils.createStream(jssc, String.class, String.class, > StringDecoder.class, StringDecoder.class, props, topicMap, > StorageLevels.MEMORY_AND_DISK_2); > > the applications stops with an error: > > 14/06/10 22:28:06 ERROR kafka.KafkaReceiver: Error receiving data > java.lang.NoSuchMethodException: > java.lang.Object.(kafka.utils.VerifiableProperties) > at java.lang.Class.getConstructor0(Unknown Source) > at java.lang.Class.getConstructor(Unknown Source) > at > org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:108) > at > org.apache.spark.streaming.dstream.NetworkReceiver.start(NetworkInputDStream.scala:126) > > I have tried Spark versions 0.9.0-incubating, 0.9.0 and 0.9.1, but the error > occurs everywhere. Kafka StringDecoder class has the constructor with > VerifiableProperties parameter and all required classes are in the same uber > jar, so it is strange that scala/java cannot find it with reflection api. > Maybe there is some problem with Manifest/ClassTag usage in KafkaUtils or > KafkaInputDStream classes, but I'm not a Scala expert and cannot be sure > about it. The problematic code is the same from version 0.9 to the current > one, so it's still there. Unit test from the Spark project is working fine > with every KafkaUtils method, because the test does not try to register the > kafka stream, only checks the interface. > > Currently it is possible to use Kafka to Spark Streaming pipeline from Java > only with the default String message decoders, which makes this tool almost > useless (unless you are a great JSON fan). > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209p7347.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: NoSuchMethodError in KafkaReciever
I had this same problem as well. I ended up just adding the necessary code
in KafkaUtil and compiling my own spark jar. Something like this for the
"raw" stream:
def createRawStream(
jssc: JavaStreamingContext,
kafkaParams: JMap[String, String],
topics: JMap[String, JInt]
): JavaPairDStream[Array[Byte], Array[Byte]] = {
new KafkaInputDStream[Array[Byte], Array[Byte], DefaultDecoder,
DefaultDecoder](
jssc.ssc, kafkaParams.toMap,
Map(topics.mapValues(_.intValue()).toSeq: _*),
StorageLevel.MEMORY_AND_DISK_SER_2)
}
On Tue, Jun 10, 2014 at 2:15 PM, mpieck wrote:
> Hi,
>
> I have the same problem when running Kafka to Spark Streaming pipeline from
> Java with explicitely specified message decoders. I had thought, that it
> was
> related to Eclipse environment, as suggested here, but it's not the case. I
> have coded an example based on class:
>
>
> https://github.com/apache/spark/blob/branch-0.9/examples/src/main/java/org/apache/spark/streaming/examples/JavaKafkaWordCount.java
>
> and have builded shaded uber jar with all the deps and tried to run it from
> command line. When I use the createStream method from the example class
> like
> this:
>
> KafkaUtils.createStream(jssc, "zookeeper:port", "test", topicMap);
>
> everything is working fine, but when I explicitely specify message decoder
> classes used in this method with another overloaded createStream method:
>
> KafkaUtils.createStream(jssc, String.class, String.class,
> StringDecoder.class, StringDecoder.class, props, topicMap,
> StorageLevels.MEMORY_AND_DISK_2);
>
> the applications stops with an error:
>
> 14/06/10 22:28:06 ERROR kafka.KafkaReceiver: Error receiving data
> java.lang.NoSuchMethodException:
> java.lang.Object.(kafka.utils.VerifiableProperties)
> at java.lang.Class.getConstructor0(Unknown Source)
> at java.lang.Class.getConstructor(Unknown Source)
> at
>
> org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:108)
> at
>
> org.apache.spark.streaming.dstream.NetworkReceiver.start(NetworkInputDStream.scala:126)
>
> I have tried Spark versions 0.9.0-incubating, 0.9.0 and 0.9.1, but the
> error
> occurs everywhere. Kafka StringDecoder class has the constructor with
> VerifiableProperties parameter and all required classes are in the same
> uber
> jar, so it is strange that scala/java cannot find it with reflection api.
> Maybe there is some problem with Manifest/ClassTag usage in KafkaUtils or
> KafkaInputDStream classes, but I'm not a Scala expert and cannot be sure
> about it. The problematic code is the same from version 0.9 to the current
> one, so it's still there. Unit test from the Spark project is working fine
> with every KafkaUtils method, because the test does not try to register the
> kafka stream, only checks the interface.
>
> Currently it is possible to use Kafka to Spark Streaming pipeline from Java
> only with the default String message decoders, which makes this tool almost
> useless (unless you are a great JSON fan).
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209p7347.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: NoSuchMethodError in KafkaReciever
Hi, I have the same problem when running Kafka to Spark Streaming pipeline from Java with explicitely specified message decoders. I had thought, that it was related to Eclipse environment, as suggested here, but it's not the case. I have coded an example based on class: https://github.com/apache/spark/blob/branch-0.9/examples/src/main/java/org/apache/spark/streaming/examples/JavaKafkaWordCount.java and have builded shaded uber jar with all the deps and tried to run it from command line. When I use the createStream method from the example class like this: KafkaUtils.createStream(jssc, "zookeeper:port", "test", topicMap); everything is working fine, but when I explicitely specify message decoder classes used in this method with another overloaded createStream method: KafkaUtils.createStream(jssc, String.class, String.class, StringDecoder.class, StringDecoder.class, props, topicMap, StorageLevels.MEMORY_AND_DISK_2); the applications stops with an error: 14/06/10 22:28:06 ERROR kafka.KafkaReceiver: Error receiving data java.lang.NoSuchMethodException: java.lang.Object.(kafka.utils.VerifiableProperties) at java.lang.Class.getConstructor0(Unknown Source) at java.lang.Class.getConstructor(Unknown Source) at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:108) at org.apache.spark.streaming.dstream.NetworkReceiver.start(NetworkInputDStream.scala:126) I have tried Spark versions 0.9.0-incubating, 0.9.0 and 0.9.1, but the error occurs everywhere. Kafka StringDecoder class has the constructor with VerifiableProperties parameter and all required classes are in the same uber jar, so it is strange that scala/java cannot find it with reflection api. Maybe there is some problem with Manifest/ClassTag usage in KafkaUtils or KafkaInputDStream classes, but I'm not a Scala expert and cannot be sure about it. The problematic code is the same from version 0.9 to the current one, so it's still there. Unit test from the Spark project is working fine with every KafkaUtils method, because the test does not try to register the kafka stream, only checks the interface. Currently it is possible to use Kafka to Spark Streaming pipeline from Java only with the default String message decoders, which makes this tool almost useless (unless you are a great JSON fan). -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209p7347.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: NoSuchMethodError in KafkaReciever
Will give it a shot, later. BTW, this forced me to move to Scala! Decided to
design our aggregation frame-work in scala for now.
On 07-Mar-2014, at 6:02 AM, Tathagata Das wrote:
> I dont have a Eclipse setup so I am not sure what is going on here. I would
> try to use maven in the command line with a pom to see if this compiles.
> Also, try to cleanup your system maven cache. Who knows if it had pulled in a
> wrong version of kafka 0.8 and using it all the time. Blowing away the cache
> and clean compiling will make sure the right kafka will be loaded.
>
> Hope this helps.
>
> TD
>
>
> On Sat, Mar 1, 2014 at 8:26 PM, venki-kratos wrote:
> I am trying to user code similar to following :
> public JavaPairDStream openStream() {
> HashMap kafkaParams = Maps.newHashMap();
> kafkaParams.put(ZK_CONNECT,kafkaConfig.getString(ZK_CONNECT));
>
> kafkaParams.put(CONSUMER_GRP_ID,kafkaConfig.getString(CONSUMER_GRP_ID));
>
> Map topicMap = Maps.newHashMap();
> topicMap.put(kafkaConfig.getString(ZK_TOPIC),
> kafkaConfig.getInteger(CONSUMER_THREAD_COUNT, 1));
> JavaPairDStream inputStream =
> KafkaUtils.createStream(streamingContext,
> String.class, String.class,
> StringDecoder.class,
> StringDecoder.class,
> kafkaParams,
> topicMap, StorageLevel.MEMORY_AND_DISK_SER_2());
> return inputStream;
> }
>
> I have spark-streaming_2.10-0.9.0-incubating.jar and
> spark-streaming-kafka_2.10-0.9.0-incubating.jar
> in the classpath using POM and m2e in Eclipse. JVM version is set to 1.6
>
> I get the following error,
>
> 14/03/02 09:29:15 INFO kafka.KafkaReceiver: Connected to localhost:2181
> 14/03/02 09:29:15 ERROR kafka.KafkaReceiver: Error receiving data
> java.lang.NoSuchMethodException:
> java.lang.Object.(kafka.utils.VerifiableProperties)
> at java.lang.Class.getConstructor0(Class.java:2763)
> at java.lang.Class.getConstructor(Class.java:1693)
> at
> org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:108)
> at
> org.apache.spark.streaming.dstream.NetworkReceiver.start(NetworkInputDStream.scala:126)
> at
> org.apache.spark.streaming.scheduler.NetworkInputTracker$ReceiverExecutor$$anonfun$8.apply(NetworkInputTracker.scala:173)
> at
> org.apache.spark.streaming.scheduler.NetworkInputTracker$ReceiverExecutor$$anonfun$8.apply(NetworkInputTracker.scala:169)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:884)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:884)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
> at org.apache.spark.scheduler.Task.run(Task.scala:53)
> at .
>
> This is similar to code in JavaKafkaStreamSuite.testKafkaStream. I find that
> the kafka jar - kafka_2.10-0.8.0 does have such a constructor.
>
> What is going wrong? Can someone help solve this mystery and help with my
> misery? Basically stuck for last 2 days - as I am a Java Guy and would like
> to develop downstream code in Java
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: NoSuchMethodError in KafkaReciever
I dont have a Eclipse setup so I am not sure what is going on here. I would
try to use maven in the command line with a pom to see if this compiles.
Also, try to cleanup your system maven cache. Who knows if it had pulled in
a wrong version of kafka 0.8 and using it all the time. Blowing away the
cache and clean compiling will make sure the right kafka will be loaded.
Hope this helps.
TD
On Sat, Mar 1, 2014 at 8:26 PM, venki-kratos wrote:
> I am trying to user code similar to following :
> public JavaPairDStream openStream() {
> HashMap kafkaParams = Maps.newHashMap();
> kafkaParams.put(ZK_CONNECT,kafkaConfig.getString(ZK_CONNECT));
>
> kafkaParams.put(CONSUMER_GRP_ID,kafkaConfig.getString(CONSUMER_GRP_ID));
>
> Map topicMap = Maps.newHashMap();
> topicMap.put(kafkaConfig.getString(ZK_TOPIC),
> kafkaConfig.getInteger(CONSUMER_THREAD_COUNT, 1));
> JavaPairDStream inputStream =
> KafkaUtils.createStream(streamingContext,
> String.class, String.class,
>
> StringDecoder.class,
>
> StringDecoder.class,
> kafkaParams,
> topicMap, StorageLevel.MEMORY_AND_DISK_SER_2());
> return inputStream;
> }
>
> I have spark-streaming_2.10-0.9.0-incubating.jar and
> spark-streaming-kafka_2.10-0.9.0-incubating.jar
> in the classpath using POM and m2e in Eclipse. JVM version is set to 1.6
>
> I get the following error,
>
> 14/03/02 09:29:15 INFO kafka.KafkaReceiver: Connected to localhost:2181
> 14/03/02 09:29:15 ERROR kafka.KafkaReceiver: Error receiving data
> java.lang.NoSuchMethodException:
> java.lang.Object.(kafka.utils.VerifiableProperties)
> at java.lang.Class.getConstructor0(Class.java:2763)
> at java.lang.Class.getConstructor(Class.java:1693)
> at
>
> org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:108)
> at
>
> org.apache.spark.streaming.dstream.NetworkReceiver.start(NetworkInputDStream.scala:126)
> at
>
> org.apache.spark.streaming.scheduler.NetworkInputTracker$ReceiverExecutor$$anonfun$8.apply(NetworkInputTracker.scala:173)
> at
>
> org.apache.spark.streaming.scheduler.NetworkInputTracker$ReceiverExecutor$$anonfun$8.apply(NetworkInputTracker.scala:169)
> at
>
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:884)
> at
>
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:884)
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
> at org.apache.spark.scheduler.Task.run(Task.scala:53)
> at .
>
> This is similar to code in JavaKafkaStreamSuite.testKafkaStream. I find
> that
> the kafka jar - kafka_2.10-0.8.0 does have such a constructor.
>
> What is going wrong? Can someone help solve this mystery and help with my
> misery? Basically stuck for last 2 days - as I am a Java Guy and would like
> to develop downstream code in Java
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
