That error indicates a message bigger than the buffer's capacity https://issues.apache.org/jira/browse/KAFKA-1196
On Tue, Apr 26, 2016 at 3:07 AM, Michel Hubert <mich...@phact.nl> wrote: > Hi, > > > > > > I use a Kafka direct stream approach. > > My Spark application was running ok. > > This morning we upgraded to CDH 5.7.0 > > And when I re-started my Spark application I get exceptions. > > > > It seems a problem with the direct stream approach. > > Any ideas how to fix this? > > > > > > > > User class threw exception: org.apache.spark.SparkException: Job aborted due > to stage failure: Task 3 in stage 0.0 failed 4 times, most recent failure: > Lost task 3.3 in stage 0.0 (TID 26, bfravicsvr81439-cld.opentsp.com): > java.lang.IllegalArgumentException > > at java.nio.Buffer.limit(Buffer.java:267) > > at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38) > > at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100) > > at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98) > > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > > at scala.collection.immutable.Range.foreach(Range.scala:141) > > at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > > at scala.collection.AbstractTraversable.map(Traversable.scala:105) > > at kafka.api.TopicData$.readFrom(FetchResponse.scala:98) > > at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170) > > at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169) > > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > > at scala.collection.immutable.Range.foreach(Range.scala:141) > > at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) > > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) > > at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169) > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135) > > at > org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:192) > > at > org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:208) > > at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) > > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > > at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) > > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > > at > org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126) > > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) > > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > > at org.apache.spark.scheduler.Task.run(Task.scala:89) > > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
