Re: Error Kafka/Spark. Ran out of messages before reaching ending offset
I reinstalled Kafka and it works, I work with virtual machines and someone changed the host of one of the Kafkas without telling anybody. 2016-05-06 16:11 GMT+02:00 Cody Koeninger : > Yeah, so that means the driver talked to kafka and kafka told it the > highest available offset was 2723431. Then when the executor tried to > consume messages, it stopped getting messages before reaching that > offset. That almost certainly means something's wrong with Kafka, > have you looked at your kafka logs? I doubt it's anything to do with > elasticsearch. > > On Fri, May 6, 2016 at 4:22 AM, Guillermo Ortiz > wrote: > > This is the complete error. > > > > 2016-05-06 11:18:05,424 [task-result-getter-0] INFO > > org.apache.spark.scheduler.TaskSetManager - Finished task 5.0 in stage > > 13.0 (TID 60) in 11692 ms on xx (6/8) > > 2016-05-06 11:18:08,978 [task-result-getter-1] WARN > > org.apache.spark.scheduler.TaskSetManager - Lost task 7.0 in stage > > 13.0 (TID 62, xxx): java.lang.AssertionError: assertion > > failed: Ran out of messages before reaching ending offset 2723431 for > > topic kafka-global-paas partition 2 start 2705506. This should not > > happen, and indicates that messages may have been lost > > at scala.Predef$.assert(Predef.scala:179) > > at > org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:211) > > at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) > > at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) > > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > > at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:48) > > at > org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:67) > > at > org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:67) > > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > > at org.apache.spark.scheduler.Task.run(Task.scala:88) > > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > > > 2016-05-06 11:18:08,982 [sparkDriver-akka.actor.default-dispatcher-18] > > INFO org.apache.spark.scheduler.TaskSetManager - Starting task 7.1 in > > stage 13.0 (TID 63, , partition 7,RACK_LOCAL, 2052 bytes) > > 2016-05-06 11:18:10,013 [JobGenerator] INFO > > org.apache.spark.streaming.scheduler.JobScheduler - Added jobs for > > time 146252629 ms > > 2016-05-06 11:18:10,015 [JobGenerator] INFO > > org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing > > graph for time 146252629 ms > > > > 2016-05-06 11:11 GMT+02:00 Guillermo Ortiz : > >> I think that it's a kafka error, but I'm starting thinking if it could > >> be something about elasticsearch since I have seen more people with > >> same error using elasticsearch. I have no idea. > >> > >> 2016-05-06 11:05 GMT+02:00 Guillermo Ortiz : > >>> I'm trying to read data from Spark and index to ES with its library > >>> (es-hadoop 2.2.1 version). > >>> IIt was working right for a while but now it has started to happen > this error. > >>> I have delete the checkpoint and even the kafka topic and restart all > >>> the machines with kafka and zookeeper but it didn't fix it. > >>> > >>> User class threw exception: org.apache.spark.SparkException: Job > >>> aborted due to stage failure: Task 4 in stage 1.0 failed 4 times, most > >>> recent failure: Lost task 4.3 in stage 1.0 (TID 12, x): > >>> java.lang.AssertionError: assertion failed: Ran out of messages before > >>> reaching ending offset 1226116 for topic kafka-global-paas partition 7 > >>> start 1212156. This should not happen, and indicates that messages may > >>> have been lost > >>> at scala.Predef$.assert(Predef.scala:179) > >>> at > org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:211) > >>> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) > >>> at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) > >>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > >>> > >>> I read some threads with this error but it didn't help me. > > > > - > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > >
Re: Error Kafka/Spark. Ran out of messages before reaching ending offset
Yeah, so that means the driver talked to kafka and kafka told it the highest available offset was 2723431. Then when the executor tried to consume messages, it stopped getting messages before reaching that offset. That almost certainly means something's wrong with Kafka, have you looked at your kafka logs? I doubt it's anything to do with elasticsearch. On Fri, May 6, 2016 at 4:22 AM, Guillermo Ortiz wrote: > This is the complete error. > > 2016-05-06 11:18:05,424 [task-result-getter-0] INFO > org.apache.spark.scheduler.TaskSetManager - Finished task 5.0 in stage > 13.0 (TID 60) in 11692 ms on xx (6/8) > 2016-05-06 11:18:08,978 [task-result-getter-1] WARN > org.apache.spark.scheduler.TaskSetManager - Lost task 7.0 in stage > 13.0 (TID 62, xxx): java.lang.AssertionError: assertion > failed: Ran out of messages before reaching ending offset 2723431 for > topic kafka-global-paas partition 2 start 2705506. This should not > happen, and indicates that messages may have been lost > at scala.Predef$.assert(Predef.scala:179) > at > org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:211) > at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) > at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:48) > at > org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:67) > at > org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:67) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:88) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > 2016-05-06 11:18:08,982 [sparkDriver-akka.actor.default-dispatcher-18] > INFO org.apache.spark.scheduler.TaskSetManager - Starting task 7.1 in > stage 13.0 (TID 63, , partition 7,RACK_LOCAL, 2052 bytes) > 2016-05-06 11:18:10,013 [JobGenerator] INFO > org.apache.spark.streaming.scheduler.JobScheduler - Added jobs for > time 146252629 ms > 2016-05-06 11:18:10,015 [JobGenerator] INFO > org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing > graph for time 146252629 ms > > 2016-05-06 11:11 GMT+02:00 Guillermo Ortiz : >> I think that it's a kafka error, but I'm starting thinking if it could >> be something about elasticsearch since I have seen more people with >> same error using elasticsearch. I have no idea. >> >> 2016-05-06 11:05 GMT+02:00 Guillermo Ortiz : >>> I'm trying to read data from Spark and index to ES with its library >>> (es-hadoop 2.2.1 version). >>> IIt was working right for a while but now it has started to happen this >>> error. >>> I have delete the checkpoint and even the kafka topic and restart all >>> the machines with kafka and zookeeper but it didn't fix it. >>> >>> User class threw exception: org.apache.spark.SparkException: Job >>> aborted due to stage failure: Task 4 in stage 1.0 failed 4 times, most >>> recent failure: Lost task 4.3 in stage 1.0 (TID 12, x): >>> java.lang.AssertionError: assertion failed: Ran out of messages before >>> reaching ending offset 1226116 for topic kafka-global-paas partition 7 >>> start 1212156. This should not happen, and indicates that messages may >>> have been lost >>> at scala.Predef$.assert(Predef.scala:179) >>> at >>> org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:211) >>> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) >>> at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) >>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) >>> >>> I read some threads with this error but it didn't help me. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Error Kafka/Spark. Ran out of messages before reaching ending offset
This is the complete error. 2016-05-06 11:18:05,424 [task-result-getter-0] INFO org.apache.spark.scheduler.TaskSetManager - Finished task 5.0 in stage 13.0 (TID 60) in 11692 ms on xx (6/8) 2016-05-06 11:18:08,978 [task-result-getter-1] WARN org.apache.spark.scheduler.TaskSetManager - Lost task 7.0 in stage 13.0 (TID 62, xxx): java.lang.AssertionError: assertion failed: Ran out of messages before reaching ending offset 2723431 for topic kafka-global-paas partition 2 start 2705506. This should not happen, and indicates that messages may have been lost at scala.Predef$.assert(Predef.scala:179) at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:211) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:48) at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:67) at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:67) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-05-06 11:18:08,982 [sparkDriver-akka.actor.default-dispatcher-18] INFO org.apache.spark.scheduler.TaskSetManager - Starting task 7.1 in stage 13.0 (TID 63, , partition 7,RACK_LOCAL, 2052 bytes) 2016-05-06 11:18:10,013 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobScheduler - Added jobs for time 146252629 ms 2016-05-06 11:18:10,015 [JobGenerator] INFO org.apache.spark.streaming.scheduler.JobGenerator - Checkpointing graph for time 146252629 ms 2016-05-06 11:11 GMT+02:00 Guillermo Ortiz : > I think that it's a kafka error, but I'm starting thinking if it could > be something about elasticsearch since I have seen more people with > same error using elasticsearch. I have no idea. > > 2016-05-06 11:05 GMT+02:00 Guillermo Ortiz : >> I'm trying to read data from Spark and index to ES with its library >> (es-hadoop 2.2.1 version). >> IIt was working right for a while but now it has started to happen this >> error. >> I have delete the checkpoint and even the kafka topic and restart all >> the machines with kafka and zookeeper but it didn't fix it. >> >> User class threw exception: org.apache.spark.SparkException: Job >> aborted due to stage failure: Task 4 in stage 1.0 failed 4 times, most >> recent failure: Lost task 4.3 in stage 1.0 (TID 12, x): >> java.lang.AssertionError: assertion failed: Ran out of messages before >> reaching ending offset 1226116 for topic kafka-global-paas partition 7 >> start 1212156. This should not happen, and indicates that messages may >> have been lost >> at scala.Predef$.assert(Predef.scala:179) >> at >> org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:211) >> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) >> at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) >> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) >> >> I read some threads with this error but it didn't help me. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Error Kafka/Spark. Ran out of messages before reaching ending offset
I think that it's a kafka error, but I'm starting thinking if it could be something about elasticsearch since I have seen more people with same error using elasticsearch. I have no idea. 2016-05-06 11:05 GMT+02:00 Guillermo Ortiz : > I'm trying to read data from Spark and index to ES with its library > (es-hadoop 2.2.1 version). > IIt was working right for a while but now it has started to happen this error. > I have delete the checkpoint and even the kafka topic and restart all > the machines with kafka and zookeeper but it didn't fix it. > > User class threw exception: org.apache.spark.SparkException: Job > aborted due to stage failure: Task 4 in stage 1.0 failed 4 times, most > recent failure: Lost task 4.3 in stage 1.0 (TID 12, x): > java.lang.AssertionError: assertion failed: Ran out of messages before > reaching ending offset 1226116 for topic kafka-global-paas partition 7 > start 1212156. This should not happen, and indicates that messages may > have been lost > at scala.Predef$.assert(Predef.scala:179) > at > org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:211) > at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) > at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > > I read some threads with this error but it didn't help me. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Error Kafka/Spark. Ran out of messages before reaching ending offset
I'm trying to read data from Spark and index to ES with its library (es-hadoop 2.2.1 version). IIt was working right for a while but now it has started to happen this error. I have delete the checkpoint and even the kafka topic and restart all the machines with kafka and zookeeper but it didn't fix it. User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 1.0 failed 4 times, most recent failure: Lost task 4.3 in stage 1.0 (TID 12, x): java.lang.AssertionError: assertion failed: Ran out of messages before reaching ending offset 1226116 for topic kafka-global-paas partition 7 start 1212156. This should not happen, and indicates that messages may have been lost at scala.Predef$.assert(Predef.scala:179) at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:211) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) I read some threads with this error but it didn't help me. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org