I don't see how that would be possible. I am reading from a live stream of data through kafka.
On Sat 12 Mar, 2016 20:28 Ted Yu, <yuzhih...@gmail.com> wrote: > Interesting. > If kv._1 was null, shouldn't the NPE have come from getPartition() (line > 105) ? > > Was it possible that records.next() returned null ? > > On Fri, Mar 11, 2016 at 11:20 PM, Prabhu Joseph < > prabhujose.ga...@gmail.com> wrote: > >> Looking at ExternalSorter.scala line 192, i suspect some input record has >> Null key. >> >> 189 while (records.hasNext) { >> 190 addElementsRead() >> 191 kv = records.next() >> 192 map.changeValue((getPartition(kv._1), kv._1), update) >> >> >> >> On Sat, Mar 12, 2016 at 12:48 PM, Prabhu Joseph < >> prabhujose.ga...@gmail.com> wrote: >> >>> Looking at ExternalSorter.scala line 192 >>> >>> 189 >>> while (records.hasNext) { addElementsRead() kv = records.next() >>> map.changeValue((getPartition(kv._1), kv._1), update) >>> maybeSpillCollection(usingMap = true) } >>> >>> On Sat, Mar 12, 2016 at 12:31 PM, Saurabh Guru <saurabh.g...@gmail.com> >>> wrote: >>> >>>> I am seeing the following exception in my Spark Cluster every few days >>>> in production. >>>> >>>> 2016-03-12 05:30:00,541 - WARN TaskSetManager - Lost task 0.0 in stage >>>> 12528.0 (TID 18792, ip-1X-1XX-1-1XX.us <http://ip-10-180-1-188.us> >>>> -west-1.compute.internal >>>> ): java.lang.NullPointerException >>>> at >>>> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:192) >>>> at >>>> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) >>>> at >>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) >>>> at >>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >>>> at org.apache.spark.scheduler.Task.run(Task.scala:89) >>>> at >>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> >>>> I have debugged in local machine but haven’t been able to pin point the >>>> cause of the error. Anyone knows why this might occur? Any suggestions? >>>> >>>> >>>> Thanks, >>>> Saurabh >>>> >>>> >>>> >>>> >>> >> >