[ https://issues.apache.org/jira/browse/SPARK-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14681886#comment-14681886 ]
Cody Koeninger commented on SPARK-9780: --------------------------------------- Makes sense, traveling currently but I'll put in a PR > In case of invalid initialization of KafkaDirectStream, NPE is thrown > --------------------------------------------------------------------- > > Key: SPARK-9780 > URL: https://issues.apache.org/jira/browse/SPARK-9780 > Project: Spark > Issue Type: Bug > Components: Streaming > Affects Versions: 1.3.1, 1.4.1 > Reporter: Grigory Turunov > Priority: Minor > > [o.a.s.streaming.kafka.KafkaRDD.scala#L143|https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaRDD.scala#L143] > In initialization of KafkaRDDIterator, there is an addition of > TaskCompletionListener to the context, which calls close() to the consumer, > which is not initialized yet (and will be initialized 12 lines after that). > If something happens in this 12 lines (in my case there was a private > constructor for valueDecoder), an Exception, which is thrown, triggers > context.markTaskCompleted() in > [o.a.s.scheduler.Task.scala#L90|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/Task.scala#L90] > which throws NullPointerException, when tries to call close() for > non-initialized consumer. > This masks original exception - so it is very hard to understand, what is > happening. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org