Grigory Turunov created SPARK-9780: -------------------------------------- Summary: In case of invalid initialization of KafkaDirectStream, NPE is thrown Key: SPARK-9780 URL: https://issues.apache.org/jira/browse/SPARK-9780 Project: Spark Issue Type: Bug Components: Streaming Affects Versions: 1.4.1, 1.3.1 Reporter: Grigory Turunov Priority: Minor
[o.a.s.streaming.kafka.KafkaRDD.scala#L143|https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaRDD.scala#L143] In initialization of KafkaRDDIterator, there is an addition of TaskCompletionListener to the context, which calls close() to the consumer, which is not initialized yet (and will be initialized 12 lines after that). If something happens in this 12 lines (in my case there was a private constructor for valueDecoder), an Exception, which is thrown, triggers context.markTaskCompleted() in [o.a.s.scheduler.Task.scala#L90|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/Task.scala#L90] which throws NullPointerException, when tries to call close() for non-initialized consumer. This masks original exception - so it is very hard to understand, what is happening. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org