[ https://issues.apache.org/jira/browse/SPARK-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated SPARK-2034: ----------------------------------- Assignee: Sean Owen > KafkaInputDStream doesn't close resources and may prevent JVM shutdown > ---------------------------------------------------------------------- > > Key: SPARK-2034 > URL: https://issues.apache.org/jira/browse/SPARK-2034 > Project: Spark > Issue Type: Bug > Components: Streaming > Affects Versions: 1.0.0 > Reporter: Sean Owen > Assignee: Sean Owen > Fix For: 1.0.1, 1.1.0 > > > Tobias noted today on the mailing list: > {quote} > I am trying to use Spark Streaming with Kafka, which works like a > charm -- except for shutdown. When I run my program with "sbt > run-main", sbt will never exit, because there are two non-daemon > threads left that don't die. > I created a minimal example at > <https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-kafkadoesntshutdown-scala>. > It starts a StreamingContext and does nothing more than connecting to > a Kafka server and printing what it receives. Using the `future { ... > }` construct, I shut down the StreamingContext after some seconds and > then print the difference between the threads at start time and at end > time. The output can be found at > <https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-output1>. > There are a number of threads remaining that will prevent sbt from > exiting. > When I replace `KafkaUtils.createStream(...)` with a call that does > exactly the same, except that it calls `consumerConnector.shutdown()` > in `KafkaReceiver.onStop()` (which it should, IMO), the output is as > shown at > <https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-output2>. > Does anyone have *any* idea what is going on here and why the program > doesn't shut down properly? The behavior is the same with both kafka > 0.8.0 and 0.8.1.1, by the way. > {quote} > Something similar was noted last year: > http://mail-archives.apache.org/mod_mbox/spark-dev/201309.mbox/%3c1380220041.2428.yahoomail...@web160804.mail.bf1.yahoo.com%3E > > KafkaInputDStream doesn't close ConsumerConnector in onStop(), and does not > close the Executor it creates. The latter leaves non-daemon threads and can > prevent the JVM from shutting down even if streaming is closed properly. -- This message was sent by Atlassian JIRA (v6.2#6252)