Andrei Taleanu created SPARK-20321: -------------------------------------- Summary: Spark UI cannot be shutdown in spark streaming app Key: SPARK-20321 URL: https://issues.apache.org/jira/browse/SPARK-20321 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.1.0 Reporter: Andrei Taleanu
When an exception thrown the transform stage is handled in foreachRDD and the streaming context is forced to stop, the SparkUI appears to hang and continually dump the following logs in an infinite loop: {noformat} ... 2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from select, 0/0 selected 2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select 2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from select, 0/0 selected 2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select ... {noformat} Unfortunately I don't have a minimal example that reproduces this issue but here is what I can share: {noformat} val dstream = pull data from kafka val mapped = dstream transform { rdd => val data = getData // Perform a call that potentially throws an exception // broadcast the data // flatMap the RDD using the data } mapped.foreachRDD { try { // write some data in a DB } catch { case t: Throwable => dstream.context.stop(stopSparkContext = true, stopGracefully = false) } } mapped.foreachRDD { try { // write data to Kafka // manually checkpoint the Kafka offsets (because I need them in JSON format) } catch { case t: Throwable => dstream.context.stop(stopSparkContext = true, stopGracefully = false) } } {noformat} The issue appears when stop is invoked. At the point when SparkUI is stopped, it enters that infinite loop. Initially I thought it relates to Jetty, as the version used in SparkUI had some bugs (e.g. [this one|https://bugs.eclipse.org/bugs/show_bug.cgi?id=452465]). I bumped Jetty to a more recent version (March 2017) but still got the error. I encountered this issue with Spark 2.1.0 built with Hadoop 2.6 on top of Mesos. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org