[ https://issues.apache.org/jira/browse/SPARK-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967514#comment-15967514 ]
Andrei Taleanu commented on SPARK-20321: ---------------------------------------- Not sure what it would help. I could provide more logs, but I'm not sure how that helps as they're typical stopping logs. Also, the SparkUI remains accessible when this happens and you can see a single job that runs forever. I've tried recreating a minimal example but somehow I can't reproduce it in another minimal streaming app. In the app where I've encountered this issue it happens every time so it's really easy to reproduce. I'm not sure what I should be looking at so I can provide more details. > Spark UI cannot be shutdown in spark streaming app > -------------------------------------------------- > > Key: SPARK-20321 > URL: https://issues.apache.org/jira/browse/SPARK-20321 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.1.0 > Reporter: Andrei Taleanu > > When an exception thrown the transform stage is handled in foreachRDD and the > streaming context is forced to stop, the SparkUI appears to hang and > continually dump the following logs in an infinite loop: > {noformat} > ... > 2017-04-12 14:11:13,470 > [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG > org.spark_project.jetty.io.SelectorManager - Selector loop woken up from > select, 0/0 selected > 2017-04-12 14:11:13,470 > [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG > org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select > 2017-04-12 14:11:13,470 > [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG > org.spark_project.jetty.io.SelectorManager - Selector loop woken up from > select, 0/0 selected > 2017-04-12 14:11:13,470 > [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG > org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select > ... > {noformat} > Unfortunately I don't have a minimal example that reproduces this issue but > here is what I can share: > {noformat} > val dstream = pull data from kafka > val mapped = dstream.transform { rdd => > val data = getData // Perform a call that potentially throws an exception > // broadcast the data > // flatMap the RDD using the data > } > mapped.foreachRDD { > try { > // write some data in a DB > } catch { > case t: Throwable => > dstream.context.stop(stopSparkContext = true, stopGracefully = false) > } > } > mapped.foreachRDD { > try { > // write data to Kafka > // manually checkpoint the Kafka offsets (because I need them in JSON > format) > } catch { > case t: Throwable => > dstream.context.stop(stopSparkContext = true, stopGracefully = false) > } > } > {noformat} > The issue appears when stop is invoked. At the point when SparkUI is stopped, > it enters that infinite loop. Initially I thought it relates to Jetty, as the > version used in SparkUI had some bugs (e.g. [this > one|https://bugs.eclipse.org/bugs/show_bug.cgi?id=452465]). I bumped Jetty to > a more recent version (March 2017) and built Spark 2.1.0 with that one but > still got the error. > I encountered this issue with Spark 2.1.0 built with Hadoop 2.6 on top of > Mesos. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org