[
https://issues.apache.org/jira/browse/SPARK-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrei Taleanu updated SPARK-20321:
-----------------------------------
Description:
When an exception thrown the transform stage is handled in foreachRDD and the
streaming context is forced to stop, the SparkUI appears to hang and
continually dump the following logs in an infinite loop:
{noformat}
...
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from
select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on
select
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from
select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on
select
...
{noformat}
Unfortunately I don't have a minimal example that reproduces this issue but
here is what I can share:
{noformat}
val dstream = pull data from kafka
val mapped = dstream.transform { rdd =>
val data = getData // Perform a call that potentially throws an exception
// broadcast the data
// flatMap the RDD using the data
}
mapped.foreachRDD {
try {
// write some data in a DB
} catch {
case t: Throwable =>
dstream.context.stop(stopSparkContext = true, stopGracefully = false)
}
}
mapped.foreachRDD {
try {
// write data to Kafka
// manually checkpoint the Kafka offsets (because I need them in JSON
format)
} catch {
case t: Throwable =>
dstream.context.stop(stopSparkContext = true, stopGracefully = false)
}
}
{noformat}
The issue appears when stop is invoked. At the point when SparkUI is stopped,
it enters that infinite loop. Initially I thought it relates to Jetty, as the
version used in SparkUI had some bugs (e.g. [this
one|https://bugs.eclipse.org/bugs/show_bug.cgi?id=452465]). I bumped Jetty to a
more recent version (March 2017) and built Spark 2.1.0 with that one but still
got the error.
I encountered this issue with Spark 2.1.0 built with Hadoop 2.6 on top of Mesos.
was:
When an exception thrown the transform stage is handled in foreachRDD and the
streaming context is forced to stop, the SparkUI appears to hang and
continually dump the following logs in an infinite loop:
{noformat}
...
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from
select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on
select
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from
select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1]
DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on
select
...
{noformat}
Unfortunately I don't have a minimal example that reproduces this issue but
here is what I can share:
{noformat}
val dstream = pull data from kafka
val mapped = dstream transform { rdd =>
val data = getData // Perform a call that potentially throws an exception
// broadcast the data
// flatMap the RDD using the data
}
mapped.foreachRDD {
try {
// write some data in a DB
} catch {
case t: Throwable =>
dstream.context.stop(stopSparkContext = true, stopGracefully = false)
}
}
mapped.foreachRDD {
try {
// write data to Kafka
// manually checkpoint the Kafka offsets (because I need them in JSON
format)
} catch {
case t: Throwable =>
dstream.context.stop(stopSparkContext = true, stopGracefully = false)
}
}
{noformat}
The issue appears when stop is invoked. At the point when SparkUI is stopped,
it enters that infinite loop. Initially I thought it relates to Jetty, as the
version used in SparkUI had some bugs (e.g. [this
one|https://bugs.eclipse.org/bugs/show_bug.cgi?id=452465]). I bumped Jetty to a
more recent version (March 2017) but still got the error.
I encountered this issue with Spark 2.1.0 built with Hadoop 2.6 on top of Mesos.
> Spark UI cannot be shutdown in spark streaming app
> --------------------------------------------------
>
> Key: SPARK-20321
> URL: https://issues.apache.org/jira/browse/SPARK-20321
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Andrei Taleanu
>
> When an exception thrown the transform stage is handled in foreachRDD and the
> streaming context is forced to stop, the SparkUI appears to hang and
> continually dump the following logs in an infinite loop:
> {noformat}
> ...
> 2017-04-12 14:11:13,470
> [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG
> org.spark_project.jetty.io.SelectorManager - Selector loop woken up from
> select, 0/0 selected
> 2017-04-12 14:11:13,470
> [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG
> org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select
> 2017-04-12 14:11:13,470
> [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG
> org.spark_project.jetty.io.SelectorManager - Selector loop woken up from
> select, 0/0 selected
> 2017-04-12 14:11:13,470
> [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG
> org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select
> ...
> {noformat}
> Unfortunately I don't have a minimal example that reproduces this issue but
> here is what I can share:
> {noformat}
> val dstream = pull data from kafka
> val mapped = dstream.transform { rdd =>
> val data = getData // Perform a call that potentially throws an exception
> // broadcast the data
> // flatMap the RDD using the data
> }
> mapped.foreachRDD {
> try {
> // write some data in a DB
> } catch {
> case t: Throwable =>
> dstream.context.stop(stopSparkContext = true, stopGracefully = false)
> }
> }
> mapped.foreachRDD {
> try {
> // write data to Kafka
> // manually checkpoint the Kafka offsets (because I need them in JSON
> format)
> } catch {
> case t: Throwable =>
> dstream.context.stop(stopSparkContext = true, stopGracefully = false)
> }
> }
> {noformat}
> The issue appears when stop is invoked. At the point when SparkUI is stopped,
> it enters that infinite loop. Initially I thought it relates to Jetty, as the
> version used in SparkUI had some bugs (e.g. [this
> one|https://bugs.eclipse.org/bugs/show_bug.cgi?id=452465]). I bumped Jetty to
> a more recent version (March 2017) and built Spark 2.1.0 with that one but
> still got the error.
> I encountered this issue with Spark 2.1.0 built with Hadoop 2.6 on top of
> Mesos.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
