[jira] [Updated] (SPARK-9844) File appender race condition during SparkWorker shutdown
[ https://issues.apache.org/jira/browse/SPARK-9844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-9844: --- Priority: Major (was: Minor) > File appender race condition during SparkWorker shutdown > > > Key: SPARK-9844 > URL: https://issues.apache.org/jira/browse/SPARK-9844 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.3.0, 1.4.0 >Reporter: Alex Liu > > We find this issue still exists in 1.3.1 > {code} > ERROR [Thread-6] 2015-07-28 22:49:57,653 SparkWorker-0 ExternalLogger.java:96 > - Error writing stream to file > /var/lib/spark/worker/worker-0/app-20150728224954-0003/0/stderr > ERROR [Thread-6] 2015-07-28 22:49:57,653 SparkWorker-0 ExternalLogger.java:96 > - java.io.IOException: Stream closed > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at > java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at java.io.BufferedInputStream.read1(BufferedInputStream.java:283) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at java.io.FilterInputStream.read(FilterInputStream.java:107) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70) > ~[spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1618) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,656 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > {code} > at > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala#L159 > The process auto shuts down, but the log appenders are still running, which > causes the error log messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-9844) File appender race condition during SparkWorker shutdown
[ https://issues.apache.org/jira/browse/SPARK-9844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-9844: - Assignee: Bryan Cutler > File appender race condition during SparkWorker shutdown > > > Key: SPARK-9844 > URL: https://issues.apache.org/jira/browse/SPARK-9844 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.3.0, 1.4.0 >Reporter: Alex Liu >Assignee: Bryan Cutler > Fix For: 1.6.1, 2.0.0 > > > We find this issue still exists in 1.3.1 > {code} > ERROR [Thread-6] 2015-07-28 22:49:57,653 SparkWorker-0 ExternalLogger.java:96 > - Error writing stream to file > /var/lib/spark/worker/worker-0/app-20150728224954-0003/0/stderr > ERROR [Thread-6] 2015-07-28 22:49:57,653 SparkWorker-0 ExternalLogger.java:96 > - java.io.IOException: Stream closed > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at > java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at java.io.BufferedInputStream.read1(BufferedInputStream.java:283) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at java.io.FilterInputStream.read(FilterInputStream.java:107) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70) > ~[spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1618) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,656 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > {code} > at > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala#L159 > The process auto shuts down, but the log appenders are still running, which > causes the error log messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-9844) File appender race condition during SparkWorker shutdown
[ https://issues.apache.org/jira/browse/SPARK-9844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Liu updated SPARK-9844: Description: We find this issue still exists in 1.3.1 {code} ERROR [Thread-6] 2015-07-28 22:49:57,653 SparkWorker-0 ExternalLogger.java:96 - Error writing stream to file /var/lib/spark/worker/worker-0/app-20150728224954-0003/0/stderr ERROR [Thread-6] 2015-07-28 22:49:57,653 SparkWorker-0 ExternalLogger.java:96 - java.io.IOException: Stream closed ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 - at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) ~[na:1.8.0_40] ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 - at java.io.BufferedInputStream.read1(BufferedInputStream.java:283) ~[na:1.8.0_40] ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 - at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_40] ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 - at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[na:1.8.0_40] ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 - at org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70) ~[spark-core_2.10-1.3.1.1.jar:1.3.1.1] ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 - at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39) [spark-core_2.10-1.3.1.1.jar:1.3.1.1] ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 - at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39) [spark-core_2.10-1.3.1.1.jar:1.3.1.1] ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 - at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39) [spark-core_2.10-1.3.1.1.jar:1.3.1.1] ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 - at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1618) [spark-core_2.10-1.3.1.1.jar:1.3.1.1] ERROR [Thread-6] 2015-07-28 22:49:57,656 SparkWorker-0 ExternalLogger.java:96 - at org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38) [spark-core_2.10-1.3.1.1.jar:1.3.1.1] {code} at https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala#L159 The process auto shuts down, but the log appenders are still running, which causes the error log messages. > File appender race condition during SparkWorker shutdown > > > Key: SPARK-9844 > URL: https://issues.apache.org/jira/browse/SPARK-9844 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.3.0, 1.4.0 >Reporter: Alex Liu >Priority: Minor > > We find this issue still exists in 1.3.1 > {code} > ERROR [Thread-6] 2015-07-28 22:49:57,653 SparkWorker-0 ExternalLogger.java:96 > - Error writing stream to file > /var/lib/spark/worker/worker-0/app-20150728224954-0003/0/stderr > ERROR [Thread-6] 2015-07-28 22:49:57,653 SparkWorker-0 ExternalLogger.java:96 > - java.io.IOException: Stream closed > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at > java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at java.io.BufferedInputStream.read1(BufferedInputStream.java:283) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,654 SparkWorker-0 ExternalLogger.java:96 > - at java.io.FilterInputStream.read(FilterInputStream.java:107) > ~[na:1.8.0_40] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70) > ~[spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22:49:57,655 SparkWorker-0 ExternalLogger.java:96 > - at > org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39) > [spark-core_2.10-1.3.1.1.jar:1.3.1.1] > ERROR [Thread-6] 2015-07-28 22