[
https://issues.apache.org/jira/browse/SPARK-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963688#comment-13963688
]
Kan Zhang commented on SPARK-1407:
----------------------------------
I encountered similar issue. It is not limited to YARN, but shows up in
Standalone mode as well. When System.exit is invoked without calling sc.stop(),
Hadoop FileSystem's shutdown hook gets called, which closes fs without flushing
client buffer and event log file got truncated. Calling sc.stop() seems to work
in my test, which does flush hadoopDataStream buffer before closing. However,
this may not completely solve the problem as events are written to buffer
asynchronously by SparkListenerBus thread.
> EventLogging to HDFS doesn't work properly on yarn
> --------------------------------------------------
>
> Key: SPARK-1407
> URL: https://issues.apache.org/jira/browse/SPARK-1407
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
>
> When running on spark on yarn and accessing an HDFS file (like in the
> SparkHdfsLR example) while using the event logging configured to write logs
> to HDFS, it throws an exception at the end of the application.
> SPARK_JAVA_OPTS=-Dspark.eventLog.enabled=true
> -Dspark.eventLog.dir=hdfs:///history/spark/
> 14/04/03 13:41:31 INFO yarn.ApplicationMaster$$anon$1: Invoking sc stop from
> shutdown hook
> Exception in thread "Thread-41" java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:398)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1465)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1450)
> at
> org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:116)
> at
> org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:137)
> at
> org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:137)
> at scala.Option.foreach(Option.scala:236)
> at org.apache.spark.util.FileLogger.flush(FileLogger.scala:137)
> at
> org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:69)
> at
> org.apache.spark.scheduler.EventLoggingListener.onApplicationEnd(EventLoggingListener.scala:101)
> at
> org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$13.apply(SparkListenerBus.scala:67)
> at
> org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$13.apply(SparkListenerBus.scala:67)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.SparkListenerBus$class.postToAll(SparkListenerBus.scala:67)
> at
> org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:31)
> at
> org.apache.spark.scheduler.LiveListenerBus.post(LiveListenerBus.scala:78)
> at
> org.apache.spark.SparkContext.postApplicationEnd(SparkContext.scala:1081)
> at org.apache.spark.SparkContext.stop(SparkContext.scala:828)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:460)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
