[ https://issues.apache.org/jira/browse/SPARK-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kousuke Saruta updated SPARK-2970: ---------------------------------- Priority: Critical (was: Major) > spark-sql script ends with IOException when EventLogging is enabled > ------------------------------------------------------------------- > > Key: SPARK-2970 > URL: https://issues.apache.org/jira/browse/SPARK-2970 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.1.0 > Environment: CDH5.1.0 (Hadoop 2.3.0) > Reporter: Kousuke Saruta > Priority: Critical > Fix For: 1.1.0 > > > When spark-sql script run with spark.eventLog.enabled set true, it ends with > IOException because FileLogger can not create APPLICATION_COMPLETE file in > HDFS. > It's is because a shutdown hook of SparkSQLCLIDriver is executed after a > shutdown hook of org.apache.hadoop.fs.FileSystem is executed. > When spark.eventLog.enabled is true, the hook of SparkSQLCLIDriver finally > try to create a file to mark the application finished but the hook of > FileSystem try to close FileSystem. -- This message was sent by Atlassian JIRA (v6.2#6252) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org