sheng created SPARK-8333:
----------------------------
Summary: Spark failed to delete temp directory created by
HiveContext
Key: SPARK-8333
URL: https://issues.apache.org/jira/browse/SPARK-8333
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.4.0
Environment: Windows7 64bit
Reporter: sheng
Priority: Minor
Spark 1.4.0 failed to stop SparkContext.
{code:title=LocalHiveTest.scala|borderStyle=solid}
val sc = new SparkContext("local", "local-hive-test", new SparkConf())
val hc = Utils.createHiveContext(sc)
... // execute some HiveQL statements
sc.stop()
{code}
sc.stop() failed to execute, it threw the following exception:
{quote}
15/06/13 03:19:06 INFO Utils: Shutdown hook called
15/06/13 03:19:06 INFO Utils: Deleting directory
C:\Users\moshangcheng\AppData\Local\Temp\spark-d6d3c30e-512e-4693-a436-485e2af4baea
15/06/13 03:19:06 ERROR Utils: Exception while deleting Spark temp dir:
C:\Users\moshangcheng\AppData\Local\Temp\spark-d6d3c30e-512e-4693-a436-485e2af4baea
java.io.IOException: Failed to delete:
C:\Users\moshangcheng\AppData\Local\Temp\spark-d6d3c30e-512e-4693-a436-485e2af4baea
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:963)
at
org.apache.spark.util.Utils$$anonfun$1$$anonfun$apply$mcV$sp$5.apply(Utils.scala:204)
at
org.apache.spark.util.Utils$$anonfun$1$$anonfun$apply$mcV$sp$5.apply(Utils.scala:201)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.spark.util.Utils$$anonfun$1.apply$mcV$sp(Utils.scala:201)
at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2292)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Utils.scala:2262)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(Utils.scala:2262)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(Utils.scala:2262)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(Utils.scala:2262)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(Utils.scala:2262)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(Utils.scala:2262)
at scala.util.Try$.apply(Try.scala:161)
at
org.apache.spark.util.SparkShutdownHookManager.runAll(Utils.scala:2262)
at
org.apache.spark.util.SparkShutdownHookManager$$anon$6.run(Utils.scala:2244)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{quote}
It seems this bug is introduced by this SPARK-6907. In SPARK-6907, a local hive
metastore is created in a temp directory. The problem is the local hive
metastore is not shut down correctly. At the end of application, if
SparkContext.stop() is called, it tries to delete the temp directory which is
still used by the local hive metastore, and throws an exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
