[ https://issues.apache.org/jira/browse/SPARK-20594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li resolved SPARK-20594. ----------------------------- Resolution: Fixed Assignee: zuotingbing Fix Version/s: 2.2.0 > The staging directory should be appended with ".hive-staging" to avoid being > deleted if we set hive.exec.stagingdir under the table directory without > start with "." > -------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-20594 > URL: https://issues.apache.org/jira/browse/SPARK-20594 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.2, 2.1.0 > Reporter: zuotingbing > Assignee: zuotingbing > Fix For: 2.2.0 > > > The staging directory should be appended with ".hive-staging" to avoid being > deleted when we set hive.exec.stagingdir under the table directory without > start with "." > spark-sql> set hive.exec.stagingdir=./test; > spark-sql> insert overwrite table test_table1 select * from test_table; > we got error as follows: > 2017-05-04 15:21:06,948 INFO org.apache.hadoop.hive.common.FileUtils: > deleting > hdfs://nameservice/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1 > 2017-05-04 15:21:06,987 INFO org.apache.hadoop.fs.TrashPolicyDefault: Moved: > 'hdfs://nameservice/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1' > to trash at: > hdfs://nameservice/user/mr/.Trash/Current/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1 > 2017-05-04 15:21:06,987 INFO org.apache.hadoop.hive.common.FileUtils: Moved > to trash: > hdfs://nameservice/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1 > 2017-05-04 15:21:07,001 ERROR org.apache.hadoop.hdfs.KeyProviderCache: Could > not find uri with key [dfs.encryption.key.provider.uri] to create a > keyProvider !! > 2017-05-04 15:21:07,007 INFO hive.ql.metadata.Hive: Replacing > src:hdfs://nameservice/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1/-ext-10000/part-00000, > dest: hdfs://nameservice/spark/ztb.db/test_table1/part-00000, Status:false > 2017-05-04 15:21:07,024 ERROR > org.apache.spark.sql.hive.thriftserver.SparkSQLDriver: Failed in [insert > overwrite table test_table1 select * from test_table] > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:633) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply$mcV$sp(HiveClientImpl.scala:646) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:646) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:646) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:280) > at > org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:227) > at > org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:226) > at > org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:269) > at > org.apache.spark.sql.hive.client.HiveClientImpl.loadTable(HiveClientImpl.scala:645) > at > org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:290) > at > org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:143) > at > org.apache.spark.sql.hive.execution.InsertIntoHiveTable.executeCollect(InsertIntoHiveTable.scala:308) > at org.apache.spark.sql.Dataset.<init>(Dataset.scala:186) > at org.apache.spark.sql.Dataset.<init>(Dataset.scala:167) > at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:65) > at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:601) > at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:682) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:331) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:247) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move > file/directory from > hdfs://nameservice/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1/-ext-10000/part-00000 > to hdfs://nameservice/spark/ztb.db/test_table1/part-00000 > at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2899) > at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1589) -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org