ajithme commented on a change in pull request #27039: [SPARK-29174][SQL] Support LOCAL in INSERT OVERWRITE DIRECTORY to data source URL: https://github.com/apache/spark/pull/27039#discussion_r367787109
########## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ########## @@ -809,7 +805,12 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder(conf) { } if (!path.isEmpty) { - val customLocation = Some(CatalogUtils.stringToURI(path)) + val customLocation = if (ctx.LOCAL() != null) { + // force scheme to be file rather than fs.default.name Review comment: Sure, i checked in Hive 2.3.5, it throws a exception if any other scheme is used apart form `file:` #### success scenario ``` 0: jdbc:hive2://QWE:10000/> insert overwrite local directory 'file:/tmp/sample' select 1; No rows affected (1.436 seconds) 0: jdbc:hive2://QWE:10000/> insert overwrite local directory '/tmp/sample' select 1; No rows affected (1.474 seconds) ``` #### fail scenario Below is the stack from hive server on client ``` 0: jdbc:hive2://QWE:10000/> insert overwrite local directory 'hdfs:///tmp/sample' select 1; WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source file:/tmp/root/28572e2b-63d0-49fd-9938-85b9741c7413/hive_2020-01-17_14-32-02_389_528060968304222996-1/-mr-10000 to destination hdfs:/tmp/sample at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257) at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source file:/tmp/root/28572e2b-63d0-49fd-9938-85b9741c7413/hive_2020-01-17_14-32-02_389_528060968304222996-1/-mr-10000 to destination hdfs:/tmp/sample at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:104) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:259) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1232) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:255) ... 11 more Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs:/tmp/sample, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:665) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86) at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:630) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:861) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:625) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1437) at org.apache.hadoop.hive.ql.exec.MoveTask.moveFileFromDfsToLocal(MoveTask.java:143) at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:101) ... 20 more (state=08S01,code=1) ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org