[ https://issues.apache.org/jira/browse/SPARK-25404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16613369#comment-16613369 ]
Apache Spark commented on SPARK-25404: -------------------------------------- User 'fjh100456' has created a pull request for this issue: https://github.com/apache/spark/pull/22412 > Staging path may not on the expected place when table path contains the > stagingDir string > ----------------------------------------------------------------------------------------- > > Key: SPARK-25404 > URL: https://issues.apache.org/jira/browse/SPARK-25404 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.1 > Reporter: Jinhua Fu > Priority: Minor > > Considering the follow scenario: > > {code:java} > SET hive.exec.stagingdir=temp; > CREATE TABLE tempTableA(key int) location '/spark/temp/tempTableA'; > INSERT OVERWRITE TABLE tempTableA SELECT 1; > {code} > We expect the staging path under the table path, such as > '/spark/temp/tempTableA/.hive-stagingXXX'(SPARK-20594), but actually it is > '/spark/tempXXX'. > I'm not quite sure why we use the 'if ... else ...' when getting a > stagingDir, but it maybe the cause of this bug. > > {code:java} > // SaveAsHiveFile.scala > private def getStagingDir( > inputPath: Path, > hadoopConf: Configuration, > stagingDir: String): Path = { > ...... > var stagingPathName: String = > if (inputPathName.indexOf(stagingDir) == -1) { > new Path(inputPathName, stagingDir).toString > } else { > // The 'indexOf' may not get expected position, and this may be the cause > of this bug. > inputPathName.substring(0, inputPathName.indexOf(stagingDir) + > stagingDir.length) > } > ...... > } > {code} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org