Kent Yao created SPARK-31684: -------------------------------- Summary: Overwrite partition failed with 'WRONG FS' when the target partition is not belong to the filesystem as same as the table Key: SPARK-31684 URL: https://issues.apache.org/jira/browse/SPARK-31684 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.4.5, 2.3.4, 2.2.3, 2.1.3, 3.0.0, 3.1.0 Reporter: Kent Yao
With https://issues.apache.org/jira/browse/SPARK-18107, we conditionally disable the underlying replace(overwrite) and instead do delete in spark side and only do copy in hive side to bypass the performance issue - https://issues.apache.org/jira/browse/HIVE-11940 Additionally, if the table location and partition location do not belong to the same [[FileSystem]], We should not disable hive overwrite. Otherwise, hive will use the [[FileSystem]] instance belong to the table location to copy files, which will fail [[FileSystem#checkPath]] see https://github.com/apache/hive/blob/rel/release-2.3.7/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1648-L1659 -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org