viirya commented on issue #26159: [SPARK-29506][SQL] Use dynamicPartitionOverwrite in FileCommitProtocol when insert into hive table URL: https://github.com/apache/spark/pull/26159#issuecomment-547574941 @rezasafi yea I know that PR and read some discussion there. My question is this seems not specified to dynamic partition overwrite case. I did not see an analysis about it is specified for dynamic partition overwrite case, or I miss it? The problem happens because the filename to write was produced by same params between tasks retries, so the same path was written and FileAlreadyExistsException was thrown. Even dynamic partition overwrite is disabled, I did not see there is difference in determining the path to write. So wondering if it is specified to dynamic partition overwrite or not.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
