WinkerDu opened a new pull request #26971: [SPARK-30320][SQL] Fix insert overwrite to DataSource table with dynamic partition error URL: https://github.com/apache/spark/pull/26971 ### What changes were proposed in this pull request? Task attempt id is append to dynamic partition staging dir, commitTask in HadoopMapReduceCommitProtocol carries attempt id in TaskCommitMessage ### Why are the changes needed? This PR fix insert overwrite to DataSource table with dynamic partition error when running multiple task attempts. suppose there are one task attempt and one speculative task attempt, the speculative would raise FileAlreadyExistsException because of same staging dir attempt tasks commit ### Does this PR introduce any user-facing change? This PR add a configuration 'spark.max.local.task.failures' to set max failure time in LOCAL mode, default 1 ### How was this patch tested? Added UT
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
