[jira] [Commented] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning
[ https://issues.apache.org/jira/browse/SPARK-41551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17694629#comment-17694629 ] Apache Spark commented on SPARK-41551: -- User 'steveloughran' has created a pull request for this issue: https://github.com/apache/spark/pull/40221 > Improve/complete PathOutputCommitProtocol support for dynamic partitioning > -- > > Key: SPARK-41551 > URL: https://issues.apache.org/jira/browse/SPARK-41551 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Priority: Minor > > Followup to SPARK-40034 as > * that is incomplete as it doesn't record the partitions > * as long at the job doesn't call `newTaskTempFileAbsPath()`, and slow > renames are ok, both s3a committers are actually OK to use. > It's only that newTaskTempFileAbsPath operation which is unsupported in s3a > committers; the post-job dir rename is O(data) but file by file rename is > correct for a non-atomic job commit. > # Cut PathOutputCommitProtocol.newTaskTempFile; to update super > partitionPaths (needs a setter). The superclass can't just say if (committer > instance of PathOutputCommitter as spark-core needs to compile with older > hadoop versions) > # downgrade failure in setup to log (info?) > # retain failure in the newTaskTempFileAbsPath call. > Testing: yes -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning
[ https://issues.apache.org/jira/browse/SPARK-41551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17694632#comment-17694632 ] Apache Spark commented on SPARK-41551: -- User 'steveloughran' has created a pull request for this issue: https://github.com/apache/spark/pull/40221 > Improve/complete PathOutputCommitProtocol support for dynamic partitioning > -- > > Key: SPARK-41551 > URL: https://issues.apache.org/jira/browse/SPARK-41551 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Priority: Minor > > Followup to SPARK-40034 as > * that is incomplete as it doesn't record the partitions > * as long at the job doesn't call `newTaskTempFileAbsPath()`, and slow > renames are ok, both s3a committers are actually OK to use. > It's only that newTaskTempFileAbsPath operation which is unsupported in s3a > committers; the post-job dir rename is O(data) but file by file rename is > correct for a non-atomic job commit. > # Cut PathOutputCommitProtocol.newTaskTempFile; to update super > partitionPaths (needs a setter). The superclass can't just say if (committer > instance of PathOutputCommitter as spark-core needs to compile with older > hadoop versions) > # downgrade failure in setup to log (info?) > # retain failure in the newTaskTempFileAbsPath call. > Testing: yes -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning
[ https://issues.apache.org/jira/browse/SPARK-41551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17649775#comment-17649775 ] Steve Loughran commented on SPARK-41551: So there's an interesting little "feature" of HadoopMapReduceCommitProtocol.newTaskTempFile() which is: If you call newTaskTempFile(tac, None, ext) when dynamicPartitionOverwrite is true, *and* spark-core was compiled with assertions -Xelide-below at a level which excludes assert(), then in job commit the entire directory tree is destroyed -both output and (implicitly) the .spark-staging dir. makes for a fairly messy job failure. The good news: spark builds don't do that, and since spark-core/spark-sql itself doesn't seem to invoke newTaskTempFile(_, None, _) in dynamic partition mode, it's not a serious risk. Is it worth hardening? > Improve/complete PathOutputCommitProtocol support for dynamic partitioning > -- > > Key: SPARK-41551 > URL: https://issues.apache.org/jira/browse/SPARK-41551 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Priority: Minor > > Followup to SPARK-40034 as > * that is incomplete as it doesn't record the partitions > * as long at the job doesn't call `newTaskTempFileAbsPath()`, and slow > renames are ok, both s3a committers are actually OK to use. > It's only that newTaskTempFileAbsPath operation which is unsupported in s3a > committers; the post-job dir rename is O(data) but file by file rename is > correct for a non-atomic job commit. > # Cut PathOutputCommitProtocol.newTaskTempFile; to update super > partitionPaths (needs a setter). The superclass can't just say if (committer > instance of PathOutputCommitter as spark-core needs to compile with older > hadoop versions) > # downgrade failure in setup to log (info?) > # retain failure in the newTaskTempFileAbsPath call. > Testing: yes -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning
[ https://issues.apache.org/jira/browse/SPARK-41551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17651380#comment-17651380 ] Apache Spark commented on SPARK-41551: -- User 'steveloughran' has created a pull request for this issue: https://github.com/apache/spark/pull/39185 > Improve/complete PathOutputCommitProtocol support for dynamic partitioning > -- > > Key: SPARK-41551 > URL: https://issues.apache.org/jira/browse/SPARK-41551 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Priority: Minor > > Followup to SPARK-40034 as > * that is incomplete as it doesn't record the partitions > * as long at the job doesn't call `newTaskTempFileAbsPath()`, and slow > renames are ok, both s3a committers are actually OK to use. > It's only that newTaskTempFileAbsPath operation which is unsupported in s3a > committers; the post-job dir rename is O(data) but file by file rename is > correct for a non-atomic job commit. > # Cut PathOutputCommitProtocol.newTaskTempFile; to update super > partitionPaths (needs a setter). The superclass can't just say if (committer > instance of PathOutputCommitter as spark-core needs to compile with older > hadoop versions) > # downgrade failure in setup to log (info?) > # retain failure in the newTaskTempFileAbsPath call. > Testing: yes -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning
[ https://issues.apache.org/jira/browse/SPARK-41551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17651381#comment-17651381 ] Apache Spark commented on SPARK-41551: -- User 'steveloughran' has created a pull request for this issue: https://github.com/apache/spark/pull/39185 > Improve/complete PathOutputCommitProtocol support for dynamic partitioning > -- > > Key: SPARK-41551 > URL: https://issues.apache.org/jira/browse/SPARK-41551 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Priority: Minor > > Followup to SPARK-40034 as > * that is incomplete as it doesn't record the partitions > * as long at the job doesn't call `newTaskTempFileAbsPath()`, and slow > renames are ok, both s3a committers are actually OK to use. > It's only that newTaskTempFileAbsPath operation which is unsupported in s3a > committers; the post-job dir rename is O(data) but file by file rename is > correct for a non-atomic job commit. > # Cut PathOutputCommitProtocol.newTaskTempFile; to update super > partitionPaths (needs a setter). The superclass can't just say if (committer > instance of PathOutputCommitter as spark-core needs to compile with older > hadoop versions) > # downgrade failure in setup to log (info?) > # retain failure in the newTaskTempFileAbsPath call. > Testing: yes -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning
[ https://issues.apache.org/jira/browse/SPARK-41551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17651384#comment-17651384 ] Steve Loughran commented on SPARK-41551: PR up. PathOutputCommitProtocol stops anyone trying to use a parent dir as the absolute path in dynamic update mode, as HadoopMapReduceCommitProtocol.commitJob() will blindly delete the entire dir tree at that point. I'm not convinced that feature is particularly safe. > Improve/complete PathOutputCommitProtocol support for dynamic partitioning > -- > > Key: SPARK-41551 > URL: https://issues.apache.org/jira/browse/SPARK-41551 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Priority: Minor > > Followup to SPARK-40034 as > * that is incomplete as it doesn't record the partitions > * as long at the job doesn't call `newTaskTempFileAbsPath()`, and slow > renames are ok, both s3a committers are actually OK to use. > It's only that newTaskTempFileAbsPath operation which is unsupported in s3a > committers; the post-job dir rename is O(data) but file by file rename is > correct for a non-atomic job commit. > # Cut PathOutputCommitProtocol.newTaskTempFile; to update super > partitionPaths (needs a setter). The superclass can't just say if (committer > instance of PathOutputCommitter as spark-core needs to compile with older > hadoop versions) > # downgrade failure in setup to log (info?) > # retain failure in the newTaskTempFileAbsPath call. > Testing: yes -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
