malinjawi opened a new pull request, #12562: URL: https://github.com/apache/gluten/pull/12562
## What changes are proposed in this pull request? The columnar V2 write path commits every task attempt unconditionally — it ignores `BatchWrite.useCommitCoordinator`. With speculative execution or a retried stage, two attempts of the same partition can both call `DataWriter.commit()` and produce duplicate output. This brings `WritingColumnarBatchSparkTask` in line with vanilla Spark's `DataWritingSparkTask`: - Thread `useCommitCoordinator` from the `BatchWrite` into the task, and gate `commit()` behind `OutputCommitCoordinator.canCommit`, throwing the standard commit-denied error when the attempt loses. - Skip `abort()` once a commit has succeeded. This path updates native write metrics after commit, and a failure there was aborting already-durable output. (Vanilla has no post-commit code that can throw, so it doesn't need this guard; we do.) - Count rows as `Long` instead of `Int` to avoid overflow on large partitions. ## How was this patch tested? New `ColumnarDataWritingSparkTaskSuite` drives the task with a recording writer factory and a stub commit coordinator: a denied attempt throws and aborts+closes the writer, a failure after commit does not abort, a failure before commit does, row counts accumulate past `Int.MaxValue`, and the coordinator-off path is unchanged. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (reviewed and tested before submission) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
