littlelittlewhite09 opened a new pull request, #12523:
URL: https://github.com/apache/gluten/pull/12523
Daft: [GLUTEN-12474][VL] Re-add partition-column sort for row-based fallbac…
Problem:
When writing to a dynamically-partitioned table via the row-based fallback
WriteFilesExec (e.g. ORC/Hive tables where the native Velox write is not
applicable), the job fails with FileAlreadyExistsException.
Root cause:
- With planned write enabled, `V1Writes` drops the write's required sort
at logical optimization time when the child already provides the ordering (e.g.
a `SortAggregate` keyed by the partition column outputs rows already ordered by
that column). So the physical `WriteFilesExec.requiredChildOrdering` is `Nil`.
- Gluten then replaces the `SortAggregate` with an unordered native hash
aggregate, and `EliminateLocalSort` eagerly removes the now-redundant local
sort feeding it. The ordering the write implicitly depended on is destroyed.
- When the write falls back to the row-based `WriteFilesExec`, its
`DynamicPartitionDataSingleWriter` requires rows sorted by the partition
columns. Unsorted rows make it re-create an already-closed partition file and
fail with `FileAlreadyExistsException`.
## What changes are proposed in this pull request?
fix #12474
## How was this patch tested?
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
