pingzh opened a new pull request, #5473:
URL: https://github.com/apache/datafusion-comet/pull/5473
## Which issue does this PR close?
Part of #5352. This is the first foundational PR and does not close the
issue.
## Rationale for this change
Comet’s native shuffle currently writes partition data to local files.
Supporting remote shuffle services requires a transport-independent mechanism
for delivering complete encoded shuffle blocks to task-owned JVM callbacks.
This PR introduces that foundation without changing shuffle planning, adding
Celeborn integration, or modifying existing shuffle behavior.
## What changes are included in this PR?
- Add a Java `ShufflePartitionPusher` callback interface for task-owned
shuffle output.
- Add a Rust callback abstraction and JNI adapter that:
- Safely invokes callbacks from native execution threads.
- Preserves Java exceptions across the JNI boundary.
- Validates partition identifiers and payload lengths.
- Add `RssPartitionWriter`, which:
- Encodes complete, length-prefixed Arrow IPC shuffle blocks.
- Supports existing shuffle compression codecs and dictionary-encoded
batches.
- Enforces configurable frame limits and partition lifecycle ordering.
- Preserves shuffle metrics and propagates callback failures.
- Expose the writer for integration in subsequent PRs.
## How are these changes tested?
- 15 new Rust tests cover frame boundaries, partition routing, all
compression codecs, dictionary batches, error propagation, frame limits, and
lifecycle validation.
- Three new JNI tests cover partition and payload validation.
- All 50 native shuffle unit tests pass.
- All 28 existing JVM native-shuffle regression tests pass.
- Full-workspace Rust Clippy, Rust formatting, ScalaStyle, Spotless, CI
preflight, and the complete four-module Maven build all pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
