ziting-openai commented on code in PR #5491:
URL: https://github.com/apache/datafusion-comet/pull/5491#discussion_r3868056348
##########
native/shuffle/src/shuffle_writer.rs:
##########
@@ -83,6 +120,30 @@ impl ShuffleWriterExec {
tracing_enabled: bool,
write_buffer_size: usize,
max_buffer_bytes: Option<usize>,
+ ) -> Result<Self> {
+ Self::try_new_with_destination(
+ input,
+ partitioning,
+ codec,
+ ShuffleWriterDestination::Local {
+ output_data_file,
+ output_index_file,
+ },
+ tracing_enabled,
+ write_buffer_size,
+ max_buffer_bytes,
+ )
+ }
+
+ /// Creates a shuffle writer for a local or task-owned remote destination.
+ pub fn try_new_with_destination(
Review Comment:
[P1] Push the actual task-callback/planner integration instead of the
already-merged 3/n tree. This destination-aware constructor already landed in
#5481; the current #5491 head `8281c3490c9cc007e846d02f3bc6ef500bf7ba16` and
its base `4897161704b7b8b7dfa909f4bf897c6508b11117` have the exact same full
Git tree, `82e424b01282e03d9209a0bd93285313efe428a8`, so merging this PR adds
no repository content. More importantly, production
`native/core/src/execution/planner.rs` still calls only the local
`ShuffleWriterExec::try_new` path and explicitly rejects every RSS descriptor
with `RSS shuffle partition writers are not supported until remote shuffle
execution is enabled`; its existing planner test even asserts that rejection.
No JNI callback registration, iterator/context propagation, production planner
binding, or advertised Spark integration test exists in this head. As a result
every real RSS task still fails before execution despite the unit tests
constructing this constructor direct
ly. Rebase onto #5481's merged commit and push the actual 4/n callback,
planner, JNI, and integration-test changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
