pingzh opened a new pull request, #5491: URL: https://github.com/apache/datafusion-comet/pull/5491
## Which issue does this PR close? Part of #5352. This is the fourth foundational PR and does not close the issue. Previous PRs: - https://github.com/apache/datafusion-comet/pull/5473 - https://github.com/apache/datafusion-comet/pull/5476 - https://github.com/apache/datafusion-comet/pull/5481 ## Rationale for this change Previous PRs introduced RSS partition writers, destination-aware shuffle plans, and native RSS execution. However, serialized RSS plans cannot execute without connecting their task-owned JVM callback to the native planner. This PR completes that connection while preserving existing local shuffle behavior and JNI compatibility. ## What changes are included in this PR? - Add task-scoped RSS callback registration through a separate JNI method, preserving the existing `createPlan` signature. - Pass optional shuffle callbacks through `CometExecIterator` into native execution contexts and the physical planner. - Bind RSS shuffle plans to their task-owned callback and preserve callback isolation between tasks. - Reject RSS plans without callbacks, RSS plans containing local output paths, and callbacks supplied to local shuffle plans. - Apply the JVM’s conservative maximum byte-array size to RSS frame limits. - Preserve original callback exceptions across JNI boundaries. - Clean up native resources and all shuffle inputs when callback registration fails. ## How are these changes tested? - Added planner tests covering RSS binding, task isolation, destination validation, and end-to-end native execution. - Added Spark/JNI integration tests covering complete RSS frames, original callback exception identity, existing JNI signature compatibility, and registration-failure cleanup. - Passed 105 Rust tests across planner, shuffle, JNI bridge, and protobuf suites. - Passed all 35 tests in `CometNativeShuffleSuite`. - Passed workspace-wide Clippy, formatting checks, and the full Maven reactor build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
