Weixin-Xu opened a new pull request, #13377:
URL: https://github.com/apache/gluten/pull/13377
## What changes are proposed in this pull request?
This PR fixes Bolt shuffle integration issues affecting Celeborn, Uniffle,
broadcast exchanges, and native benchmark replay.
- Use Bolt-specific shuffle reader/writer APIs for Celeborn and Uniffle,
and resolve the JNI push callback from the supplied transport object.
- Fix shuffle byte accounting, compression defaults, and missing metrics.
Preserve the first Arrow batch during adaptive writer sizing.
- Preserve broadcast build keys during canonicalization so exchange reuse
satisfies Spark distribution requirements. Cover query correctness and
`ValidateRequirements` in the regression tests.
- Fall back from unsupported empty-schema columnar shuffle and construct
spill URIs with `java.net.URI`.
- Preserve shuffle inputs for dump/replay and update native benchmarks to
the current Bolt shuffle APIs.
Add regression coverage for RSS round trips, broadcast canonicalization,
shuffle metrics, and spill paths.
The RSS regression tests require a corresponding Bolt engine fix for
uncompressed payloads and batches below the row-vector compression threshold.
That fix will be submitted separately to Bolt.
## How was this patch tested?
These changes were tested as part of the integrated Bolt CI candidate
using Bolt revision `071e54764f887dfb4337260e440f2f12f6ab82f5` with the
required engine fixes applied.
Integrated validation included Celeborn and Uniffle integration cases, the
native C++/UDF reference case, and native benchmark runs.
Added regressions cover:
- Uncompressed and LZ4 RSS round trips with Bolt, V2, and Arrow batches.
- An RSS pusher independent of the Celeborn pusher class.
- Broadcast build-key preservation and Spark distribution validation.
- Shuffle metrics and spill-path URI handling.
After splitting and rebasing this topic onto Apache Gluten main,
patch-equivalence checks passed. Removing the external engine patch from this
PR left all 21 Gluten source, build, and test files unchanged.
`git diff --check` passed.
The standalone rebased branch has not been rebuilt or runtime-tested
independently.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
