pingzh opened a new pull request, #5627: URL: https://github.com/apache/datafusion-comet/pull/5627
## Which issue does this PR close? Closes #5529. Follow-up to #5352. ## Rationale for this change Comet installs a bootstrap hook in Celeborn's shared client factory to track transport ownership. Failures while installing or invoking that hook could escape into Celeborn client creation and disrupt ordinary, non-Comet Celeborn traffic. The hook also remained installed after the final Comet client released it. ## What changes are included in this PR? - Treat reflective, I/O, and runtime hook instrumentation failures as non-fatal, disable precise transport tracking, and fall back to Celeborn `PushState` accounting. - Reference-count the shared hook and remove the exact proxy when the final Comet client releases it, while preserving unrelated Celeborn bootstraps and their order. - Synchronize hook release with active bootstrap calls, keep stale bootstrap snapshots inert, and restore tracking when a later Comet client reacquires the shared factory. - Preserve already-observed transport ownership during fallback and serialize raw-push state binding so executor admission is never released early. ## How are these changes tested? - Added regression coverage for bootstrap instrumentation failures, ordinary Celeborn client creation, shared hook ownership, final-owner removal, stale snapshots, reacquisition, fatal bootstrap errors, and concurrent hook release. - Added fallback accounting coverage for recreated push states, concurrent submissions, cleanup races, retries, callbacks, and pending Netty writes. - Ran `CelebornShufflePartitionPusherSuite` with the default Spark 4.1 profile: 53 tests passed. - Ran Spotless and the Scala 2.12 parser check successfully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
