pingzh opened a new pull request, #5668: URL: https://github.com/apache/datafusion-comet/pull/5668
## Which issue does this PR close? Closes #5527. ## Rationale for this change A large row can exceed native Celeborn's byte-admission budget even below the configured frame limit, then fail identically on every stage retry. This change makes the defaults consistent and lets an oversized row complete through local Comet shuffle. ## What changes are included in this PR? - Raise default executor admission from 256 MiB to 512 MiB to accommodate the 64 MiB frame limit and encoding workspace. Keep the remote memory limits enforced. - Preserve a typed size-limit failure across Rust, DataFusion, and JNI. Diagnostics identify the configured limits, encoding workspace, and minimum required reservation. - Coordinate a persistent local-shuffle decision on the driver, invalidate all remote map output, and retry the complete map stage with Comet's local writer and reader. Fence stale attempts and preserve native operators and Arrow shuffle format. - Require Spark 3.5.1 or newer for obsolete-stage result filtering. With dynamic allocation, require external shuffle service or shuffle tracking to retain local fallback files; other configurations keep ordinary Spark/Celeborn shuffle. Preserve fallback files during executor shutdown and clean them on explicit shuffle removal. - Document the behavior and register the recovery suite in Linux and macOS CI. ## How are these changes tested? - 45 Rust RSS tests and 2 JNI error-propagation tests passed, including 38/50/63 MiB rows under the new default budget. - Spark 3.5.9: 240 tests across eight configuration, planning, pusher, manager, writer, reader, input-RDD, and recovery suites passed. - Spark 4.1.3: 185 tests across seven affected suites passed; 99 manager, reader, and recovery tests passed after the final changes. - Regression coverage includes a 70 MiB row through local Comet shuffle without AQE, plus scaled automatic recovery with AQE enabled and disabled. Two-map tests wait for one remote map to complete before another fails, then verify both maps are recomputed locally. These tests use a fake Celeborn client with real native encoding, Spark scheduling, and local shuffle I/O. - Native library build, Clippy with warnings denied, Rust formatting, Spotless, Scalastyle, and CI suite-registration checks passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
