sunchao commented on PR #5533: URL: https://github.com/apache/datafusion-comet/pull/5533#issuecomment-5721907319
Rebased onto current `main` (`d0c10f2d`) and addressed the remaining requests in `7df125174`: - Added `spark.comet.exec.preserveEvaluationMasks.enabled` (default `true`) with a documented opt-out and native-execution controls. - Filed #6006 for the broader throwing-expression, conditional, and aggregate FILTER work; linked it from the docs and enrollment assertions. - Restored a minimal `unbase64_operator_masks.sql` with skipped-malformed, consumed-malformed, dispatcher, and native positive controls. - Combined this PR's aggregate restoration with `preserveSparkAggregateBuffers` through one helper. It preserves the different boundaries: Celeborn stays within its stage; a falling-back Final may cross unmaterialized shuffle exchanges. #5421 is still open rather than part of the rebased main, so its separate helper is not present in this branch. - Added the requested early bailout for plans without an enrolled expression or protection tag. Validated on Spark 4.1.3, Scala 2.13.17, and JDK 17. All **110 executed targeted tests passed**: `CometEvaluationMaskSuite` (9), `CometExecRuleSuite` (32), `CometScalarFunctionSuite` (14), `CometCelebornShufflePlanningSuite` (50), and the applicable `unbase64` SQL registrations (5, including both restored-fixture dispatcher modes). One additional Spark-3.5-only SQL registration skipped its body through the version gate and is excluded from that count. Full JVM test compilation, Spotless, Scalastyle, and `git diff --check` passed. The JVM tests used the native library from the successful [CI run for the exact rebased main commit](https://github.com/apache/datafusion-comet/actions/runs/35275954309) (`d0c10f2d397ae81b97f187c0dcb4d8f042f2cd9f`), after verifying the artifact digest and that native source/build inputs were unchanged. A local Cargo build was blocked because the configured registry lacks DataFusion 55.1.0; no registry or dependency was changed. Other Spark/Scala versions, macOS, and the full repository suite were not run locally. Hosted PR CI is separate from these results. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
