andygrove opened a new pull request, #5321: URL: https://github.com/apache/datafusion-comet/pull/5321
## Which issue does this PR close? This backports fixes for the following issues to `branch-1.0`, for a 1.0.1 patch release: - #5093 - #5073 - #5137 - #5281 ## Rationale for this change 1.0.0 was released from `branch-1.0`, and four bug fixes have merged to `main` since the branch point. All four are user-visible failures rather than enhancements, so they belong in a patch release: - **#5137** (`priority:high`): `ShuffleScanExec` and `ExpandExec` stamp a catalyst schema onto a batch without casting, so Arrow rejects a column whose nested `nullable` flags are narrower than the declared type. The task aborts deterministically for the partition's data shape, and the direct-read shuffle path is enabled by default. - **#5281**: any `ScalaUDF` or Java UDF whose closure was captured by a class from a user jar (`--jars` / `spark.jars`) fails with a `ClassCastException` masking a `ClassNotFoundException`, because the Tokio worker thread attaches to the JVM with no context classloader. - **#5093**: the ANSI overflow pre-scan in unary negation reads null slots, so a MIN sentinel left behind in a null slot by `filter`, `slice`, or FFI raises a spurious overflow error on a query that should succeed. - **#5073**: ANSI errors from native `next_day` and `make_date` surface as `CometNativeException`, losing Spark's exception class, error class, and SQLSTATE. `main` also carries #5225 (duplicate `CheckOverflow` evaluation for decimal division). That one is deliberately excluded: #5190 is labeled `enhancement`, the double wrapper produces correct results in LEGACY, TRY, and ANSI modes, and the fix reorganizes ownership of decimal overflow wrapping, which is more churn than a patch release warrants. ## What changes are included in this PR? Four clean cherry-picks from `main`, in merge order, with no conflicts and no changes to the original commits: - `578379339` fix: skip null slots when checking overflow in unary negation (#5162) - `ef6019719` fix: surface next_day and make_date ANSI errors as Spark exceptions (#5167) - `cd4d0e25d` fix: normalize nested field nullability in ShuffleScanExec and ExpandExec (#5138) - `003f60894` fix: propagate the Spark task ClassLoader to JVM UDF calls (#5282) ## How are these changes tested? Each cherry-pick carries the regression tests from its original PR. Verified on this branch: - `make` (native plus all Maven modules, Spark 4.1): BUILD SUCCESS - `cargo test -p datafusion-comet-common -p datafusion-comet-spark-expr -p datafusion-comet`: 793 passed, 0 failed, covering the new `native/common/src/schema.rs` and the rewritten `negative.rs` - `CometScalaUDFClassLoaderSuite` and `CometTemporalExpressionSuite`: 37 tests, 0 failures, including `ScalaUDF closure captured by a user-jar class` for both `nativeScan=true` and `nativeScan=false`, and `next_day and make_date ANSI errors match Spark exceptions` - `CometExpressionSuite` unary negation tests: 3 tests, 0 failures, including `unary negative integer overflow test` - `dev/ci/check-suites.py`: passes, with `CometScalaUDFClassLoaderSuite` registered in both `pr_build_linux.yml` and `pr_build_macos.yml` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
