snmvaughan opened a new pull request, #7171: URL: https://github.com/apache/datafusion-comet/pull/7171
## Which issue does this PR close? Closes #7170. ## Rationale for this change Spark 4.0.5, 4.1.4 and 4.2.1 are in release candidates (4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3). They include two fixes that change results Comet reproduces natively: - [SPARK-58821](https://issues.apache.org/jira/browse/SPARK-58821) (apache/spark#58047): when `stop - start` overflows a long but the sequence is short, `sequence` returns its elements, where earlier releases raise `INTERNAL_ERROR("Unreachable code reached.")`. For example, `sequence(-9223372036854775808L, 9223372036854775807L, 9223372036854775807L)` returns `[-9223372036854775808, -1, 9223372036854775806]`. - [SPARK-60118](https://issues.apache.org/jira/browse/SPARK-60118) (apache/spark#59334): casting a string to decimal no longer applies the integral-digit limit to zero, so a zero with a large exponent such as `'0e79'` casts to 0 instead of NULL (or `NUMERIC_OUT_OF_SUPPORTED_RANGE` under ANSI). Comet keeps the old results, so on those releases it raises the internal error and returns NULL where Spark doesn't. That fails the `expect_error(Unreachable code reached)` case in `sequence.sql`, and the `CometNativeCastSuite` string-to-decimal tests whose inputs include such zeros (`check 0 mantissa` and the generated-value tests). Neither fix leaves anything in the plan for Comet to detect, so Comet picks the behavior by Spark version, as it already does for SPARK-59602 in `ArraySetSupport.normalizesFloats`. ## What changes are included in this PR? - `SparkFixes` (new) holds the version checks. Both fixes are in 4.0.5+, 4.1.4+, 4.2.1+ and 4.3+. - `sequence`: a new native variant, `spark_sequence_recomputed`, returns the exact length when the subtraction overflows but the length is within the limit. `CometSequence.nativeFunction` picks it where `SparkFixes.sequenceRecomputesOverflowedLength` holds, and `spark_sequence` otherwise. Lengths over the limit raise the limit error in both. - `cast`: a new `Cast` proto field, `decimal_zero_any_exponent`, which `CometCast` sets from `SparkFixes.castsZeroWithAnyExponentToDecimal`. When it is set, `parse_string_to_decimal` casts a zero with any exponent to 0 instead of raising `NUMERIC_OUT_OF_SUPPORTED_RANGE`. - The internal-error case moves from `sequence.sql` to `CometArrayExpressionSuite`, because SQL file version directives compare only major.minor and can't express a patch-release boundary. - Audit notes for `sequence` and `cast`. ## How are these changes tested? - Rust unit tests: - `recomputed_length_after_overflow` covers SPARK-58821's three cases with both variants. - `checked_length_matches_exact_released_spark_arithmetic` now runs in both modes. - `test_parse_string_to_decimal_zero_with_large_exponent` covers zeros with large exponents in both modes, including values from the cast fuzz tests. - `CometArrayExpressionSuite`: - `sequence native variant by Spark version` pins the version boundaries and checks that `convert` asks for the variant of the running Spark. - `sequence whose stop - start overflows a long` compares Comet with Spark on the three overflow cases: the elements where the fix is present, and the internal error from both engines otherwise. - `CometNativeCastSuite`: `cast StringType to DecimalType zero with any exponent by Spark version` pins the version boundaries and checks the proto field. `check 0 mantissa` gains `0e37` and `0e38`, on either side of the old zero limit. - `cargo fmt --check`, `cargo clippy --all-targets --workspace -- -D warnings` and spotless are clean, all `datafusion-comet-spark-expr` unit tests pass, and the Scala code compiles against Spark 4.0.4. I haven't run the Scala suites locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
