HyukjinKwon opened a new pull request, #328: URL: https://github.com/apache/spark-connect-rust/pull/328
### What changes were proposed in this pull request? [SPARK-61136](https://issues.apache.org/jira/browse/SPARK-61136): With `spark.sql.legacy.allowNegativeScaleOfDecimal`, a decimal can have a negative scale. For example, `CAST(11 AS DECIMAL(1, -1))` is `1E+1`: the unscaled value 1 at the scale -1. PySpark collects such a decimal as PyArrow gives it: `Decimal("1E+1")`, of which `int()` is 10. The Rust core formats the decimal string of a collected value. It now writes a negative scale as the unscaled value times ten to the negated scale, in scientific notation, as PyArrow's `Decimal` of it. So the client collects `Decimal("1E+1")` too, and `Decimal("-1E+2")` for `-123` cast to `DECIMAL(3, -2)`. ### Why are the changes needed? For a scale that is not positive, the core formatted the unscaled value alone. So the client collected `Decimal("1")` for `1E+1`, a value ten times smaller. Apache Spark's `test_negative_decimal` failed on this client: `[0, 1] != [0, 10]`. ### Does this PR introduce _any_ user-facing change? Yes. The decimals of a negative scale are collected with their values, as with PySpark 4.3.0. They were wrong before. ### How was this patch tested? - A new live test collects decimals cast to `DECIMAL(1, -1)` and `DECIMAL(3, -2)`, and checks their values and their strings. It passes with official PySpark 4.3.0, and fails without this change. - The unit test of the decimal strings of the core checks negative scales. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2196 passed, 8 skipped), and the live drop-in and Arrow suites (183 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
