jack-leitch98 opened a new pull request, #59688: URL: https://github.com/apache/spark/pull/59688
### What changes were proposed in this pull request? Change `VariantUtil.checkDecimal` to bound `d.scale()` against `MAX_DECIMAL16_PRECISION` (the spec-defined ceiling of 38 that applies to all Variant decimal types) rather than the per-type `maxPrecision`. Tracked in [SPARK-60443](https://issues.apache.org/jira/browse/SPARK-60443). ### Why are the changes needed? Per the [Variant encoding spec](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md), scale for `decimal4`, `decimal8`, and `decimal16` is uniformly bounded by `[0, 38]`. The per-type restriction is on precision, not scale. Current `checkDecimal` conflates the two: ```java if (d.precision() > maxPrecision || d.scale() > maxPrecision) { ``` For `decimal4`, `maxPrecision = 9`, so a spec-conformant value like `0.00042626262` (unscaled int `42626262`, precision 8, scale 11) throws `MALFORMED_VARIANT` even though the encoding is valid. Values written by `parquet-variant`'s `VariantJsonParser` that choose decimal width by unscaled-int fit (per parquet-java PR [#3335](https://github.com/apache/parquet-java/pull/3335)) fail to read in Spark, while DuckDB, Snowflake, and parquet-variant's own reader accept them. ### Does this PR introduce _any_ user-facing change? Yes. Reads of Variant decimal values with scale greater than the type precision no longer throw. Values that were previously rejected now return the expected decimal. ### How was this patch tested? Added unit tests in `VariantExpressionSuite` covering: - `decimal4` with scale 11 (previously threw, now reads) - `decimal8` with scale 19 (previously threw, now reads) - `decimal4` with scale 39 still rejected (hard spec ceiling of 38) ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
