vboo123 opened a new pull request, #57348: URL: https://github.com/apache/spark/pull/57348
### What changes were proposed in this pull request? Add a new opt-in utility method `XSDToSchema.extendDecimalPrecision(schema: StructType, maxPrecision: Int = 38): StructType` that widens `DecimalType` precision to accommodate full integer-digit capacity. For each `DecimalType(p, s)` in the schema, the method raises precision to `min(maxPrecision, p + s)` while keeping scale unchanged. It recurses into nested `StructType` fields and `ArrayType` elements. The default inference behavior of `XSDToSchema.read()` is intentionally unchanged: it still maps XSD `totalDigits`/`fractionDigits` to `DecimalType(totalDigits, fractionDigits)` per the W3C spec. Users opt in by calling `extendDecimalPrecision` on the returned schema. ### Why are the changes needed? `XSDToSchema.read()` maps an XSD decimal restriction with `totalDigits=t` and `fractionDigits=f` to `DecimalType(t, f)`. This is spec-correct, but Spark's `DecimalType` scale is fixed, whereas the XSD `fractionDigits` facet is only a maximum. A value that uses fewer fractional digits and more integer digits overflows precision. For example, `totalDigits=20, fractionDigits=5` produces `DecimalType(20, 5)` which allows only 15 integer digits. Values exceeding that are silently read as `null` in non-ANSI mode, causing data loss with no warning. This matches the fix proposed by the JIRA reporter: an opt-in utility rather than changing default behavior. JIRA: https://issues.apache.org/jira/browse/SPARK-56486 ### Does this PR introduce _any_ user-facing change? Yes. A new public utility method `XSDToSchema.extendDecimalPrecision` is added. Existing behavior of `XSDToSchema.read()` is unchanged. ### How was this patch tested? Three new unit tests in `XSDToSchemaSuite`: - Verifies decimal widening and capping at max precision (e.g. `(12,6)` -> `(18,6)`, `(38,18)` unchanged) - Verifies recursion into nested struct fields and array elements - Verifies an explicit `maxPrecision` parameter is respected Ran the suite locally (all pass). Full lint/test matrix via GitHub Actions. ### Was this patch authored or co-authored using generative AI tooling? Generated using Kiro (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
