HyukjinKwon opened a new pull request, #58607:
URL: https://github.com/apache/spark/pull/58607
### What changes were proposed in this pull request?
Add an opt-in `spark.sql.variant.maxNestingDepth` (internal, default `-1` =
unlimited = unchanged).
When set to a positive value, rendering a variant value whose nesting
exceeds it fails instead of
recursing without bound. The limit is threaded as a parameter into the
recursive variant read
paths so `common/variant` does not need `SQLConf` access:
- `Variant.toJson` / `toJsonImpl` and `VariantVal.toJson` gain overloads
carrying the limit; the
existing signatures delegate with `-1` (unchanged).
- The SQL cast/`variant_get`-to-string path, the Parquet variant shredding
read, and
`to_json` obtain the limit once (per expression / per task), not per row.
### Why are the changes needed?
`Variant.toJsonImpl` recursed per nested element with no bound, so a deeply
nested variant could
exhaust the stack when rendered. The existing size limit does not prevent
this (a small-per-level
variant can nest very deeply within the size cap). This adds an optional
bound.
### Does this PR introduce _any_ user-facing change?
No by default. When `spark.sql.variant.maxNestingDepth` is set to a positive
value, variant values
nested more deeply than the limit raise an error when rendered.
### How was this patch tested?
New `VariantExpressionSuite` case: a deeply nested variant renders fully
when the limit is unset or
generous and is rejected when the limit is smaller than its depth.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
Co-authored-by: Isaac <[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
