HyukjinKwon opened a new pull request, #58598: URL: https://github.com/apache/spark/pull/58598
### What changes were proposed in this pull request? Adds a new config `spark.sql.avro.catalystTypeParsingMaxDepth` (default `-1`, disabled). During Avro schema inference, the Catalyst type carried in the `spark.sql.catalyst.type` schema property is parsed with the recursive-descent Catalyst parser. When the config is set to a positive value, the nesting depth of that type string is checked first and rejected if it exceeds the limit, before the parser recurses. ### Why are the changes needed? A deeply nested value in that property can exhaust the driver stack while parsing during schema inference. This lets operators optionally bound the nesting depth. The check is off by default, so there is no behavior change unless it is set. ### Does this PR introduce _any_ user-facing change? Yes: a new opt-in config `spark.sql.avro.catalystTypeParsingMaxDepth`, default `-1` (disabled), which preserves the previous behavior. When set to a positive value, an Avro `spark.sql.catalyst.type` whose nesting depth exceeds the limit is rejected. ### How was this patch tested? Added a case to `AvroSchemaHelperSuite` asserting a deeply nested type is rejected when the limit is set, and parses unchanged with the default. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
