eason-yuchen-liu commented on code in PR #47425:
URL: https://github.com/apache/spark/pull/47425#discussion_r1690106801
##########
connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala:
##########
@@ -170,4 +173,23 @@ private[sql] object AvroOptions extends DataSourceOptions {
// When STABLE_ID_FOR_UNION_TYPE is enabled, the option allows to configure
the prefix for fields
// of Avro Union type.
val STABLE_ID_PREFIX_FOR_UNION_TYPE =
newOption("stableIdentifierPrefixForUnionType")
+
+ /**
+ * Adds support for recursive fields. If this option is not specified or is
set to 0, recursive
+ * fields are not permitted. Setting it to 1 drops all recursive fields, 2
allows recursive
+ * fields to be recursed once, and 3 allows it to be recursed twice and so
on, up to 15.
+ * Values larger than 15 are not allowed in order avoid inadvertently
creating very large schemas.
Review Comment:
Protobuf has max depth of 10 and it is hardcoded. 15 is used because some
users have demand for up to 12 and 3 more is given as buffer. I agree that it
will be better if users can increase the max depth at will. Since Protobuf also
does not support it. This config can be added in a future PR for both Protobuf
and Avro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
