[ https://issues.apache.org/jira/browse/SPARK-48964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Jie resolved SPARK-48964. ------------------------------ Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47458 [https://github.com/apache/spark/pull/47458] > Fix the discrepancy between implementation, comment and documentation of > option recursive.fields.max.depth in ProtoBuf connector > -------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-48964 > URL: https://issues.apache.org/jira/browse/SPARK-48964 > Project: Spark > Issue Type: Documentation > Components: Connect > Affects Versions: 3.5.0, 4.0.0, 3.5.1, 3.5.2, 3.5.3 > Reporter: Yuchen Liu > Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > After the three PRs ([https://github.com/apache/spark/pull/38922,] > [https://github.com/apache/spark/pull/40011,] > [https://github.com/apache/spark/pull/40141]) working on the same option, > there are some legacy comments and documentation that has not been updated to > the latest implementation. This task should consolidate them. Below is the > correct description of the behavior. > The `recursive.fields.max.depth` parameter can be specified in the > from_protobuf options to control the maximum allowed recursion depth for a > field. Setting `recursive.fields.max.depth` to 1 drops all-recursive fields, > setting it to 2 allows it to be recursed once, and setting it to 3 allows it > to be recursed twice. Attempting to set the `recursive.fields.max.depth` to a > value greater than 10 is not allowed. If the `recursive.fields.max.depth` is > specified to a value smaller than 1, recursive fields are not permitted. The > default value of the option is -1. if a protobuf record has more depth for > recursive fields than the allowed value, it will be truncated and some fields > may be discarded. This check is based on the fully qualified field type. SQL > Schema for the protobuf message > {code:java} > message Person { string name = 1; Person bff = 2 }{code} > will vary based on the value of `recursive.fields.max.depth`. > {code:java} > 1: struct<name: string> > 2: struct<name string, bff: <name: string>> > 3: struct<name string, bff: <name: string, bff: struct<name: string>>> ... > {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org