SandishKumarHN commented on code in PR #38922: URL: https://github.com/apache/spark/pull/38922#discussion_r1053690877
########## connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala: ########## @@ -92,17 +108,35 @@ object SchemaConverters { MapType(keyType, valueType, valueContainsNull = false).defaultConcreteType, nullable = false)) case MESSAGE => - if (existingRecordNames.contains(fd.getFullName)) { + // If the `recursive.fields.max.depth` value is not specified, it will default to -1; + // recursive fields are not permitted. Setting it to 0 drops all recursive fields, + // 1 allows it to be recursed once, and 2 allows it to be recursed twice and so on. + // A value greater than 10 is not allowed, and if a protobuf record has more depth for + // recursive fields than the allowed value, it will be truncated and some fields may be + // discarded. + // SQL Schema for the protobuf message `message Person { string name = 1; Person bff = 2}` + // will vary based on the value of "recursive.fields.max.depth". + // 0: struct<name: string, bff: null> + // 1: struct<name string, bff: <name: string, bff: null>> + // 2: struct<name string, bff: <name: string, bff: struct<name: string, bff: null>>> ... + val recordName = fd.getMessageType.getFullName Review Comment: @cloud-fan fd.getFullName gives a fully qualified name along with a field name, we needed the fully qualified type name. we made this decision above. here is the difference. ``` println(s"${fd.getFullName} : ${fd.getMessageType.getFullName}") org.apache.spark.sql.protobuf.protos.Employee.ic : org.apache.spark.sql.protobuf.protos.IC org.apache.spark.sql.protobuf.protos.IC.icManager : org.apache.spark.sql.protobuf.protos.Employee org.apache.spark.sql.protobuf.protos.Employee.ic : org.apache.spark.sql.protobuf.protos.IC org.apache.spark.sql.protobuf.protos.IC.icManager : org.apache.spark.sql.protobuf.protos.Employee org.apache.spark.sql.protobuf.protos.Employee.em : org.apache.spark.sql.protobuf.protos.EM ``` @rangadi previous code ```fd.getFullName``` fully qualified name along with a field name works to find out recursion. so before we just use to throw errors on any recursion field. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org