[ https://issues.apache.org/jira/browse/SPARK-38245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494691#comment-17494691 ]
Erik Krogen commented on SPARK-38245: ------------------------------------- {quote} now I am curious to know why including the namespace would "get messy quickly" {quote} Namespaces (in my experience) are often fairly long, so let's say we have records like {{org.apache.spark.avrotest.RecordOne}}. This is a very long and unwieldy field name! Plus, I don't think we allow periods in field names, so we would probably replace them with underscores. Personally I feel I would rather refer to a field like {{member1}} than a field like {{org_apache_spark_avrotest_RecordOne}} -- but I suppose that there are valid arguments for the longer one. {quote} I am wondering if other solutions could help make this decoding of avro files to DataFrame loss-less (e.g. keeping a mapping of positional name to field names, etc.) {quote} I am not really seeing how the conversion is lossy, currently. Taking the example I shared above with an int/long union, the current schema will be: {code} root |-- foo: struct | |-- member0: int | |-- member1: long {code} If you want to derive the original union branches, you can just check the types in the schema. The mapping you described is already maintained implicitly in the mapping of field names to their types. > Avro Complex Union Type return `member$I` > ----------------------------------------- > > Key: SPARK-38245 > URL: https://issues.apache.org/jira/browse/SPARK-38245 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.2.1 > Environment: +OS+ > * Debian GNU/Linux 10 (Docker Container) > +packages & others+ > * spark-avro_2.12-3.2.1 > * python 3.7.3 > * pyspark 3.2.1 > * spark-3.2.1-bin-hadoop3.2 > * Docker version 20.10.12 > Reporter: Teddy Crepineau > Priority: Major > Labels: avro, newbie > > *Short Description* > When reading complex union types from Avro files, there seems to be some > information lost as the name of the record is omitted and {{member$i}} is > instead returned. > *Long Description* > +Error+ > Given the Avro schema {{{}schema.avsc{}}}, I would expected the schema when > reading the avro file using {{read_avro.py}} to be as {{{}expected.txt{}}}. > Instead, I get the schema output in {{reality.txt}} where {{RecordOne}} > became {{{}member0{}}}, etc. > This causes information lost and makes the DataFrame unusable. > From my understanding this behavior was implemented > [here.|https://github.com/databricks/spark-avro/pull/117] > > {code:java|title=read_avro.py} > df = spark.read.format("avro").load("path/to/my/file.avro") > df.printSchema() > {code} > {code:java|title=schema.avsc} > { > "type": "record", > "name": "SomeData", > "namespace": "my.name.space", > "fields": [ > { > "name": "ts", > "type": { > "type": "long", > "logicalType": "timestamp-millis" > } > }, > { > "name": "field_id", > "type": [ > "null", > "string" > ], > "default": null > }, > { > "name": "values", > "type": [ > { > "type": "record", > "name": "RecordOne", > "fields": [ > { > "name": "field_a", > "type": "long" > }, > { > "name": "field_b", > "type": { > "type": "enum", > "name": "FieldB", > "symbols": [ > "..." > ], > } > }, > { > "name": "field_C", > "type": { > "type": "array", > "items": "long" > } > } > ] > }, > { > "type": "record", > "name": "RecordTwo", > "fields": [ > { > "name": "field_a", > "type": "long" > } > ] > } > ] > } > ] > }{code} > {code:java|title=expected.txt} > root > |-- ts: timestamp (nullable = true) > |-- field_id: string (nullable = true) > |-- values: struct (nullable = true) > | |-- RecordOne: struct (nullable = true) > | | |-- field_a: long (nullable = true) > | | |-- field_b: string (nullable = true) > | | |-- field_c: array (nullable = true) > | | | |-- element: long (containsNull = true) > | |-- RecordTwo: struct (nullable = true) > | | |-- field_a: long (nullable = true) > {code} > {code:java|title=reality.txt} > root > |-- ts: timestamp (nullable = true) > |-- field_id: string (nullable = true) > |-- values: struct (nullable = true) > | |-- member0: struct (nullable = true) > | | |-- field_a: long (nullable = true) > | | |-- field_b: string (nullable = true) > | | |-- field_c: array (nullable = true) > | | | |-- element: long (containsNull = true) > | |-- member1: struct (nullable = true) > | | |-- field_a: long (nullable = true) > {code} -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org