[ https://issues.apache.org/jira/browse/HIVE-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliot West updated HIVE-15316: ------------------------------- Labels: avro avrostorage (was: ) > CTAS STORED AS AVRO: AvroTypeException Found default.record_0, expecting union > ------------------------------------------------------------------------------ > > Key: HIVE-15316 > URL: https://issues.apache.org/jira/browse/HIVE-15316 > Project: Hive > Issue Type: Bug > Components: Hive > Affects Versions: 2.1.0 > Reporter: David Maughan > Priority: Minor > Labels: avro, avrostorage > > There's an issue when querying a table that has been created as Avro via CTAS > when the target struct is at least 2 struct-levels deep. It can be replicated > with the following steps: > {code} > CREATE TABLE a > STORED AS AVRO > AS > SELECT named_struct('c', named_struct('d', 1)) as b; > SELECT b FROM a; > org.apache.avro.AvroTypeException: Found default.record_0, expecting union > {code} > The reason for this is that during table creation, the Avro schema is > generated from the Hive columns in {{AvroSerDe}} and then passed through the > Avro Schema Parser: {{new Schema.Parser().parse(schema.toString())}}. For the > above example, this creates the below schema in the Avro file. Note that the > lowest level struct, {{record_0}} has {{"namespace": "default"}}. > {code} > { > "type": "record", > "name": "a", > "namespace": "default", > "fields": [ > { > "name": "b", > "type": [ > "null", > { > "type": "record", > "name": "record_1", > "namespace": "", > "doc": "struct<c:struct<d:int>>", > "fields": [ > { > "name": "c", > "type": [ > "null", > { > "type": "record", > "name": "record_0", > "namespace": "default", > "doc": "struct<d:int>", > "fields": [ > { > "name": "d", > "type": [ "null", "int" ], > "doc": "int", > "default": null > } > ] > } > ], > "doc": "struct<d:int>", > "default": null > } > ] > } > ], > "default": null > } > ] > } > {code} > On a subsequent select query, the Avro schema is again generated from the > Hive columns. However, this time it is not passed through the Avro Schema > Parser and the {{namespace}} attribute is not present in {{record_0}}. The > actual Error message _"Found default.record_0, expecting union"_ is slightly > misleading. Although it is expecting a union, it is specifically expecting a > null or a record named {{record_0}} but it finds {{default.record_0}}. > I believe this is a bug in Avro. I'm not sure whether the correct behaviour > is to cascade the namespace down or not but it is definitely an inconsistency > between creating a schema via the builders and parser. I've created > [AVRO-1965|https://issues.apache.org/jira/browse/AVRO-1965] for this. > However, I believe that defensively passing the schema through the Avro > Schema Parser on a select query would fix this issue in Hive without an Avro > fix and version bump in Hive. -- This message was sent by Atlassian JIRA (v6.3.15#6346)