[ https://issues.apache.org/jira/browse/HIVE-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14251986#comment-14251986 ]
Jonathan Bender commented on HIVE-7049: --------------------------------------- Seems like we can get away with the following patch (confirm the fileSchema AKA writer's schema is actually a union type before trying to find the type that the reader schema expects). If not, just use the schema as is (it should be promoted to a union by Avro). This worked for me in local testing. ```diff --git a/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java b/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java index ce933ff..032761c 100644 --- a/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java +++ b/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java @@ -265,9 +265,12 @@ private Object deserializeNullableUnion(Object datum, Schema fileSchema, Schema if(schema.getType().equals(Schema.Type.NULL)) { return null; } + Schema writerSchema = fileSchema; + if (writerSchema != null && writerSchema.getType().equals(Schema.Type.UNION)) { + writerSchema = writerSchema.getTypes().get(tag); + } - return worker(datum, fileSchema == null ? null : fileSchema.getTypes().get(tag), schema, - SchemaToTypeInfo.generateTypeInfo(schema)); + return worker(datum, writerSchema, schema, SchemaToTypeInfo.generateTypeInfo(schema)); } ``` > Unable to deserialize AVRO data when file schema and record schema are > different and nullable > --------------------------------------------------------------------------------------------- > > Key: HIVE-7049 > URL: https://issues.apache.org/jira/browse/HIVE-7049 > Project: Hive > Issue Type: Bug > Reporter: Mohammad Kamrul Islam > Assignee: Mohammad Kamrul Islam > Attachments: HIVE-7049.1.patch > > > It mainly happens when > 1 )file schema and record schema are not same > 2 ) Record schema is nullable but file schema is not. > The potential code location is at class AvroDeserialize > > {noformat} > if(AvroSerdeUtils.isNullableType(recordSchema)) { > return deserializeNullableUnion(datum, fileSchema, recordSchema, > columnType); > } > {noformat} > In the above code snippet, recordSchema is verified if it is nullable. But > the file schema is not checked. > I tested with these values: > {noformat} > recordSchema= ["null","string"] > fielSchema= "string" > {noformat} > And i got the following exception <line numbers might not be the same due to > mu debugged code version>. > {noformat} > org.apache.avro.AvroRuntimeException: Not a union: "string" > at org.apache.avro.Schema.getTypes(Schema.java:272) > at > org.apache.hadoop.hive.serde2.avro.AvroDeserializer.deserializeNullableUnion(AvroDeserializer.java:275) > at > org.apache.hadoop.hive.serde2.avro.AvroDeserializer.worker(AvroDeserializer.java:205) > at > org.apache.hadoop.hive.serde2.avro.AvroDeserializer.workerBase(AvroDeserializer.java:188) > at > org.apache.hadoop.hive.serde2.avro.AvroDeserializer.deserialize(AvroDeserializer.java:174) > at > org.apache.hadoop.hive.serde2.avro.TestAvroDeserializer.verifyNullableType(TestAvroDeserializer.java:487) > at > org.apache.hadoop.hive.serde2.avro.TestAvroDeserializer.canDeserializeNullableTypes(TestAvroDeserializer.java:407) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)