[
https://issues.apache.org/jira/browse/HIVE-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14251986#comment-14251986
]
Brock Noland edited comment on HIVE-7049 at 1/25/15 9:57 PM:
-------------------------------------------------------------
Seems like we can get away with the following patch (confirm the fileSchema AKA
writer's schema is actually a union type before trying to find the type that
the reader schema expects). If not, just use the schema as is (it should be
promoted to a union by Avro).
This worked for me in local testing.
{noformat}
diff --git
a/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
b/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
index ce933ff..032761c 100644
---
a/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
+++
b/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
@@ -265,9 +265,12 @@ private Object deserializeNullableUnion(Object datum,
Schema fileSchema, Schema
if(schema.getType().equals(Schema.Type.NULL)) {
return null;
}
+ Schema writerSchema = fileSchema;
+ if (writerSchema != null &&
writerSchema.getType().equals(Schema.Type.UNION)) {
+ writerSchema = writerSchema.getTypes().get(tag);
+ }
- return worker(datum, fileSchema == null ? null :
fileSchema.getTypes().get(tag), schema,
- SchemaToTypeInfo.generateTypeInfo(schema));
+ return worker(datum, writerSchema, schema,
SchemaToTypeInfo.generateTypeInfo(schema));
}
{noformat}
was (Author: jonathan.bender):
Seems like we can get away with the following patch (confirm the fileSchema AKA
writer's schema is actually a union type before trying to find the type that
the reader schema expects). If not, just use the schema as is (it should be
promoted to a union by Avro).
This worked for me in local testing.
```diff --git
a/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
b/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
index ce933ff..032761c 100644
---
a/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
+++
b/src/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
@@ -265,9 +265,12 @@ private Object deserializeNullableUnion(Object datum,
Schema fileSchema, Schema
if(schema.getType().equals(Schema.Type.NULL)) {
return null;
}
+ Schema writerSchema = fileSchema;
+ if (writerSchema != null &&
writerSchema.getType().equals(Schema.Type.UNION)) {
+ writerSchema = writerSchema.getTypes().get(tag);
+ }
- return worker(datum, fileSchema == null ? null :
fileSchema.getTypes().get(tag), schema,
- SchemaToTypeInfo.generateTypeInfo(schema));
+ return worker(datum, writerSchema, schema,
SchemaToTypeInfo.generateTypeInfo(schema));
}
```
> Unable to deserialize AVRO data when file schema and record schema are
> different and nullable
> ---------------------------------------------------------------------------------------------
>
> Key: HIVE-7049
> URL: https://issues.apache.org/jira/browse/HIVE-7049
> Project: Hive
> Issue Type: Bug
> Reporter: Mohammad Kamrul Islam
> Assignee: Mohammad Kamrul Islam
> Attachments: HIVE-7049.1.patch
>
>
> It mainly happens when
> 1 )file schema and record schema are not same
> 2 ) Record schema is nullable but file schema is not.
> The potential code location is at class AvroDeserialize
>
> {noformat}
> if(AvroSerdeUtils.isNullableType(recordSchema)) {
> return deserializeNullableUnion(datum, fileSchema, recordSchema,
> columnType);
> }
> {noformat}
> In the above code snippet, recordSchema is verified if it is nullable. But
> the file schema is not checked.
> I tested with these values:
> {noformat}
> recordSchema= ["null","string"]
> fielSchema= "string"
> {noformat}
> And i got the following exception <line numbers might not be the same due to
> mu debugged code version>.
> {noformat}
> org.apache.avro.AvroRuntimeException: Not a union: "string"
> at org.apache.avro.Schema.getTypes(Schema.java:272)
> at
> org.apache.hadoop.hive.serde2.avro.AvroDeserializer.deserializeNullableUnion(AvroDeserializer.java:275)
> at
> org.apache.hadoop.hive.serde2.avro.AvroDeserializer.worker(AvroDeserializer.java:205)
> at
> org.apache.hadoop.hive.serde2.avro.AvroDeserializer.workerBase(AvroDeserializer.java:188)
> at
> org.apache.hadoop.hive.serde2.avro.AvroDeserializer.deserialize(AvroDeserializer.java:174)
> at
> org.apache.hadoop.hive.serde2.avro.TestAvroDeserializer.verifyNullableType(TestAvroDeserializer.java:487)
> at
> org.apache.hadoop.hive.serde2.avro.TestAvroDeserializer.canDeserializeNullableTypes(TestAvroDeserializer.java:407)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
