[
https://issues.apache.org/jira/browse/HIVE-17394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ratandeep Ratti updated HIVE-17394:
-----------------------------------
Summary: AvroSerde is regenerating TypeInfo objects for each nullable Avro
field for every row (was: AvroSerde is regenerating TypeInfo objects for each
nullable Avro field in a row)
> AvroSerde is regenerating TypeInfo objects for each nullable Avro field for
> every row
> -------------------------------------------------------------------------------------
>
> Key: HIVE-17394
> URL: https://issues.apache.org/jira/browse/HIVE-17394
> Project: Hive
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Ratandeep Ratti
>
> The following methods in {{AvroDeserializer}} keep regenerating TypeInfo
> objects for every nullable field in a row.
> This is happening in the following methods.
> {code}
> private Object deserializeNullableUnion(Object datum, Schema fileSchema,
> Schema recordSchema) throws AvroSerdeException {
> // elided
> line 312: return worker(datum, fileSchema, newRecordSchema,
> SchemaToTypeInfo.generateTypeInfo(newRecordSchema, null));
> }
> ..
> private Object deserializeSingleItemNullableUnion(Object datum, Schema Schema
> recordSchema)
> // elided
> line 357: return worker(datum, currentFileSchema, schema,
> SchemaToTypeInfo.generateTypeInfo(schema, null));
> {code}
> This is really bad in terms of performance. I'm not sure why didn't we use
> the TypeInfo we already have instead of generating again for each nullable
> field. If you look at the {{worker}} method which calls the method
> {{deserializeNullableUnion}} the typeInfo corresponding to the nullable field
> column is already determined. Not sure why we have to determine that
> information again.
> More the cache in SchmaToTypeInfo does not help in nullable Avro records case
> as checking if an Avro record schema object already exists in the cache
> requires traversing the all the fields in the record schema.
> I've attached profiling snapshot which shows maximum time is being spent in
> the cache.
> One way of fixing this IMO is to make use of the column TypeInfo which is
> already passed in the worker method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
