[ https://issues.apache.org/jira/browse/HIVE-17394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ratandeep Ratti updated HIVE-17394: ----------------------------------- Attachment: AvroSerDeUnionTypeInfo.png AvroSerDe.nps > AvroSerde is regenerating TypeInfo objects for each nullable Avro field for > every row > ------------------------------------------------------------------------------------- > > Key: HIVE-17394 > URL: https://issues.apache.org/jira/browse/HIVE-17394 > Project: Hive > Issue Type: Bug > Affects Versions: 1.1.0 > Reporter: Ratandeep Ratti > Attachments: AvroSerDe.nps, AvroSerDeUnionTypeInfo.png > > > The following methods in {{AvroDeserializer}} keep regenerating TypeInfo > objects for every nullable field in a row. > This is happening in the following methods. > {code} > private Object deserializeNullableUnion(Object datum, Schema fileSchema, > Schema recordSchema) throws AvroSerdeException { > // elided > line 312: return worker(datum, fileSchema, newRecordSchema, > SchemaToTypeInfo.generateTypeInfo(newRecordSchema, null)); > } > .. > private Object deserializeSingleItemNullableUnion(Object datum, Schema Schema > recordSchema) > // elided > line 357: return worker(datum, currentFileSchema, schema, > SchemaToTypeInfo.generateTypeInfo(schema, null)); > {code} > This is really bad in terms of performance. I'm not sure why didn't we use > the TypeInfo we already have instead of generating again for each nullable > field. If you look at the {{worker}} method which calls the method > {{deserializeNullableUnion}} the typeInfo corresponding to the nullable field > column is already determined. Not sure why we have to determine that > information again. > More the cache in SchmaToTypeInfo does not help in nullable Avro records case > as checking if an Avro record schema object already exists in the cache > requires traversing the all the fields in the record schema. > I've attached profiling snapshot which shows maximum time is being spent in > the cache. > One way of fixing this IMO is to make use of the column TypeInfo which is > already passed in the worker method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)