Rajesh Balamohan created HIVE-24649: ---------------------------------------
Summary: Optimise Hive::addWriteNotificationLog for large data inserts Key: HIVE-24649 URL: https://issues.apache.org/jira/browse/HIVE-24649 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Rajesh Balamohan When loading dynamic partition with large dataset, it spends lot of time in "Hive::loadDynamicPartitions --> addWriteNotificationLog". Though it is for same for same table, it ends up loading table and partition details for every partition and writes to notification log. Also, "Partition" details may be already present in {{PartitionDetails}} object in {{Hive::loadDynamicPartitions}}. This is unnecessarily recomputed again in {{HiveMetaStore::add_write_notification_log}} Lines of interest: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3028 https://github.com/apache/hive/blob/89073a94354f0cc14ec4ae0a43e05aae29276b4d/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L8500 -- This message was sent by Atlassian Jira (v8.3.4#803005)