Rajesh Balamohan created HIVE-24776: ---------------------------------------
Summary: Reduce HMS DB calls during stats updates Key: HIVE-24776 URL: https://issues.apache.org/jira/browse/HIVE-24776 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan When adding large number of partitions (100s/1000s) in a table, it ends up making lots of getTable calls which are not needed. Lines mentioned below may vary slightly in apache-master. {noformat} at org.datanucleus.api.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:620) at org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:637) at org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:646) at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2112) at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2150) at org.apache.hadoop.hive.metastore.ObjectStore.ensureGetMTable(ObjectStore.java:4578) at org.apache.hadoop.hive.metastore.ObjectStore.ensureGetTable(ObjectStore.java:4588) at org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:9264) at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at com.sun.proxy.$Proxy27.updatePartitionColumnStatistics(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartitonColStatsInternal(HiveMetaStore.java:6679) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:8655) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:8592) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at com.sun.proxy.$Proxy28.set_aggr_stats_for(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:19060) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:19044) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)