Daniel Dai created HIVE-21249: --------------------------------- Summary: Reduce memory footprint in ObjectStore.refreshPrivileges Key: HIVE-21249 URL: https://issues.apache.org/jira/browse/HIVE-21249 Project: Hive Issue Type: Bug Components: Standalone Metastore Reporter: Daniel Dai Assignee: Daniel Dai
We found there're could be many records in TBL_COL_PRIVS for a single table (a table granted to many users), thus result a OOM in ObjectStore.listTableAllColumnGrants. We shall reduce the memory footprint for ObjectStore.refreshPrivileges. Here is the stack of OOM: {code} org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:690) org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:710) org.apache.hadoop.hive.metastore.ObjectStore.listTableAllColumnGrants(ObjectStore.java:6629) org.apache.hadoop.hive.metastore.ObjectStore.refreshPrivileges(ObjectStore.java:6200) sun.reflect.NativeMethodAccessorImpl.invoke0(Native method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) com.sun.proxy.$Proxy32.refreshPrivileges(, line not available) org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.refresh_privileges(HiveMetaStore.java:6507) sun.reflect.NativeMethodAccessorImpl.invoke0(Native method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) com.sun.proxy.$Proxy34.refresh_privileges(, line not available) org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17608) org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17592) org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636) org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631) java.security.AccessController.doPrivileged(Native method) javax.security.auth.Subject.doAs(Subject.java:422) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688) org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631) org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)