[jira] [Commented] (HIVE-24711) hive metastore memory leak
[ https://issues.apache.org/jira/browse/HIVE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17280830#comment-17280830 ] Karen Coppage commented on HIVE-24711: -- I can't diagnose anything based on just these log snippets but please check if your version contains this fix: HIVE-22700; if not it might help you. > hive metastore memory leak > -- > > Key: HIVE-24711 > URL: https://issues.apache.org/jira/browse/HIVE-24711 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Affects Versions: 3.1.0 >Reporter: LinZhongwei >Priority: Major > > hdp version:3.1.5.31-1 > hive version:3.1.0.3.1.5.31-1 > hadoop version:3.1.1.3.1.5.31-1 > We find that the hive metastore has memory leak if we set > compactor.initiator.on to true. > If we disable the configuration, the memory leak disappear. > How can we resolve this problem? > Even if we set the heap size of hive metastore to 40 GB, after 1 month the > hive metastore service will be down with outofmemory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24711) hive metastore memory leak
[ https://issues.apache.org/jira/browse/HIVE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277863#comment-17277863 ] LinZhongwei commented on HIVE-24711: If this config cannot be recognized by hdp hive, hive metastore restarting will fail. > hive metastore memory leak > -- > > Key: HIVE-24711 > URL: https://issues.apache.org/jira/browse/HIVE-24711 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Affects Versions: 3.1.0 >Reporter: LinZhongwei >Priority: Major > > hdp version:3.1.5.31-1 > hive version:3.1.0.3.1.5.31-1 > hadoop version:3.1.1.3.1.5.31-1 > We find that the hive metastore has memory leak if we set > compactor.initiator.on to true. > If we disable the configuration, the memory leak disappear. > How can we resolve this problem? > Even if we set the heap size of hive metastore to 40 GB, after 1 month the > hive metastore service will be down with outofmemory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24711) hive metastore memory leak
[ https://issues.apache.org/jira/browse/HIVE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277860#comment-17277860 ] LinZhongwei commented on HIVE-24711: yes. Following messages are from hivemetastore.log. But in my hdp version hive, I can not find metastore.housekeeping.threads.on in the config dir and ambari web . I use 'grep -R -i "metastore.housekeeping.threads.on"'. Because when I turned off 'compactor.Initiator' , PartitionDiscoveryTask logs disappeared. I think when I turned off compactor.Initiator, 'metastore.housekeeping.threads.on' was also turned off. I will try to set it on hive, and restart hive metastore. 2021-02-03T16:56:21,709 ERROR [PartitionDiscoveryTask-2]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(197)) - MetaException(message:java.security.AccessControlException: Permission denied: user=hive, access=WRITE, inode="/apps/edl_cn/staging/edl_cn.PAYMENT_EVENT_DELTA_incremental/etl_run_id=20200625005959":gp_etl_edl_batch:gp_etl_edl_batch:drwxr-xr-x > hive metastore memory leak > -- > > Key: HIVE-24711 > URL: https://issues.apache.org/jira/browse/HIVE-24711 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Affects Versions: 3.1.0 >Reporter: LinZhongwei >Priority: Major > > hdp version:3.1.5.31-1 > hive version:3.1.0.3.1.5.31-1 > hadoop version:3.1.1.3.1.5.31-1 > We find that the hive metastore has memory leak if we set > compactor.initiator.on to true. > If we disable the configuration, the memory leak disappear. > How can we resolve this problem? > Even if we set the heap size of hive metastore to 40 GB, after 1 month the > hive metastore service will be down with outofmemory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24711) hive metastore memory leak
[ https://issues.apache.org/jira/browse/HIVE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277146#comment-17277146 ] Karen Coppage commented on HIVE-24711: -- That error looks like it is from PartitionManagementTask (which runs if metastore.housekeeping.threads.on=true and metastore.task.threads.remote contains "org.apache.hadoop.hive.metastore.PartitionManagementTask"), not the compactor.Initiator. Do you see any doas errors like that from compactor.Initiator? > hive metastore memory leak > -- > > Key: HIVE-24711 > URL: https://issues.apache.org/jira/browse/HIVE-24711 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Affects Versions: 3.1.0 >Reporter: LinZhongwei >Priority: Major > > hdp version:3.1.5.31-1 > hive version:3.1.0.3.1.5.31-1 > hadoop version:3.1.1.3.1.5.31-1 > We find that the hive metastore has memory leak if we set > compactor.initiator.on to true. > If we disable the configuration, the memory leak disappear. > How can we resolve this problem? > Even if we set the heap size of hive metastore to 40 GB, after 1 month the > hive metastore service will be down with outofmemory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24711) hive metastore memory leak
[
https://issues.apache.org/jira/browse/HIVE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276829#comment-17276829
]
LinZhongwei commented on HIVE-24711:
This is the source code. Is FileSystem.closeAllForUGI(ugi) missing ?
final UserGroupInformation ugi;
try {
ugi = UserGroupInformation.getCurrentUser();
} catch (IOException e) {
throw new RuntimeException(e);
}
partFutures.add(threadPool.submit(new Callable() {
@Override
public Partition call() throws Exception {
ugi.doAs(new PrivilegedExceptionAction() {
@Override
public Object run() throws Exception {
try {
boolean madeDir = createLocationForAddedPartition(table,
part);
if (addedPartitions.put(new PartValEqWrapper(part),
madeDir) != null) {
// Technically, for ifNotExists case, we could insert one and discard the other
// because the first one now "exists", but it seems better to report the
problem
// upstream as such a command doesn't make sense.
throw new MetaException("Duplicate partitions in the
list: " + part);
}
initializeAddedPartition(table, part, madeDir);
} catch (MetaException e) {
throw new IOException(e.getMessage(), e);
}
return null;
}
});
return part;
}
}));
}
> hive metastore memory leak
> --
>
> Key: HIVE-24711
> URL: https://issues.apache.org/jira/browse/HIVE-24711
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore
>Affects Versions: 3.1.0
>Reporter: LinZhongwei
>Priority: Major
>
> hdp version:3.1.5.31-1
> hive version:3.1.0.3.1.5.31-1
> hadoop version:3.1.1.3.1.5.31-1
> We find that the hive metastore has memory leak if we set
> compactor.initiator.on to true.
> If we disable the configuration, the memory leak disappear.
> How can we resolve this problem?
> Even if we set the heap size of hive metastore to 40 GB, after 1 month the
> hive metastore service will be down with outofmemory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24711) hive metastore memory leak
[ https://issues.apache.org/jira/browse/HIVE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276786#comment-17276786 ] LinZhongwei commented on HIVE-24711: I will try to turn on ranger based authorization. > hive metastore memory leak > -- > > Key: HIVE-24711 > URL: https://issues.apache.org/jira/browse/HIVE-24711 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Affects Versions: 3.1.0 >Reporter: LinZhongwei >Priority: Major > > hdp version:3.1.5.31-1 > hive version:3.1.0.3.1.5.31-1 > hadoop version:3.1.1.3.1.5.31-1 > We find that the hive metastore has memory leak if we set > compactor.initiator.on to true. > If we disable the configuration, the memory leak disappear. > How can we resolve this problem? > Even if we set the heap size of hive metastore to 40 GB, after 1 month the > hive metastore service will be down with outofmemory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24711) hive metastore memory leak
[ https://issues.apache.org/jira/browse/HIVE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276783#comment-17276783 ] LinZhongwei commented on HIVE-24711: Here is a authorization related configuration in the hive-site.xml hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider Here is hivemetastore-site.xml http://www.w3.org/2001/XInclude";> hive.compactor.initiator.on true hive.compactor.worker.threads 10 hive.metastore.dml.events true hive.metastore.event.listeners hive.metastore.metrics.enabled true hive.metastore.transactional.event.listeners org.apache.hive.hcatalog.listener.DbNotificationListener hive.server2.metrics.enabled true hive.service.metrics.hadoop2.component hivemetastore hive.service.metrics.reporter HADOOP2 > hive metastore memory leak > -- > > Key: HIVE-24711 > URL: https://issues.apache.org/jira/browse/HIVE-24711 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Affects Versions: 3.1.0 >Reporter: LinZhongwei >Priority: Major > > hdp version:3.1.5.31-1 > hive version:3.1.0.3.1.5.31-1 > hadoop version:3.1.1.3.1.5.31-1 > We find that the hive metastore has memory leak if we set > compactor.initiator.on to true. > If we disable the configuration, the memory leak disappear. > How can we resolve this problem? > Even if we set the heap size of hive metastore to 40 GB, after 1 month the > hive metastore service will be down with outofmemory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24711) hive metastore memory leak
[ https://issues.apache.org/jira/browse/HIVE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276772#comment-17276772 ] LinZhongwei commented on HIVE-24711: Our hive metastore just enable storage based authorization. And I find these error messages in the hivemetastore.log. Caused by: java.security.AccessControlException: Permission denied: user=hive, access=WRITE, inode="/apps/finance/fdop/fdop_stg/fdop_ft_etl_stg/batch_date=2020-07-07/batch_seq_num=5":gp_fin_fdop_batch:gp_fin_fdop_batch:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:261) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1859) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1843) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1793) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAccess(FSNamesystem.java:7804) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.checkAccess(NameNodeRpcServer.java:2217) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.checkAccess(ClientNamenodeProtocolServerSideTranslatorPB.java:1659) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) at org.apache.hadoop.hive.shims.Hadoop23Shims.wrapAccessException(Hadoop23Shims.java:947) ~[hive-exec-3.1.0.3.1.5.31-1.jar:3.1.0.3.1.5.31-1] at org.apache.hadoop.hive.shims.Hadoop23Shims.checkFileAccess(Hadoop23Shims.java:931) ~[hive-exec-3.1.0.3.1.5.31-1.jar:3.1.0.3.1.5.31-1] at org.apache.hadoop.hive.common.FileUtils.checkFileAccessWithImpersonation(FileUtils.java:402) ~[hive-common-3.1.0.3.1.5.31-1.jar:3.1.0.3.1.5.31-1] at org.apache.hadoop.hive.common.FileUtils.checkFileAccessWithImpersonation(FileUtils.java:370) ~[hive-common-3.1.0.3.1.5.31-1.jar:3.1.0.3.1.5.31-1] > hive metastore memory leak > -- > > Key: HIVE-24711 > URL: https://issues.apache.org/jira/browse/HIVE-24711 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Affects Versions: 3.1.0 >Reporter: LinZhongwei >Priority: Major > > hdp version:3.1.5.31-1 > hive version:3.1.0.3.1.5.31-1 > hadoop version:3.1.1.3.1.5.31-1 > We find that the hive metastore has memory leak if we set > compactor.initiator.on to true. > If we disable the configuration, the memory leak disappear. > How can we resolve this problem? > Even if we set the heap size of hive metastore to 40 GB, after 1 month the > hive metastore service will be down with outofmemory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24711) hive metastore memory leak
[ https://issues.apache.org/jira/browse/HIVE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276407#comment-17276407 ] Karen Coppage commented on HIVE-24711: -- Do you see an impersonation (ugi.doAs()) failure in compactor.Initiator in HMS logs? If so, HIVE-22700 will help. > hive metastore memory leak > -- > > Key: HIVE-24711 > URL: https://issues.apache.org/jira/browse/HIVE-24711 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Affects Versions: 3.1.0 >Reporter: LinZhongwei >Priority: Major > > hdp version:3.1.5.31-1 > hive version:3.1.0.3.1.5.31-1 > hadoop version:3.1.1.3.1.5.31-1 > We find that the hive metastore has memory leak if we set > compactor.initiator.on to true. > If we disable the configuration, the memory leak disappear. > How can we resolve this problem? > Even if we set the heap size of hive metastore to 40 GB, after 1 month the > hive metastore service will be down with outofmemory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
