[
https://issues.apache.org/jira/browse/HIVE-26376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17564447#comment-17564447
]
Ayush Saxena commented on HIVE-26376:
-------------------------------------
hmm, need to explore, was trying to find if there is any related Jira in
Hadoop, found couple of them, but HDFS-3545 looked going near hive as well, so
may be related. I am also not very sure about the auth setup and all.
One more question, the hive user with which we are getting via
UserGroupInformation.getCurrentUser() is this the user with which HMS service
started or the one from the client, provided we are not using impersonation
also? if it is not of the end client, then Subject shouldn't change right?
And one more doubt as well: This is what we saw in Hive-Replication. The
FileSystem was cached and we were closing the FileSystem after shooting a
DistCp job for data copy, So, since both the threads used the same cached
FileSystem, so when one thread closed the FileSystem, the other Thread started
giving FileSystem closed exceptions during clean up task after MR jobs under
race conditions. So, this is also something we should take care, we don't land
up in such a situation. The same cached filesystem shouldn't be getting used at
more than one place, else closing at one place will screw up the other one as
well.
> Hive Metastore connection leak (OOM Error)
> ------------------------------------------
>
> Key: HIVE-26376
> URL: https://issues.apache.org/jira/browse/HIVE-26376
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 3.1.2
> Environment: !Screenshot 2022-07-07 at 11.52.33 AM.png!
> Reporter: Ranith Sardar
> Assignee: Stamatis Zampetakis
> Priority: Major
> Labels: pull-request-available
> Attachments: Screenshot 2022-07-07 at 11.52.33 AM.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive version:3.1.2
> Hive metastore heap size is 14GB, Memory Leak is happening after 4-5 days,
> hive meta-store throwing error with OOM.
> If we disable the configuration, the memory leak disappears.
> In the case of, Heap dump size 3.5GB, a large number of filesystem objects(>
> 9k instances) are being retained. It's occupying most of the heap space.
> Added snapshot from the eclipse MAT.
> Bellow are part of the stack trace for OOM error:
> {code:java}
> at
> org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;)Lorg/apache/hadoop/fs/FileStatus;
> (FileUtils.java:801)
> at
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/fs/Path;Ljava/util/EnumSet;)V
> (StorageBasedAuthorizationProvider.java:371)
> at
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;)V
> (StorageBasedAuthorizationProvider.java:346)
> at
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(Lorg/apache/hadoop/hive/metastore/api/Database;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;)V
> (StorageBasedAuthorizationProvider.java:154)
> at
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener.authorizeReadDatabase(Lorg/apache/hadoop/hive/metastore/events/PreReadDatabaseEvent;)V
> (AuthorizationPreEventListener.java:208)
> at
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener.onEvent(Lorg/apache/hadoop/hive/metastore/events/PreEventContext;)V
> (AuthorizationPreEventListener.java:153)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(Lorg/apache/hadoop/hive/metastore/events/PreEventContext;)V
> (HiveMetaStore.java:3221)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(Ljava/lang/String;)Lorg/apache/hadoop/hive/metastore/api/Database;
> (HiveMetaStore.java:1352){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
