[
https://issues.apache.org/jira/browse/HIVE-26376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563607#comment-17563607
]
Ayush Saxena commented on HIVE-26376:
-------------------------------------
Wild guess from the trace and problem. The config would be to disable
FileSystem cache.
{{fs.hdfs.impl.disable.cache}} will disable caching of FIleSystems for all HDFS
Fs, similar configs for all FileSystems is there.
So, if there is no cached FileSystem, so no memory occupancy and no OOM. But
that config is at Hadoop end
> Hive Metastore connection leak (OOM Error)
> ------------------------------------------
>
> Key: HIVE-26376
> URL: https://issues.apache.org/jira/browse/HIVE-26376
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 3.1.2
> Environment: !Screenshot 2022-07-07 at 11.52.33 AM.png!
> Reporter: Ranith Sardar
> Priority: Major
> Attachments: Screenshot 2022-07-07 at 11.52.33 AM.png
>
>
> Hive version:3.1.2
> Hive metastore heap size is 14GB, Memory Leak is happening after 4-5 days,
> hive meta-store throwing error with OOM.
> If we disable the configuration, the memory leak disappears.
> In the case of, Heap dump size 3.5GB, a large number of filesystem objects(>
> 9k instances) are being retained. It's occupying most of the heap space.
> Added snapshot from the eclipse MAT.
> Bellow are part of the stack trace for OOM error:
> {code:java}
> at
> org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;)Lorg/apache/hadoop/fs/FileStatus;
> (FileUtils.java:801)
> at
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/fs/Path;Ljava/util/EnumSet;)V
> (StorageBasedAuthorizationProvider.java:371)
> at
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;)V
> (StorageBasedAuthorizationProvider.java:346)
> at
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(Lorg/apache/hadoop/hive/metastore/api/Database;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;)V
> (StorageBasedAuthorizationProvider.java:154)
> at
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener.authorizeReadDatabase(Lorg/apache/hadoop/hive/metastore/events/PreReadDatabaseEvent;)V
> (AuthorizationPreEventListener.java:208)
> at
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener.onEvent(Lorg/apache/hadoop/hive/metastore/events/PreEventContext;)V
> (AuthorizationPreEventListener.java:153)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(Lorg/apache/hadoop/hive/metastore/events/PreEventContext;)V
> (HiveMetaStore.java:3221)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(Ljava/lang/String;)Lorg/apache/hadoop/hive/metastore/api/Database;
> (HiveMetaStore.java:1352){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
