[jira] [Updated] (HIVE-11499) Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs
[ https://issues.apache.org/jira/browse/HIVE-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-11499: Attachment: HIVE-11499.4.patch > Datanucleus leaks classloaders when used using embedded metastore with > HiveServer2 with UDFs > > > Key: HIVE-11499 > URL: https://issues.apache.org/jira/browse/HIVE-11499 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.1.1, 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-11499.1.patch, HIVE-11499.3.patch, > HIVE-11499.4.patch, HS2-NucleusCache-Leak.tiff > > > When UDFs are used, we create a new classloader to add the UDF jar. Similar > to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the > classloaders > (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161). > JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext > reference > (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115). > Until we call NucleusContext#close, the classloader cache is not cleared. > In case of UDFs this can lead to permgen leak, as shown in the attached > screenshot, where NucleusContext holds on to several URLClassloader objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11499) Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs
[ https://issues.apache.org/jira/browse/HIVE-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-11499: Attachment: HIVE-11499.3.patch [~thejas] Patch attached incorporating your feedback. > Datanucleus leaks classloaders when used using embedded metastore with > HiveServer2 with UDFs > > > Key: HIVE-11499 > URL: https://issues.apache.org/jira/browse/HIVE-11499 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.1.1, 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-11499.1.patch, HIVE-11499.3.patch, > HS2-NucleusCache-Leak.tiff > > > When UDFs are used, we create a new classloader to add the UDF jar. Similar > to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the > classloaders > (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161). > JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext > reference > (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115). > Until we call NucleusContext#close, the classloader cache is not cleared. > In case of UDFs this can lead to permgen leak, as shown in the attached > screenshot, where NucleusContext holds on to several URLClassloader objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11499) Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs
[ https://issues.apache.org/jira/browse/HIVE-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-11499: Attachment: HIVE-11499.1.patch > Datanucleus leaks classloaders when used using embedded metastore with > HiveServer2 with UDFs > > > Key: HIVE-11499 > URL: https://issues.apache.org/jira/browse/HIVE-11499 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.1.1, 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-11499.1.patch, HS2-NucleusCache-Leak.tiff > > > When UDFs are used, we create a new classloader to add the UDF jar. Similar > to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the > classloaders > (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161). > JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext > reference > (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115). > Until we call NucleusContext#close, the classloader cache is not cleared. > In case of UDFs this can lead to permgen leak, as shown in the attached > screenshot, where NucleusContext holds on to several URLClassloader objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11499) Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs
[ https://issues.apache.org/jira/browse/HIVE-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-11499: Description: When UDFs are used, we create a new classloader to add the UDF jar. Similar to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the classloaders (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161). JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext reference (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115). Until we call NucleusContext#close, the classloader cache is not cleared. In case of UDFs this can lead to permgen leak, as shown in the attached screenshot, where NucleusContext holds on to several URLClassloader objects. (was: When UDFs are used, we create a new classloader to add the UDF jar. Similar to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the classloaders (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161). JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext reference (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115). Until we call NucleusContext#close, the classloader cache is not cleared. In case of UDFs this can lead as shows in the attached screenshot, where NucleusContext holds on to several URLClassloader objects.) > Datanucleus leaks classloaders when used using embedded metastore with > HiveServer2 with UDFs > > > Key: HIVE-11499 > URL: https://issues.apache.org/jira/browse/HIVE-11499 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.1.1, 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HS2-NucleusCache-Leak.tiff > > > When UDFs are used, we create a new classloader to add the UDF jar. Similar > to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the > classloaders > (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161). > JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext > reference > (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115). > Until we call NucleusContext#close, the classloader cache is not cleared. > In case of UDFs this can lead to permgen leak, as shown in the attached > screenshot, where NucleusContext holds on to several URLClassloader objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11499) Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs
[ https://issues.apache.org/jira/browse/HIVE-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-11499: Summary: Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs (was: Using embedded metastore with HiveServer2 leaks classloaders when used with UDFs) > Datanucleus leaks classloaders when used using embedded metastore with > HiveServer2 with UDFs > > > Key: HIVE-11499 > URL: https://issues.apache.org/jira/browse/HIVE-11499 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.1.1, 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HS2-NucleusCache-Leak.tiff > > > When UDFs are used, we create a new classloader to add the UDF jar. Similar > to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the > classloaders > (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161). > JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext > reference > (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115). > Until we call NucleusContext#close, the classloader cache is not cleared. > In case of UDFs this can lead as shows in the attached screenshot, where > NucleusContext holds on to several URLClassloader objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
