[jira] [Commented] (HIVE-3969) Session state for hive server should be cleanup
[ https://issues.apache.org/jira/browse/HIVE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13758804#comment-13758804 ] Navis commented on HIVE-3969: - [~snarayanan] Thanks for your comment. I've just read the article. Then, Should we start with best effort strategy? What I mean is that if there is 'sun.miss.ClassLoaderUtil' then use that. If not, we might borrow some codes from hibernate (as explained http://loracular.blogspot.kr/2009/12/dynamic-class-loader-with.html) if it's not violate something but I'm not sure of license thing (LGPL v2.1?). > Session state for hive server should be cleanup > --- > > Key: HIVE-3969 > URL: https://issues.apache.org/jira/browse/HIVE-3969 > Project: Hive > Issue Type: Bug > Components: Server Infrastructure >Reporter: Navis >Assignee: Navis >Priority: Trivial > Attachments: HIVE-3969.D8325.1.patch > > > Currently "add jar" command by clients are adding child ClassLoader to worker > thread cumulatively, causing various problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3969) Session state for hive server should be cleanup
[ https://issues.apache.org/jira/browse/HIVE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755999#comment-13755999 ] Sivaramakrishnan Narayanan commented on HIVE-3969: -- [~navis] please see my blog on the subject [here|http://www.qubole.com/blog/qubole-hive-server/]. It is insufficient to reset the classloader as this doesn't close connections to jar files. I've pasted the relevant part of the blog here: There was one issue that was a little perplexing. After running for a week or so, QHS (Qubole Hive Server) starting throwing “too many files open” exceptions. A quick lsof call confirmed that there were numerous open file handles. Surprisingly, though, these all pointed to jar files. After some investigation, we found that the URLClassLoader leaks file handles to jars it opens (see this [link|http://management-platform.blogspot.in/2009/01/classloaders-keeping-jar-files-open.html] for some dirty details). These are never garbage collected. We ended up using the non-standard ClassLoaderUtil.releaseLoader to free up resources. Java 7 has a nicer solution for this where URLClassLoader has a close method that performs the necessary cleanup. > Session state for hive server should be cleanup > --- > > Key: HIVE-3969 > URL: https://issues.apache.org/jira/browse/HIVE-3969 > Project: Hive > Issue Type: Bug > Components: Server Infrastructure >Reporter: Navis >Assignee: Navis >Priority: Trivial > Attachments: HIVE-3969.D8325.1.patch > > > Currently "add jar" command by clients are adding child ClassLoader to worker > thread cumulatively, causing various problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3969) Session state for hive server should be cleanup
[ https://issues.apache.org/jira/browse/HIVE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749712#comment-13749712 ] Ashutosh Chauhan commented on HIVE-3969: Now that HS2 is committed which I believe does clean up its state between different sessions, this should no longer be a problem. Or do you still see this leak even with HS2? > Session state for hive server should be cleanup > --- > > Key: HIVE-3969 > URL: https://issues.apache.org/jira/browse/HIVE-3969 > Project: Hive > Issue Type: Bug > Components: Server Infrastructure >Reporter: Navis >Assignee: Navis >Priority: Trivial > Attachments: HIVE-3969.D8325.1.patch > > > Currently "add jar" command by clients are adding child ClassLoader to worker > thread cumulatively, causing various problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3969) Session state for hive server should be cleanup
[ https://issues.apache.org/jira/browse/HIVE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569942#comment-13569942 ] Navis commented on HIVE-3969: - If a client add jar files for initializing, hive server makes a new classloader and attach it to thread as a context loader. Cause hive server uses thread pool and there is no cleanup for threads, number of class loader increases constantly, which can make possibly a memory issue. And also it prevents using newer library containing class with same name, which means hive server should be restarted for reflecting new library. But cause current hive server cannot sustain consistent session between invocations from single client, this patch can induce other problems, which makes it incomplete. I know HiveServer2 is already on a way to be committed. Would it be better to fix session problems in hiveserver also or to make it deprecated quickly after hiveserver2 is committed? I don't know. > Session state for hive server should be cleanup > --- > > Key: HIVE-3969 > URL: https://issues.apache.org/jira/browse/HIVE-3969 > Project: Hive > Issue Type: Bug > Components: Server Infrastructure >Reporter: Navis >Assignee: Navis >Priority: Trivial > Attachments: HIVE-3969.D8325.1.patch > > > Currently "add jar" command by clients are adding child ClassLoader to worker > thread cumulatively, causing various problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3969) Session state for hive server should be cleanup
[ https://issues.apache.org/jira/browse/HIVE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569842#comment-13569842 ] Ashutosh Chauhan commented on HIVE-3969: Since there are no testcases, it will help if you can describe what specific problems this results in. > Session state for hive server should be cleanup > --- > > Key: HIVE-3969 > URL: https://issues.apache.org/jira/browse/HIVE-3969 > Project: Hive > Issue Type: Bug > Components: Server Infrastructure >Reporter: Navis >Assignee: Navis >Priority: Trivial > Attachments: HIVE-3969.D8325.1.patch > > > Currently "add jar" command by clients are adding child ClassLoader to worker > thread cumulatively, causing various problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
