[jira] [Commented] (HIVE-6364) HiveServer2 - Request serving thread should get class loader from existing SessionState
[ https://issues.apache.org/jira/browse/HIVE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941027#comment-13941027 ] Ashutosh Chauhan commented on HIVE-6364: I think [~navis] 's patch on HIVE-3969 is more complete and subsumes current patch. So, I guess we should try to get that one and close this one. Although, [~jaideepdhok] 's [earlier point | https://issues.apache.org/jira/browse/HIVE-6364?focusedCommentId=13892132&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13892132] about jars on a thread being visible across session is still there. Though, I am not sure how will that negatively impact query execution and way to fix it. That probably can be taken up as follow-up of HIVE-3969 > HiveServer2 - Request serving thread should get class loader from existing > SessionState > --- > > Key: HIVE-6364 > URL: https://issues.apache.org/jira/browse/HIVE-6364 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Jaideep Dhok > Attachments: HIVE-6364.1.patch > > > SessionState is created for each session in HS2. If we do any add jars, a > class loader is set in the SessionState's conf object. This class loader > should also be set in each thread that serves request of the same session. > Scenario (both requests are in the same session)- > {noformat} > // req 1 > add jar foo.jar // Served by thread th1, this updates class loader and sets > in SessionState.conf > // req2 served by th2, such that th1 != th2 > CREATE TEMPORARY FUNCTION foo_udf AS 'some class in foo.jar' > // This can throw class not found error, because although > // the new thread (th2) gets the same session state as th1, > // the class loader is different (Thread.currentThread.getContextClassLoader() > {noformat} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6364) HiveServer2 - Request serving thread should get class loader from existing SessionState
[ https://issues.apache.org/jira/browse/HIVE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13940018#comment-13940018 ] Jaideep Dhok commented on HIVE-6364: [~ashutoshc] I will put up a new patch. [~jdere] Add jar will always update the class loader. That's the current behaviour. I think the first class loader is set using the conf.getClassLoader method, if nothing is set it will return the default class loader. > HiveServer2 - Request serving thread should get class loader from existing > SessionState > --- > > Key: HIVE-6364 > URL: https://issues.apache.org/jira/browse/HIVE-6364 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Jaideep Dhok > Attachments: HIVE-6364.1.patch > > > SessionState is created for each session in HS2. If we do any add jars, a > class loader is set in the SessionState's conf object. This class loader > should also be set in each thread that serves request of the same session. > Scenario (both requests are in the same session)- > {noformat} > // req 1 > add jar foo.jar // Served by thread th1, this updates class loader and sets > in SessionState.conf > // req2 served by th2, such that th1 != th2 > CREATE TEMPORARY FUNCTION foo_udf AS 'some class in foo.jar' > // This can throw class not found error, because although > // the new thread (th2) gets the same session state as th1, > // the class loader is different (Thread.currentThread.getContextClassLoader() > {noformat} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6364) HiveServer2 - Request serving thread should get class loader from existing SessionState
[ https://issues.apache.org/jira/browse/HIVE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939592#comment-13939592 ] Jason Dere commented on HIVE-6364: -- Hi Jaideep, when I tried debugging hiveserver2 due to HIVE-6672, it appeared that there was a thread running for each connection (session). Non-SQL commands (such as ADD JAR), were being run within this session thread and so the classloader for the session thread had the JARs loaded. When a SQL command was executed the session thread would start a new thread, and it appeared that this new thread was using the same classloader (and had the added JARs in the classloader's list of URLs). Were you seeing different behavior in your testing (I was running this on Mac, I think with jdk 1.6, not sure if it would have been different)? In the patch, the thread's classloader is getting set to the HiveConf's classloader .. where is the HiveConf's classloader getting set from? Do we need to worry about having to make sure this classloader is updated whenever a JAR is added to the classpath? > HiveServer2 - Request serving thread should get class loader from existing > SessionState > --- > > Key: HIVE-6364 > URL: https://issues.apache.org/jira/browse/HIVE-6364 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Jaideep Dhok > Attachments: HIVE-6364.1.patch > > > SessionState is created for each session in HS2. If we do any add jars, a > class loader is set in the SessionState's conf object. This class loader > should also be set in each thread that serves request of the same session. > Scenario (both requests are in the same session)- > {noformat} > // req 1 > add jar foo.jar // Served by thread th1, this updates class loader and sets > in SessionState.conf > // req2 served by th2, such that th1 != th2 > CREATE TEMPORARY FUNCTION foo_udf AS 'some class in foo.jar' > // This can throw class not found error, because although > // the new thread (th2) gets the same session state as th1, > // the class loader is different (Thread.currentThread.getContextClassLoader() > {noformat} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6364) HiveServer2 - Request serving thread should get class loader from existing SessionState
[ https://issues.apache.org/jira/browse/HIVE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939467#comment-13939467 ] Ashutosh Chauhan commented on HIVE-6364: Although, HIVE-3969 is marked as duplicate, I don't think it is a duplicate. This one fixes the problem of having right class loader for a thread serving the query, whereas HIVE-3969 talks about unloading registered jars. So, it seems there are two independent problem, both of which needs to be fixed. [~jaideepdhok] would you like to rebase your patch. > HiveServer2 - Request serving thread should get class loader from existing > SessionState > --- > > Key: HIVE-6364 > URL: https://issues.apache.org/jira/browse/HIVE-6364 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Jaideep Dhok > Attachments: HIVE-6364.1.patch > > > SessionState is created for each session in HS2. If we do any add jars, a > class loader is set in the SessionState's conf object. This class loader > should also be set in each thread that serves request of the same session. > Scenario (both requests are in the same session)- > {noformat} > // req 1 > add jar foo.jar // Served by thread th1, this updates class loader and sets > in SessionState.conf > // req2 served by th2, such that th1 != th2 > CREATE TEMPORARY FUNCTION foo_udf AS 'some class in foo.jar' > // This can throw class not found error, because although > // the new thread (th2) gets the same session state as th1, > // the class loader is different (Thread.currentThread.getContextClassLoader() > {noformat} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6364) HiveServer2 - Request serving thread should get class loader from existing SessionState
[ https://issues.apache.org/jira/browse/HIVE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13892132#comment-13892132 ] Jaideep Dhok commented on HIVE-6364: There's another issue - In SessionState.registerJar class loader is obtained from the current thread. If the current thread was used during another session, then the current session can get some jars from that session. > HiveServer2 - Request serving thread should get class loader from existing > SessionState > --- > > Key: HIVE-6364 > URL: https://issues.apache.org/jira/browse/HIVE-6364 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Jaideep Dhok >Assignee: Jaideep Dhok > Attachments: HIVE-6364.1.patch > > > SessionState is created for each session in HS2. If we do any add jars, a > class loader is set in the SessionState's conf object. This class loader > should also be set in each thread that serves request of the same session. > Scenario (both requests are in the same session)- > {noformat} > // req 1 > add jar foo.jar // Served by thread th1, this updates class loader and sets > in SessionState.conf > // req2 served by th2, such that th1 != th2 > CREATE TEMPORARY FUNCTION foo_udf AS 'some class in foo.jar' > // This can throw class not found error, because although > // the new thread (th2) gets the same session state as th1, > // the class loader is different (Thread.currentThread.getContextClassLoader() > {noformat} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6364) HiveServer2 - Request serving thread should get class loader from existing SessionState
[ https://issues.apache.org/jira/browse/HIVE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890625#comment-13890625 ] Jaideep Dhok commented on HIVE-6364: Raised review board request - https://reviews.apache.org/r/17708/ > HiveServer2 - Request serving thread should get class loader from existing > SessionState > --- > > Key: HIVE-6364 > URL: https://issues.apache.org/jira/browse/HIVE-6364 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Jaideep Dhok > Attachments: HIVE-6364.1.patch > > > SessionState is created for each session in HS2. If we do any add jars, a > class loader is set in the SessionState's conf object. This class loader > should also be set in each thread that serves request of the same session. > Scenario (both requests are in the same session)- > {noformat} > // req 1 > add jar foo.jar // Served by thread th1, this updates class loader and sets > in SessionState.conf > // req2 served by th2, such that th1 != th2 > CREATE TEMPORARY FUNCTION foo_udf AS 'some class in foo.jar' > // This can throw class not found error, because although > // the new thread (th2) gets the same session state as th1, > // the class loader is different (Thread.currentThread.getContextClassLoader() > {noformat} -- This message was sent by Atlassian JIRA (v6.1.5#6160)