[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[ https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179202#comment-15179202 ] Hive QA commented on HIVE-13129: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12790941/HIVE-13129.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9766 tests executed *Failed tests:* {noformat} TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7157/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7157/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7157/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12790941 - PreCommit-HIVE-TRUNK-Build > CliService leaks HMS connection > --- > > Key: HIVE-13129 > URL: https://issues.apache.org/jira/browse/HIVE-13129 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13129.2.patch, HIVE-13129.patch > > > HIVE-12790 fixes the HMS connection leaking. But seems there is one more > connection from CLIService. > The init() function in CLIService will get info from DB but we never close > the HMS connection for this service main thread. > {noformat} > // creates connection to HMS and thus *must* occur after kerberos login > above > try { > applyAuthorizationConfigPolicy(hiveConf); > } catch (Exception e) { > throw new RuntimeException("Error applying authorization policy on hive > configuration: " > + e.getMessage(), e); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[ https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177859#comment-15177859 ] Chaoyu Tang commented on HIVE-13129: +1 > CliService leaks HMS connection > --- > > Key: HIVE-13129 > URL: https://issues.apache.org/jira/browse/HIVE-13129 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13129.2.patch, HIVE-13129.patch > > > HIVE-12790 fixes the HMS connection leaking. But seems there is one more > connection from CLIService. > The init() function in CLIService will get info from DB but we never close > the HMS connection for this service main thread. > {noformat} > // creates connection to HMS and thus *must* occur after kerberos login > above > try { > applyAuthorizationConfigPolicy(hiveConf); > } catch (Exception e) { > throw new RuntimeException("Error applying authorization policy on hive > configuration: " > + e.getMessage(), e); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[ https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175771#comment-15175771 ] Naveen Gangam commented on HIVE-13129: -- It makes sense to attach the HMS connection lifecycle to the service's lifecycle. +1 for me pending pre-commit tests. > CliService leaks HMS connection > --- > > Key: HIVE-13129 > URL: https://issues.apache.org/jira/browse/HIVE-13129 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13129.2.patch, HIVE-13129.patch > > > HIVE-12790 fixes the HMS connection leaking. But seems there is one more > connection from CLIService. > The init() function in CLIService will get info from DB but we never close > the HMS connection for this service main thread. > {noformat} > // creates connection to HMS and thus *must* occur after kerberos login > above > try { > applyAuthorizationConfigPolicy(hiveConf); > } catch (Exception e) { > throw new RuntimeException("Error applying authorization policy on hive > configuration: " > + e.getMessage(), e); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[ https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169470#comment-15169470 ] Aihua Xu commented on HIVE-13129: - If it's used, it's OK. But right now it's not used after that. > CliService leaks HMS connection > --- > > Key: HIVE-13129 > URL: https://issues.apache.org/jira/browse/HIVE-13129 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13129.patch > > > HIVE-12790 fixes the HMS connection leaking. But seems there is one more > connection from CLIService. > The init() function in CLIService will get info from DB but we never close > the HMS connection for this service main thread. > {noformat} > // creates connection to HMS and thus *must* occur after kerberos login > above > try { > applyAuthorizationConfigPolicy(hiveConf); > } catch (Exception e) { > throw new RuntimeException("Error applying authorization policy on hive > configuration: " > + e.getMessage(), e); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[ https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169312#comment-15169312 ] Chaoyu Tang commented on HIVE-13129: IMO, it should not be a big problem. This HMS connection can be used to serve possible HMS requests related to HS2 itself, it can live as long as HS2 does. > CliService leaks HMS connection > --- > > Key: HIVE-13129 > URL: https://issues.apache.org/jira/browse/HIVE-13129 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13129.patch > > > HIVE-12790 fixes the HMS connection leaking. But seems there is one more > connection from CLIService. > The init() function in CLIService will get info from DB but we never close > the HMS connection for this service main thread. > {noformat} > // creates connection to HMS and thus *must* occur after kerberos login > above > try { > applyAuthorizationConfigPolicy(hiveConf); > } catch (Exception e) { > throw new RuntimeException("Error applying authorization policy on hive > configuration: " > + e.getMessage(), e); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[ https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169084#comment-15169084 ] Aihua Xu commented on HIVE-13129: - When HiveServer2 stops, eventually the connection will be released. But if the server keeps running, we will always have that connection to HMS during the lifetime while it never gets used after that. And we will have one connection for each HiveServer2 instance. That can be considered as leaking I guess. > CliService leaks HMS connection > --- > > Key: HIVE-13129 > URL: https://issues.apache.org/jira/browse/HIVE-13129 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13129.patch > > > HIVE-12790 fixes the HMS connection leaking. But seems there is one more > connection from CLIService. > The init() function in CLIService will get info from DB but we never close > the HMS connection for this service main thread. > {noformat} > // creates connection to HMS and thus *must* occur after kerberos login > above > try { > applyAuthorizationConfigPolicy(hiveConf); > } catch (Exception e) { > throw new RuntimeException("Error applying authorization policy on hive > configuration: " > + e.getMessage(), e); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[ https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167726#comment-15167726 ] Hive QA commented on HIVE-13129: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12789598/HIVE-13129.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9826 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7091/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7091/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7091/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12789598 - PreCommit-HIVE-TRUNK-Build > CliService leaks HMS connection > --- > > Key: HIVE-13129 > URL: https://issues.apache.org/jira/browse/HIVE-13129 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13129.patch > > > HIVE-12790 fixes the HMS connection leaking. But seems there is one more > connection from CLIService. > The init() function in CLIService will get info from DB but we never close > the HMS connection for this service main thread. > {noformat} > // creates connection to HMS and thus *must* occur after kerberos login > above > try { > applyAuthorizationConfigPolicy(hiveConf); > } catch (Exception e) { > throw new RuntimeException("Error applying authorization policy on hive > configuration: " > + e.getMessage(), e); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[ https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167565#comment-15167565 ] Aihua Xu commented on HIVE-13129: - Only one HMS connection will be created but never released and not used later. It will accumulate leaking HMS resource and database resource if HiveServer keeps restarting. > CliService leaks HMS connection > --- > > Key: HIVE-13129 > URL: https://issues.apache.org/jira/browse/HIVE-13129 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13129.patch > > > HIVE-12790 fixes the HMS connection leaking. But seems there is one more > connection from CLIService. > The init() function in CLIService will get info from DB but we never close > the HMS connection for this service main thread. > {noformat} > // creates connection to HMS and thus *must* occur after kerberos login > above > try { > applyAuthorizationConfigPolicy(hiveConf); > } catch (Exception e) { > throw new RuntimeException("Error applying authorization policy on hive > configuration: " > + e.getMessage(), e); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[ https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167489#comment-15167489 ] Naveen Gangam commented on HIVE-13129: -- [~aihuaxu] This should only be creating one HMS connection, during HS2 startup, over the lifetime of the HS2. Do you see this causing an incremental leak? or are you proactively closing that HMS connection because you think its never used again? > CliService leaks HMS connection > --- > > Key: HIVE-13129 > URL: https://issues.apache.org/jira/browse/HIVE-13129 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13129.patch > > > HIVE-12790 fixes the HMS connection leaking. But seems there is one more > connection from CLIService. > The init() function in CLIService will get info from DB but we never close > the HMS connection for this service main thread. > {noformat} > // creates connection to HMS and thus *must* occur after kerberos login > above > try { > applyAuthorizationConfigPolicy(hiveConf); > } catch (Exception e) { > throw new RuntimeException("Error applying authorization policy on hive > configuration: " > + e.getMessage(), e); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[ https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163299#comment-15163299 ] Aihua Xu commented on HIVE-13129: - + [~thejas] [~ngangam] [~ctang.ma] You may have more background on this and can you guys take a look at this simple change? > CliService leaks HMS connection > --- > > Key: HIVE-13129 > URL: https://issues.apache.org/jira/browse/HIVE-13129 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13129.patch > > > HIVE-12790 fixes the HMS connection leaking. But seems there is one more > connection from CLIService. > The init() function in CLIService will get info from DB but we never close > the HMS connection for this service main thread. > {noformat} > // creates connection to HMS and thus *must* occur after kerberos login > above > try { > applyAuthorizationConfigPolicy(hiveConf); > } catch (Exception e) { > throw new RuntimeException("Error applying authorization policy on hive > configuration: " > + e.getMessage(), e); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[ https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163295#comment-15163295 ] Aihua Xu commented on HIVE-13129: - Attach the patch-1: we will close the HMS connection right after getting authorization policy since we are not using the connection in that service main thread anymore. All the incoming queries will be handled by thread pool and will use separate HMS connections. > CliService leaks HMS connection > --- > > Key: HIVE-13129 > URL: https://issues.apache.org/jira/browse/HIVE-13129 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13129.patch > > > HIVE-12790 fixes the HMS connection leaking. But seems there is one more > connection from CLIService. > The init() function in CLIService will get info from DB but we never close > the HMS connection for this service main thread. > {noformat} > // creates connection to HMS and thus *must* occur after kerberos login > above > try { > applyAuthorizationConfigPolicy(hiveConf); > } catch (Exception e) { > throw new RuntimeException("Error applying authorization policy on hive > configuration: " > + e.getMessage(), e); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)