[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[ https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435997#comment-15435997 ] Chun Chang commented on DRILL-4768: --- This has been verified. > Drill may leak hive meta store connection if hive meta store client call hits > error > --- > > Key: DRILL-4768 > URL: https://issues.apache.org/jira/browse/DRILL-4768 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > Fix For: 1.8.0 > > > We are seeing one drillbit creates hundreds of connections to hive meta > store. This indicates that drill is leaking those connection, and did not > close those connections properly. When such leaking happens, it may prevent > other applications from connecting to hive meta store. > It seems one cause of leaking connection happens when hive meta store client > call hits exception. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[
https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373355#comment-15373355
]
ASF GitHub Bot commented on DRILL-4768:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/543#discussion_r70490413
--- Diff:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java
---
@@ -129,7 +132,7 @@ public DrillHiveMetaStoreClient run() throws Exception {
*/
public static DrillHiveMetaStoreClient
createNonCloseableClientWithCaching(final HiveConf hiveConf)
--- End diff --
good catch. will modify.
> Drill may leak hive meta store connection if hive meta store client call hits
> error
> ---
>
> Key: DRILL-4768
> URL: https://issues.apache.org/jira/browse/DRILL-4768
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> We are seeing one drillbit creates hundreds of connections to hive meta
> store. This indicates that drill is leaking those connection, and did not
> close those connections properly. When such leaking happens, it may prevent
> other applications from connecting to hive meta store.
> It seems one cause of leaking connection happens when hive meta store client
> call hits exception.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[
https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373343#comment-15373343
]
ASF GitHub Bot commented on DRILL-4768:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/543#discussion_r70489111
--- Diff:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java
---
@@ -129,7 +132,7 @@ public DrillHiveMetaStoreClient run() throws Exception {
*/
public static DrillHiveMetaStoreClient
createNonCloseableClientWithCaching(final HiveConf hiveConf)
--- End diff --
Nitpick. Please rename this method to avoid confusion.
> Drill may leak hive meta store connection if hive meta store client call hits
> error
> ---
>
> Key: DRILL-4768
> URL: https://issues.apache.org/jira/browse/DRILL-4768
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> We are seeing one drillbit creates hundreds of connections to hive meta
> store. This indicates that drill is leaking those connection, and did not
> close those connections properly. When such leaking happens, it may prevent
> other applications from connecting to hive meta store.
> It seems one cause of leaking connection happens when hive meta store client
> call hits exception.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[ https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373295#comment-15373295 ] ASF GitHub Bot commented on DRILL-4768: --- Github user vkorukanti commented on the issue: https://github.com/apache/drill/pull/543 LGTM, +1. > Drill may leak hive meta store connection if hive meta store client call hits > error > --- > > Key: DRILL-4768 > URL: https://issues.apache.org/jira/browse/DRILL-4768 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > > We are seeing one drillbit creates hundreds of connections to hive meta > store. This indicates that drill is leaking those connection, and did not > close those connections properly. When such leaking happens, it may prevent > other applications from connecting to hive meta store. > It seems one cause of leaking connection happens when hive meta store client > call hits exception. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[
https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371984#comment-15371984
]
ASF GitHub Bot commented on DRILL-4768:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/543#discussion_r70361935
--- Diff:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java
---
@@ -209,8 +213,10 @@ private DrillHiveMetaStoreClient(final HiveConf
hiveConf) throws MetaException {
throws TException {
try {
return mClient.getAllTables(dbName);
+} catch (MetaException | UnknownDBException e) {
+ throw e;
} catch (TException e) {
- logger.warn("Failure while attempting to get hive tables", e);
+ logger.warn("Failure while attempting to get hive tables. Retries
once.", e);
--- End diff --
Add code to check if close() hits exception. In case close() fails with
exception, we log the exception and continue to reconnect(). This may leak
connection.
However, we have to maintain hive metastore connection. Otherwise, next
hive query may fail, and the only way to continue is either re-enable hive
storage plugin (which may then leak a connection DRILL-4774), or re-start
drillbit.
> Drill may leak hive meta store connection if hive meta store client call hits
> error
> ---
>
> Key: DRILL-4768
> URL: https://issues.apache.org/jira/browse/DRILL-4768
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> We are seeing one drillbit creates hundreds of connections to hive meta
> store. This indicates that drill is leaking those connection, and did not
> close those connections properly. When such leaking happens, it may prevent
> other applications from connecting to hive meta store.
> It seems one cause of leaking connection happens when hive meta store client
> call hits exception.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[
https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371971#comment-15371971
]
ASF GitHub Bot commented on DRILL-4768:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/543#discussion_r70360847
--- Diff:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java
---
@@ -209,8 +213,10 @@ private DrillHiveMetaStoreClient(final HiveConf
hiveConf) throws MetaException {
throws TException {
try {
return mClient.getAllTables(dbName);
+} catch (MetaException | UnknownDBException e) {
+ throw e;
} catch (TException e) {
- logger.warn("Failure while attempting to get hive tables", e);
+ logger.warn("Failure while attempting to get hive tables. Retries
once.", e);
--- End diff --
Thanks for the suggestion. I modified the code by calling close()
explicitly. However, to make it work, we also have to change the client from
nonClosable to Closable(), by removing the overriden close() method.
> Drill may leak hive meta store connection if hive meta store client call hits
> error
> ---
>
> Key: DRILL-4768
> URL: https://issues.apache.org/jira/browse/DRILL-4768
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> We are seeing one drillbit creates hundreds of connections to hive meta
> store. This indicates that drill is leaking those connection, and did not
> close those connections properly. When such leaking happens, it may prevent
> other applications from connecting to hive meta store.
> It seems one cause of leaking connection happens when hive meta store client
> call hits exception.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[
https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371931#comment-15371931
]
ASF GitHub Bot commented on DRILL-4768:
---
Github user vkorukanti commented on a diff in the pull request:
https://github.com/apache/drill/pull/543#discussion_r70358080
--- Diff:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java
---
@@ -209,8 +213,10 @@ private DrillHiveMetaStoreClient(final HiveConf
hiveConf) throws MetaException {
throws TException {
try {
return mClient.getAllTables(dbName);
+} catch (MetaException | UnknownDBException e) {
+ throw e;
} catch (TException e) {
- logger.warn("Failure while attempting to get hive tables", e);
+ logger.warn("Failure while attempting to get hive tables. Retries
once.", e);
--- End diff --
If it is Hive issue in reconnect(), can we close the client explicitly here
until we upgrade the Hive version? That way we can avoid more leaks.
> Drill may leak hive meta store connection if hive meta store client call hits
> error
> ---
>
> Key: DRILL-4768
> URL: https://issues.apache.org/jira/browse/DRILL-4768
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> We are seeing one drillbit creates hundreds of connections to hive meta
> store. This indicates that drill is leaking those connection, and did not
> close those connections properly. When such leaking happens, it may prevent
> other applications from connecting to hive meta store.
> It seems one cause of leaking connection happens when hive meta store client
> call hits exception.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[ https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371928#comment-15371928 ] ASF GitHub Bot commented on DRILL-4768: --- Github user jinfengni commented on the issue: https://github.com/apache/drill/pull/543 Revise the patch based on @vkorukanti 's comments. Pls let me know if you have any further comment. Otherwise, I'll run regression and merge the patch. Thanks. > Drill may leak hive meta store connection if hive meta store client call hits > error > --- > > Key: DRILL-4768 > URL: https://issues.apache.org/jira/browse/DRILL-4768 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > > We are seeing one drillbit creates hundreds of connections to hive meta > store. This indicates that drill is leaking those connection, and did not > close those connections properly. When such leaking happens, it may prevent > other applications from connecting to hive meta store. > It seems one cause of leaking connection happens when hive meta store client > call hits exception. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[ https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371626#comment-15371626 ] ASF GitHub Bot commented on DRILL-4768: --- Github user jinfengni commented on the issue: https://github.com/apache/drill/pull/543 @vkorukanti , thanks for pointing the place it might leaking. I'll revise the patch. Agree with @sudheeshkatkam that HiveStoragePlugin may leak connection (I saw Drill leaked a connection, when HiveStoragePlugin conf is updated each time). Since @sudheeshkatkam opened DRILL-4774, it would be addressed in DRILL-4774. The fix Sudheesh identified is in later version of Hive. Unless we move Drill's dependency forward, we need find some way to fix leaking for now. Also, when the hive client call hit tableNotFound / tableNotAccessible error, Drill does not have to reconnect. > Drill may leak hive meta store connection if hive meta store client call hits > error > --- > > Key: DRILL-4768 > URL: https://issues.apache.org/jira/browse/DRILL-4768 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > > We are seeing one drillbit creates hundreds of connections to hive meta > store. This indicates that drill is leaking those connection, and did not > close those connections properly. When such leaking happens, it may prevent > other applications from connecting to hive meta store. > It seems one cause of leaking connection happens when hive meta store client > call hits exception. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[ https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371522#comment-15371522 ] ASF GitHub Bot commented on DRILL-4768: --- Github user sudheeshkatkam commented on the issue: https://github.com/apache/drill/pull/543 Yes, we leak connections. There is a fix in Hive for [closing before reconnecting](https://github.com/apache/hive/commit/8fb805225c9d6960c7d39d862de6b3326650551c), but that is in a later version. Also, there is another leak. The [HiveStoragePlugin](https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveStoragePlugin.java) does not close connections (in HiveSchemaRegistry) when close is invoked. This should be done in an overridden close method. > Drill may leak hive meta store connection if hive meta store client call hits > error > --- > > Key: DRILL-4768 > URL: https://issues.apache.org/jira/browse/DRILL-4768 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > > We are seeing one drillbit creates hundreds of connections to hive meta > store. This indicates that drill is leaking those connection, and did not > close those connections properly. When such leaking happens, it may prevent > other applications from connecting to hive meta store. > It seems one cause of leaking connection happens when hive meta store client > call hits exception. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[ https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371499#comment-15371499 ] ASF GitHub Bot commented on DRILL-4768: --- Github user vkorukanti commented on the issue: https://github.com/apache/drill/pull/543 Change seems reasonable to me. One thing: is the reconnect() call not closing the existing connection? If yes why are we leaking the connections? I think there is one more place we need to do the same thing [1]. May be it is better move this to DrillHiveMetastoreClient. [1] https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveDatabaseSchema.java#L94 > Drill may leak hive meta store connection if hive meta store client call hits > error > --- > > Key: DRILL-4768 > URL: https://issues.apache.org/jira/browse/DRILL-4768 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > > We are seeing one drillbit creates hundreds of connections to hive meta > store. This indicates that drill is leaking those connection, and did not > close those connections properly. When such leaking happens, it may prevent > other applications from connecting to hive meta store. > It seems one cause of leaking connection happens when hive meta store client > call hits exception. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[ https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371089#comment-15371089 ] Jinfeng Ni commented on DRILL-4768: --- The cause of leaking connection is Drill will call reconnect(), whenever the hive meta store client call throws TException. This includes the case when "table not found error", or " table not allowed to be accessed", when the connection is still alive. In such cases, if we call reconnect(), then the previous connection will be leaked. > Drill may leak hive meta store connection if hive meta store client call hits > error > --- > > Key: DRILL-4768 > URL: https://issues.apache.org/jira/browse/DRILL-4768 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > > We are seeing one drillbit creates hundreds of connections to hive meta > store. This indicates that drill is leaking those connection, and did not > close those connections properly. When such leaking happens, it may prevent > other applications from connecting to hive meta store. > It seems one cause of leaking connection happens when hive meta store client > call hits exception. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[ https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371088#comment-15371088 ] ASF GitHub Bot commented on DRILL-4768: --- Github user jinfengni commented on the issue: https://github.com/apache/drill/pull/543 @sudheeshkatkam and @vkorukanti , can one of you review this PR? Thanks! > Drill may leak hive meta store connection if hive meta store client call hits > error > --- > > Key: DRILL-4768 > URL: https://issues.apache.org/jira/browse/DRILL-4768 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > > We are seeing one drillbit creates hundreds of connections to hive meta > store. This indicates that drill is leaking those connection, and did not > close those connections properly. When such leaking happens, it may prevent > other applications from connecting to hive meta store. > It seems one cause of leaking connection happens when hive meta store client > call hits exception. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4768) Drill may leak hive meta store connection if hive meta store client call hits error
[ https://issues.apache.org/jira/browse/DRILL-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371087#comment-15371087 ] ASF GitHub Bot commented on DRILL-4768: --- GitHub user jinfengni opened a pull request: https://github.com/apache/drill/pull/543 DRILL-4768: Fix leaking hive meta store connection in Drill's hive me… …tastore client call. do not call reconnect if the connection is still alive and the error is caused by either UnknownTableException or access error. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinfengni/incubator-drill DRILL-4768 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/543.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #543 commit 20a46950b37d05b7e5b0500f0d195ac7f418fe8d Author: Jinfeng Ni Date: 2016-07-08T04:48:53Z DRILL-4768: Fix leaking hive meta store connection in Drill's hive metastore client call. do not call reconnect if the connection is still alive and the error is caused by either UnknownTableException or access error. > Drill may leak hive meta store connection if hive meta store client call hits > error > --- > > Key: DRILL-4768 > URL: https://issues.apache.org/jira/browse/DRILL-4768 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > > We are seeing one drillbit creates hundreds of connections to hive meta > store. This indicates that drill is leaking those connection, and did not > close those connections properly. When such leaking happens, it may prevent > other applications from connecting to hive meta store. > It seems one cause of leaking connection happens when hive meta store client > call hits exception. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
