[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=638489&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-638489 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 17/Aug/21 03:23 Start Date: 17/Aug/21 03:23 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-899964266 Hi @vihangk1, could you please take a look if have secs? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 638489) Time Spent: 7.5h (was: 7h 20m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 7.5h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=643056&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-643056 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 28/Aug/21 04:01 Start Date: 28/Aug/21 04:01 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #2344: URL: https://github.com/apache/hive/pull/2344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 643056) Time Spent: 7h 40m (was: 7.5h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 7h 40m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=643057&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-643057 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 28/Aug/21 04:03 Start Date: 28/Aug/21 04:03 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #2344: URL: https://github.com/apache/hive/pull/2344 …properly ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 643057) Time Spent: 7h 50m (was: 7h 40m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 7h 50m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=644706&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-644706 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 01/Sep/21 01:36 Start Date: 01/Sep/21 01:36 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-909790783 Hi @pvary, @nrg4878, could you please take a look if have secs? Thanks! Zhihua Deng -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 644706) Time Spent: 8h (was: 7h 50m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 8h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=644757&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-644757
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 01/Sep/21 04:36
Start Date: 01/Sep/21 04:36
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r699837099
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -8142,9 +8146,11 @@ private void dropPartitionAllColumnGrantsNoTxn(
query.declareParameters("java.lang.String t1");
mSecurityDCList = (List) query.execute(dcName);
}
+try (Query q = query) {
pm.retrieveAll(mSecurityDCList);
Review comment:
NIT: formatting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 644757)
Time Spent: 8h 10m (was: 8h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=644758&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-644758
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 01/Sep/21 04:40
Start Date: 01/Sep/21 04:40
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r699838158
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -8142,9 +8146,11 @@ private void dropPartitionAllColumnGrantsNoTxn(
query.declareParameters("java.lang.String t1");
mSecurityDCList = (List) query.execute(dcName);
}
+try (Query q = query) {
Review comment:
We might have an error during execution. Should we close the query there
too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 644758)
Time Spent: 8h 20m (was: 8h 10m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=644760&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-644760
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 01/Sep/21 04:42
Start Date: 01/Sep/21 04:42
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r699838852
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -1454,12 +1455,14 @@ public ColumnStatistics getTableStats(final String
catName, final String dbName,
}
};
List list = Batchable.runBatched(batchSize, colNames, b);
+final ColumnStatistics result;
if (list.isEmpty()) {
- return null;
+ result = null;
+} else {
+ ColumnStatisticsDesc csd = new ColumnStatisticsDesc(true, dbName,
tableName);
+ csd.setCatName(catName);
+ result = makeColumnStats(list, csd, 0, engine);
}
Review comment:
What happens if there is an exception? Should we close the query there
too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 644760)
Time Spent: 8.5h (was: 8h 20m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=644761&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-644761 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 01/Sep/21 04:47 Start Date: 01/Sep/21 04:47 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-909880973 Thanks for catching this! We should close all the queries, even if there is an exception. Since Java 8 closing queries in `try-with-resources` would be the best if possible, falling back to `finally` when not possible is the second best. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 644761) Time Spent: 8h 40m (was: 8.5h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 8h 40m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=645012&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-645012 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 01/Sep/21 09:26 Start Date: 01/Sep/21 09:26 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-909880973 Thanks for catching this! We should close all the queries, even if there is an exception. Since Java 8 closing queries in `try-with-resources` would be the best if possible, falling back to `finally` when not possible is the second best. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 645012) Time Spent: 8h 50m (was: 8h 40m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 8h 50m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=645094&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-645094 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 01/Sep/21 09:36 Start Date: 01/Sep/21 09:36 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-909790783 Hi @pvary, @nrg4878, could you please take a look if have secs? Thanks! Zhihua Deng -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 645094) Time Spent: 9h (was: 8h 50m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 9h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=645104&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-645104
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 01/Sep/21 09:37
Start Date: 01/Sep/21 09:37
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r699837099
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -8142,9 +8146,11 @@ private void dropPartitionAllColumnGrantsNoTxn(
query.declareParameters("java.lang.String t1");
mSecurityDCList = (List) query.execute(dcName);
}
+try (Query q = query) {
pm.retrieveAll(mSecurityDCList);
Review comment:
NIT: formatting
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -8142,9 +8146,11 @@ private void dropPartitionAllColumnGrantsNoTxn(
query.declareParameters("java.lang.String t1");
mSecurityDCList = (List) query.execute(dcName);
}
+try (Query q = query) {
Review comment:
We might have an error during execution. Should we close the query there
too?
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -1454,12 +1455,14 @@ public ColumnStatistics getTableStats(final String
catName, final String dbName,
}
};
List list = Batchable.runBatched(batchSize, colNames, b);
+final ColumnStatistics result;
if (list.isEmpty()) {
- return null;
+ result = null;
+} else {
+ ColumnStatisticsDesc csd = new ColumnStatisticsDesc(true, dbName,
tableName);
+ csd.setCatName(catName);
+ result = makeColumnStats(list, csd, 0, engine);
}
Review comment:
What happens if there is an exception? Should we close the query there
too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 645104)
Time Spent: 9h 10m (was: 9h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=645214&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-645214 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 01/Sep/21 10:52 Start Date: 01/Sep/21 10:52 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-910165905 > Thanks for catching this! > > We should close all the queries, even if there is an exception. > Since Java 8 closing queries in `try-with-resources` would be the best if possible, falling back to `finally` when not possible is the second best. Thank you for the review! Introduces a new QueryWrapper to avoid superfluous exception handing when using `try-with-resources` with the class `Query` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 645214) Time Spent: 9h 20m (was: 9h 10m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 9h 20m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646128&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646128
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 03/Sep/21 01:28
Start Date: 03/Sep/21 01:28
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r701523455
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -1454,12 +1455,14 @@ public ColumnStatistics getTableStats(final String
catName, final String dbName,
}
};
List list = Batchable.runBatched(batchSize, colNames, b);
+final ColumnStatistics result;
if (list.isEmpty()) {
- return null;
+ result = null;
+} else {
+ ColumnStatisticsDesc csd = new ColumnStatisticsDesc(true, dbName,
tableName);
+ csd.setCatName(catName);
+ result = makeColumnStats(list, csd, 0, engine);
}
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646128)
Time Spent: 9h 40m (was: 9.5h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646127&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646127
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 03/Sep/21 01:28
Start Date: 03/Sep/21 01:28
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r701523382
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -8142,9 +8146,11 @@ private void dropPartitionAllColumnGrantsNoTxn(
query.declareParameters("java.lang.String t1");
mSecurityDCList = (List) query.execute(dcName);
}
+try (Query q = query) {
pm.retrieveAll(mSecurityDCList);
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646127)
Time Spent: 9.5h (was: 9h 20m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646129&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646129
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 03/Sep/21 01:29
Start Date: 03/Sep/21 01:29
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r701523603
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -8142,9 +8146,11 @@ private void dropPartitionAllColumnGrantsNoTxn(
query.declareParameters("java.lang.String t1");
mSecurityDCList = (List) query.execute(dcName);
}
+try (Query q = query) {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646129)
Time Spent: 9h 50m (was: 9h 40m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646764&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646764 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 06/Sep/21 07:41 Start Date: 06/Sep/21 07:41 Worklog Time Spent: 10m Work Description: pvary commented on a change in pull request #2344: URL: https://github.com/apache/hive/pull/2344#discussion_r702673961 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/QueryWrapper.java ## @@ -0,0 +1,430 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hive.metastore; + +import javax.jdo.Extent; +import javax.jdo.FetchPlan; +import javax.jdo.PersistenceManager; +import javax.jdo.Query; + +import java.util.Collection; +import java.util.List; +import java.util.Map; + +import static java.util.Objects.requireNonNull; + +/** + * A wrapper around the Query object to the caller and let the caller release Review comment: Why we need this class? If I understand correctly `Query` implements `AutoCloseable`. Would it be enough to use that? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646764) Time Spent: 10h (was: 9h 50m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 10h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646770&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646770
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 07:48
Start Date: 06/Sep/21 07:48
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702679874
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -1443,24 +1441,34 @@ public ColumnStatistics getTableStats(final String
catName, final String dbName,
}
long start = doTrace ? System.nanoTime() : 0;
Query query = pm.newQuery("javax.jdo.query.SQL", queryText);
-Object qResult = executeWithArray(query, params, queryText);
-MetastoreDirectSqlUtils.timingTrace(doTrace, queryText0 + "...)",
start, (doTrace ? System.nanoTime() : 0));
-if (qResult == null) {
- query.closeAll();
- return null;
+try {
Review comment:
Why are you not using the QueryWrapper here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646770)
Time Spent: 10h 10m (was: 10h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646772&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646772
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 07:51
Start Date: 06/Sep/21 07:51
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702681959
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -2107,15 +2115,17 @@ private ColumnStatisticsObj
prepareCSObjWithAdjustedNDV(Object[] row, int i,
makeParams(inputColNames.size()),
makeParams(inputPartNames.size()));
long start = doTrace ? System.nanoTime() : 0;
Query query = pm.newQuery("javax.jdo.query.SQL", queryText);
-Object qResult = executeWithArray(query, prepareParams(
-catName, dbName, tableName, inputPartNames, inputColNames,
engine), queryText);
-MetastoreDirectSqlUtils.timingTrace(doTrace, queryText0, start,
(doTrace ? System.nanoTime() : 0));
-if (qResult == null) {
- query.closeAll();
- return Collections.emptyList();
+try {
Review comment:
I think `addQueryAfterUse` is there for closing the queries. So we might
not need them anymore. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646772)
Time Spent: 10h 20m (was: 10h 10m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646774&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646774
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 07:54
Start Date: 06/Sep/21 07:54
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702683846
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -1554,41 +1553,45 @@ private boolean dropCreationMetadata(String catName,
String dbName, String table
constraintname =
constraintname!=null?normalizeIdentifier(constraintname):null;
List mConstraints = null;
List constraintNames = new ArrayList<>();
-Query query = null;
+Query queryForConstraintName = null;
+Query queryForMConstraint = null;
try {
- query = pm.newQuery("select constraintName from
org.apache.hadoop.hive.metastore.model.MConstraint where "
+ queryForConstraintName = pm.newQuery("select constraintName from
org.apache.hadoop.hive.metastore.model.MConstraint where "
Review comment:
Could we use try-with-resources here as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646774)
Time Spent: 10.5h (was: 10h 20m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646782&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646782 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 06/Sep/21 08:20 Start Date: 06/Sep/21 08:20 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on a change in pull request #2344: URL: https://github.com/apache/hive/pull/2344#discussion_r702701009 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/QueryWrapper.java ## @@ -0,0 +1,430 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hive.metastore; + +import javax.jdo.Extent; +import javax.jdo.FetchPlan; +import javax.jdo.PersistenceManager; +import javax.jdo.Query; + +import java.util.Collection; +import java.util.List; +import java.util.Map; + +import static java.util.Objects.requireNonNull; + +/** + * A wrapper around the Query object to the caller and let the caller release Review comment: Thank you for the review! The method `close()` of `Query` throws exception, we should catch the exception explicitly when using use with try-with-resources: ```java /** * Don't use this method directly; use closeAll() instead. It is intended for use with try-with-resources. * @throws Exception if this resource cannot be closed */ void close() throws Exception; ``` The exception would mix with the exception thrown in the `ObjectStore` or `MetaStoreDirectSql`, make it uneasy to handle on the caller, as I do not want to break the specified exceptions of the original method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646782) Time Spent: 10h 40m (was: 10.5h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 10h 40m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646794&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646794
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 08:36
Start Date: 06/Sep/21 08:36
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702712424
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -2107,15 +2115,17 @@ private ColumnStatisticsObj
prepareCSObjWithAdjustedNDV(Object[] row, int i,
makeParams(inputColNames.size()),
makeParams(inputPartNames.size()));
long start = doTrace ? System.nanoTime() : 0;
Query query = pm.newQuery("javax.jdo.query.SQL", queryText);
-Object qResult = executeWithArray(query, prepareParams(
-catName, dbName, tableName, inputPartNames, inputColNames,
engine), queryText);
-MetastoreDirectSqlUtils.timingTrace(doTrace, queryText0, start,
(doTrace ? System.nanoTime() : 0));
-if (qResult == null) {
- query.closeAll();
- return Collections.emptyList();
+try {
Review comment:
Maybe we should need this, `Query#execute` may return a lazy result,
e.g, the `org.datanucleus.store.rdbms.query.ForwardQueryResult`, closing the
query ealier may result to the exception `rs is closed` when
iterating/accessing the result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646794)
Time Spent: 10h 50m (was: 10h 40m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646795
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 08:38
Start Date: 06/Sep/21 08:38
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702713904
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -1443,24 +1441,34 @@ public ColumnStatistics getTableStats(final String
catName, final String dbName,
}
long start = doTrace ? System.nanoTime() : 0;
Query query = pm.newQuery("javax.jdo.query.SQL", queryText);
-Object qResult = executeWithArray(query, params, queryText);
-MetastoreDirectSqlUtils.timingTrace(doTrace, queryText0 + "...)",
start, (doTrace ? System.nanoTime() : 0));
-if (qResult == null) {
- query.closeAll();
- return null;
+try {
Review comment:
OK, I will refine this if the `QueryWrapper` seems to be a reasonable
choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646795)
Time Spent: 11h (was: 10h 50m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 11h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646796&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646796
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 08:40
Start Date: 06/Sep/21 08:40
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702712424
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -2107,15 +2115,17 @@ private ColumnStatisticsObj
prepareCSObjWithAdjustedNDV(Object[] row, int i,
makeParams(inputColNames.size()),
makeParams(inputPartNames.size()));
long start = doTrace ? System.nanoTime() : 0;
Query query = pm.newQuery("javax.jdo.query.SQL", queryText);
-Object qResult = executeWithArray(query, prepareParams(
-catName, dbName, tableName, inputPartNames, inputColNames,
engine), queryText);
-MetastoreDirectSqlUtils.timingTrace(doTrace, queryText0, start,
(doTrace ? System.nanoTime() : 0));
-if (qResult == null) {
- query.closeAll();
- return Collections.emptyList();
+try {
Review comment:
Maybe we should need this, `Query#execute` may return a lazy result,
e.g, the `org.datanucleus.store.rdbms.query.ForwardQueryResult`, closing the
query ealier may result to the exception `rs is closed` when
iterating/accessing the result later on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646796)
Time Spent: 11h 10m (was: 11h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646821&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646821
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 09:41
Start Date: 06/Sep/21 09:41
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702712424
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -2107,15 +2115,17 @@ private ColumnStatisticsObj
prepareCSObjWithAdjustedNDV(Object[] row, int i,
makeParams(inputColNames.size()),
makeParams(inputPartNames.size()));
long start = doTrace ? System.nanoTime() : 0;
Query query = pm.newQuery("javax.jdo.query.SQL", queryText);
-Object qResult = executeWithArray(query, prepareParams(
-catName, dbName, tableName, inputPartNames, inputColNames,
engine), queryText);
-MetastoreDirectSqlUtils.timingTrace(doTrace, queryText0, start,
(doTrace ? System.nanoTime() : 0));
-if (qResult == null) {
- query.closeAll();
- return Collections.emptyList();
+try {
Review comment:
Maybe we should need this, `Query#execute` may return a lazy result,
e.g, the `org.datanucleus.store.rdbms.query.ForwardQueryResult`, closing the
query ealier may result to the exception `Query result has been closed` when
iterating/accessing the result later on.
For example:
MetaException(message:Query result has been closed)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.handleDirectSqlError(ObjectStore.java:4298)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4255)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionColumnStatisticsInternal(ObjectStore.java:10079)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionColumnStatistics(ObjectStore.java:9995)
at
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testCacheUpdatePartitionColStats(TestCachedStore.java:365)
on `list.size()`
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L2135
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646821)
Time Spent: 11h 20m (was: 11h 10m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 11h 20m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646831&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646831
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 10:06
Start Date: 06/Sep/21 10:06
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r70233
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -2107,15 +2115,17 @@ private ColumnStatisticsObj
prepareCSObjWithAdjustedNDV(Object[] row, int i,
makeParams(inputColNames.size()),
makeParams(inputPartNames.size()));
long start = doTrace ? System.nanoTime() : 0;
Query query = pm.newQuery("javax.jdo.query.SQL", queryText);
-Object qResult = executeWithArray(query, prepareParams(
-catName, dbName, tableName, inputPartNames, inputColNames,
engine), queryText);
-MetastoreDirectSqlUtils.timingTrace(doTrace, queryText0, start,
(doTrace ? System.nanoTime() : 0));
-if (qResult == null) {
- query.closeAll();
- return Collections.emptyList();
+try {
Review comment:
Sounds like a bad practice, but we do not want to solve every problem
here 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646831)
Time Spent: 11.5h (was: 11h 20m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 11.5h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646863&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646863 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 06/Sep/21 10:58 Start Date: 06/Sep/21 10:58 Worklog Time Spent: 10m Work Description: pvary commented on a change in pull request #2344: URL: https://github.com/apache/hive/pull/2344#discussion_r702810683 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/QueryWrapper.java ## @@ -0,0 +1,430 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hive.metastore; + +import javax.jdo.Extent; +import javax.jdo.FetchPlan; +import javax.jdo.PersistenceManager; +import javax.jdo.Query; + +import java.util.Collection; +import java.util.List; +import java.util.Map; + +import static java.util.Objects.requireNonNull; + +/** + * A wrapper around the Query object to the caller and let the caller release Review comment: So the `QueryWrapper` is: _A wrapper around Query objects where the `close()` method is delegated to the wrapped object's `closeAll()` method. This way the users of the wrapper can use try-with-resources without exception handling_ Disclaimer: I am not a native English speaker, so some more refinement might be needed 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646863) Time Spent: 11h 40m (was: 11.5h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 11h 40m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646864&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646864
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 10:58
Start Date: 06/Sep/21 10:58
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702810861
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/QueryWrapper.java
##
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import javax.jdo.Extent;
+import javax.jdo.FetchPlan;
+import javax.jdo.PersistenceManager;
+import javax.jdo.Query;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A wrapper around the Query object to the caller and let the caller release
+ * the resources with no superfluous exception handing when using
try-with-resources.
+ */
+public class QueryWrapper implements Query {
+
+ private final Query delegate;
+
+ public QueryWrapper(Query query) {
+requireNonNull(query, "query is null");
+this.delegate = query;
+ }
+
+ public Query getQuery() {
Review comment:
We do not need this, we might want to remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646864)
Time Spent: 11h 50m (was: 11h 40m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 11h 50m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646865&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646865
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 11:00
Start Date: 06/Sep/21 11:00
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702811723
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/QueryWrapper.java
##
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import javax.jdo.Extent;
+import javax.jdo.FetchPlan;
+import javax.jdo.PersistenceManager;
+import javax.jdo.Query;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A wrapper around the Query object to the caller and let the caller release
+ * the resources with no superfluous exception handing when using
try-with-resources.
+ */
+public class QueryWrapper implements Query {
+
+ private final Query delegate;
+
+ public QueryWrapper(Query query) {
+requireNonNull(query, "query is null");
+this.delegate = query;
+ }
+
+ public Query getQuery() {
+return delegate;
+ }
+
+ @Override
+ public void close() {
Review comment:
I would put a comment here so nobody will try to "fix" this 😄
Something like:
_Delegates the method to closeAll(), so no exception is thrown_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646865)
Time Spent: 12h (was: 11h 50m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 12h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646875&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646875 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 06/Sep/21 11:15 Start Date: 06/Sep/21 11:15 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on a change in pull request #2344: URL: https://github.com/apache/hive/pull/2344#discussion_r702819881 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/QueryWrapper.java ## @@ -0,0 +1,430 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hive.metastore; + +import javax.jdo.Extent; +import javax.jdo.FetchPlan; +import javax.jdo.PersistenceManager; +import javax.jdo.Query; + +import java.util.Collection; +import java.util.List; +import java.util.Map; + +import static java.util.Objects.requireNonNull; + +/** + * A wrapper around the Query object to the caller and let the caller release Review comment: Thank you for your suggestions, that seems nicer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646875) Time Spent: 12h 10m (was: 12h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 12h 10m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646887&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646887
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 11:29
Start Date: 06/Sep/21 11:29
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702828047
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -1554,41 +1553,45 @@ private boolean dropCreationMetadata(String catName,
String dbName, String table
constraintname =
constraintname!=null?normalizeIdentifier(constraintname):null;
List mConstraints = null;
List constraintNames = new ArrayList<>();
-Query query = null;
+Query queryForConstraintName = null;
+Query queryForMConstraint = null;
try {
- query = pm.newQuery("select constraintName from
org.apache.hadoop.hive.metastore.model.MConstraint where "
+ queryForConstraintName = pm.newQuery("select constraintName from
org.apache.hadoop.hive.metastore.model.MConstraint where "
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646887)
Time Spent: 12h 20m (was: 12h 10m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 12h 20m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646889&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646889
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 11:29
Start Date: 06/Sep/21 11:29
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702828270
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/QueryWrapper.java
##
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import javax.jdo.Extent;
+import javax.jdo.FetchPlan;
+import javax.jdo.PersistenceManager;
+import javax.jdo.Query;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A wrapper around the Query object to the caller and let the caller release
+ * the resources with no superfluous exception handing when using
try-with-resources.
+ */
+public class QueryWrapper implements Query {
+
+ private final Query delegate;
+
+ public QueryWrapper(Query query) {
+requireNonNull(query, "query is null");
+this.delegate = query;
+ }
+
+ public Query getQuery() {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646889)
Time Spent: 12.5h (was: 12h 20m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 12.5h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646895&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646895
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 11:36
Start Date: 06/Sep/21 11:36
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702832318
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/QueryWrapper.java
##
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import javax.jdo.Extent;
+import javax.jdo.FetchPlan;
+import javax.jdo.PersistenceManager;
+import javax.jdo.Query;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * A wrapper around the Query object to the caller and let the caller release
+ * the resources with no superfluous exception handing when using
try-with-resources.
+ */
+public class QueryWrapper implements Query {
+
+ private final Query delegate;
+
+ public QueryWrapper(Query query) {
+requireNonNull(query, "query is null");
+this.delegate = query;
+ }
+
+ public Query getQuery() {
+return delegate;
+ }
+
+ @Override
+ public void close() {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646895)
Time Spent: 12h 40m (was: 12.5h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 12h 40m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=646926&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646926
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 12:33
Start Date: 06/Sep/21 12:33
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r702865797
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -1443,24 +1441,34 @@ public ColumnStatistics getTableStats(final String
catName, final String dbName,
}
long start = doTrace ? System.nanoTime() : 0;
Query query = pm.newQuery("javax.jdo.query.SQL", queryText);
-Object qResult = executeWithArray(query, params, queryText);
-MetastoreDirectSqlUtils.timingTrace(doTrace, queryText0 + "...)",
start, (doTrace ? System.nanoTime() : 0));
-if (qResult == null) {
- query.closeAll();
- return null;
+try {
Review comment:
Same as below, closing the query earlier may result to exception with
`Query result has been closed`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646926)
Time Spent: 12h 50m (was: 12h 40m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 12h 50m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=648390&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648390 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 09/Sep/21 05:15 Start Date: 09/Sep/21 05:15 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-915770438 Hi @pvary, any comments about the latest changes? 😄 Thanks, Zhihua Deng -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648390) Time Spent: 13h (was: 12h 50m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 13h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=648484&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648484
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 09/Sep/21 09:41
Start Date: 09/Sep/21 09:41
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r705165285
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -1850,8 +1819,6 @@ private long partsFoundForPartitions(
}
Deadline.checkTimeout();
}
- } catch (Exception e) {
-throwMetaOrRuntimeException(e);
Review comment:
Is this exception handling already happens in the callers, or we end up
throwing exceptions which were wrapped before, and will not be wrapped after
this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648484)
Time Spent: 13h 10m (was: 13h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 13h 10m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=648489&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648489 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 09/Sep/21 09:46 Start Date: 09/Sep/21 09:46 Worklog Time Spent: 10m Work Description: pvary commented on a change in pull request #2344: URL: https://github.com/apache/hive/pull/2344#discussion_r705169272 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java ## @@ -2125,13 +2080,12 @@ private ColumnStatisticsObj prepareCSObjWithAdjustedNDV(Object[] row, int i, } } }; -List list = Batchable.runBatched(batchSize, colNames, b); -List result = new ArrayList( -Math.min(list.size(), partNames.size())); +List result = new ArrayList(); Review comment: nit: I think we can use the `partNames().size()` to initialise the size of the list -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648489) Time Spent: 13h 20m (was: 13h 10m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 13h 20m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=648536&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648536
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 09/Sep/21 11:47
Start Date: 09/Sep/21 11:47
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r705251148
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -1850,8 +1819,6 @@ private long partsFoundForPartitions(
}
Deadline.checkTimeout();
}
- } catch (Exception e) {
-throwMetaOrRuntimeException(e);
Review comment:
The exception handling has aready happened in the caller, it's allowing
to throw MetaException or RuntimeException.
The catch-clause is to catch the exception of releasing query with
try-with-resources, as it throws the `Exception`, which should be transformed
to `MetaException` or `RuntimeException`. This is what the `QueryWrapper` wants
to solve, we do not need this after the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648536)
Time Spent: 13.5h (was: 13h 20m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 13.5h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=648539&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648539 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 09/Sep/21 11:53 Start Date: 09/Sep/21 11:53 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on a change in pull request #2344: URL: https://github.com/apache/hive/pull/2344#discussion_r705255284 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java ## @@ -2125,13 +2080,12 @@ private ColumnStatisticsObj prepareCSObjWithAdjustedNDV(Object[] row, int i, } } }; -List list = Batchable.runBatched(batchSize, colNames, b); -List result = new ArrayList( -Math.min(list.size(), partNames.size())); +List result = new ArrayList(); Review comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648539) Time Spent: 13h 40m (was: 13.5h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 13h 40m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=653932&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653932 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 22/Sep/21 07:01 Start Date: 22/Sep/21 07:01 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #2344: URL: https://github.com/apache/hive/pull/2344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 653932) Time Spent: 13h 50m (was: 13h 40m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 13h 50m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=653934&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653934 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 22/Sep/21 07:04 Start Date: 22/Sep/21 07:04 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #2344: URL: https://github.com/apache/hive/pull/2344 …properly ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 653934) Time Spent: 14h (was: 13h 50m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 14h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=658213&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-658213 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 30/Sep/21 09:05 Start Date: 30/Sep/21 09:05 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-931094725 Hi @pvary, cloud you help merge this? thank you! :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 658213) Time Spent: 14h 10m (was: 14h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 14h 10m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=658731&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-658731 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 01/Oct/21 05:16 Start Date: 01/Oct/21 05:16 Worklog Time Spent: 10m Work Description: pvary edited a comment on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-931905835 > Hi @pvary, cloud you help merge this? thank you! :) Ahh... Sorry for forgetting about this review. Could you please ping @belugabehr to check if he is OK with the current solution? Also we will need to rebase / rerun the tests. I see that theoretically it can be merged without a conflict, but I have had headache several times before with merging changes with outdated tests which caused problems for the next contributor. Thanks for keeping this PR alive @dengzhhu653! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 658731) Time Spent: 14.5h (was: 14h 20m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 14.5h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=658730&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-658730 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 01/Oct/21 05:16 Start Date: 01/Oct/21 05:16 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-931905835 > Hi @pvary, cloud you help merge this? thank you! :) Ahh... Sorry for forgetting about this review. Could you please ping @belugabehr to check if he is OK with the current solution? Also we will need to rebase / rerun the tests. I see that theoretically it can be merge without a conflict, but I have had headache several times merging changes with outdated tests which caused problems for the next contributor. Thanks for keeping this PR alive @dengzhhu653! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 658730) Time Spent: 14h 20m (was: 14h 10m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 14h 20m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=659379&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-659379 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 04/Oct/21 00:32 Start Date: 04/Oct/21 00:32 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-933060720 > > Hi @pvary, cloud you help merge this? thank you! :) > > Ahh... Sorry for forgetting about this review. Could you please ping @belugabehr to check if he is OK with the current solution? > > Also we will need to rebase / rerun the tests. I see that theoretically it can be merged without a conflict, but I have had headache several times before with merging changes with outdated tests which caused problems for the next contributor. > > Thanks for keeping this PR alive @dengzhhu653! Thank you very much for the feedback and sorry for the late reply, I will rebase the commits. Hey @belugabehr, cloud you please take another look if have secs? The issue can be seen in long live shared Metastore client(spark, client pool), the unreleased queries would accumulate to seeing the OOM. The former patch we applied to Hive version 1.2 solved it, this patch introduces try-with-resource on the query and a sanity check on the master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 659379) Time Spent: 14h 40m (was: 14.5h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 14h 40m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=659380&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-659380 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 04/Oct/21 00:39 Start Date: 04/Oct/21 00:39 Worklog Time Spent: 10m Work Description: dengzhhu653 edited a comment on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-933060720 > > Hi @pvary, cloud you help merge this? thank you! :) > > Ahh... Sorry for forgetting about this review. Could you please ping @belugabehr to check if he is OK with the current solution? > > Also we will need to rebase / rerun the tests. I see that theoretically it can be merged without a conflict, but I have had headache several times before with merging changes with outdated tests which caused problems for the next contributor. > > Thanks for keeping this PR alive @dengzhhu653! Thank you very much for the feedback and sorry for the late reply, I will rebase the commits. Hey @belugabehr, cloud you please take another look if have secs? The issue can be seen in the long live shared Metastore client(spark, client pool), the unreleased queries would accumulate to seeing the OOM. The former patch we applied to Hive version 1.2 solved it, this patch introduces try-with-resource on the query and a sanity check on the master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 659380) Time Spent: 14h 50m (was: 14h 40m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 14h 50m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=659630&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-659630
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 04/Oct/21 15:08
Start Date: 04/Oct/21 15:08
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r721458020
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -368,20 +363,22 @@ private void executeNoResult(final String queryText)
throws SQLException {
}
public Database getDatabase(String catName, String dbName) throws
MetaException{
-Query queryDbSelector = null;
-Query queryDbParams = null;
-try {
+String queryTextDbSelector= "select "
++ "\"DB_ID\", \"NAME\", \"DB_LOCATION_URI\", \"DESC\", "
++ "\"OWNER_NAME\", \"OWNER_TYPE\", \"CTLG_NAME\" , \"CREATE_TIME\",
\"DB_MANAGED_LOCATION_URI\", "
++ "\"TYPE\", \"DATACONNECTOR_NAME\", \"REMOTE_DBNAME\""
++ "FROM "+ DBS
++ " where \"NAME\" = ? and \"CTLG_NAME\" = ? ";
+String queryTextDbParams = "select \"PARAM_KEY\", \"PARAM_VALUE\" "
++ " from " + DATABASE_PARAMS + " "
++ " WHERE \"DB_ID\" = ? "
++ " AND \"PARAM_KEY\" IS NOT NULL";
Review comment:
Not related to this PR directly, but there *has* to be a better way of
doing this. This should come from an external resource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 659630)
Time Spent: 15h (was: 14h 50m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 15h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=659637&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-659637 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 04/Oct/21 15:17 Start Date: 04/Oct/21 15:17 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-933589151 I'm a little confused here. What version of JDO is Hive using? It looks to me like `Query` should already be `Autocloesable` https://github.com/datanucleus/javax.jdo/blob/fa061bf68e347b8fafca369ab589b7625d8579f3/src/main/java/javax/jdo/Query.java#L68 Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 659637) Time Spent: 15h 10m (was: 15h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 15h 10m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=659638&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-659638 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 04/Oct/21 15:18 Start Date: 04/Oct/21 15:18 Worklog Time Spent: 10m Work Description: belugabehr edited a comment on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-933589151 I'm a little confused here. What version of JDO is Hive using? It looks to me like `Query` should already be `Autocloseable` https://github.com/datanucleus/javax.jdo/blob/fa061bf68e347b8fafca369ab589b7625d8579f3/src/main/java/javax/jdo/Query.java#L68 Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 659638) Time Spent: 15h 20m (was: 15h 10m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 15h 20m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=659711&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-659711 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 04/Oct/21 16:49 Start Date: 04/Oct/21 16:49 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-933667212 > I'm a little confused here. What version of JDO is Hive using? > > It looks to me like `Query` should already be `Autocloseable` > > https://github.com/datanucleus/javax.jdo/blob/fa061bf68e347b8fafca369ab589b7625d8579f3/src/main/java/javax/jdo/Query.java#L68 > > Thanks. Hehe. I have run the exact same rounds with @dengzhhu653. `Autocloseable`close throws an `Exception`. We do not want to handle this individually, so we need the wrapper 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 659711) Time Spent: 15.5h (was: 15h 20m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 15.5h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=660004&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660004
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 00:30
Start Date: 05/Oct/21 00:30
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r721803993
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -368,20 +363,22 @@ private void executeNoResult(final String queryText)
throws SQLException {
}
public Database getDatabase(String catName, String dbName) throws
MetaException{
-Query queryDbSelector = null;
-Query queryDbParams = null;
-try {
+String queryTextDbSelector= "select "
++ "\"DB_ID\", \"NAME\", \"DB_LOCATION_URI\", \"DESC\", "
++ "\"OWNER_NAME\", \"OWNER_TYPE\", \"CTLG_NAME\" , \"CREATE_TIME\",
\"DB_MANAGED_LOCATION_URI\", "
++ "\"TYPE\", \"DATACONNECTOR_NAME\", \"REMOTE_DBNAME\""
++ "FROM "+ DBS
++ " where \"NAME\" = ? and \"CTLG_NAME\" = ? ";
+String queryTextDbParams = "select \"PARAM_KEY\", \"PARAM_VALUE\" "
++ " from " + DATABASE_PARAMS + " "
++ " WHERE \"DB_ID\" = ? "
++ " AND \"PARAM_KEY\" IS NOT NULL";
Review comment:
Thank you for the review, @belugabehr! My idea is to make the codes be
clean and the same pattern by replacing the standalone `close` to
try-with-resource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660004)
Time Spent: 15h 40m (was: 15.5h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 15h 40m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=660176&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660176 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 05/Oct/21 09:23 Start Date: 05/Oct/21 09:23 Worklog Time Spent: 10m Work Description: pvary merged pull request #2344: URL: https://github.com/apache/hive/pull/2344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660176) Time Spent: 15h 50m (was: 15h 40m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 15h 50m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=605682&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605682 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 03/Jun/21 06:16 Start Date: 03/Jun/21 06:16 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #2344: URL: https://github.com/apache/hive/pull/2344 …properly ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605682) Time Spent: 4.5h (was: 4h 20m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=606685&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-606685 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 04/Jun/21 08:13 Start Date: 04/Jun/21 08:13 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #2344: URL: https://github.com/apache/hive/pull/2344 …properly ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 606685) Time Spent: 4h 40m (was: 4.5h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=609266&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609266
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 09/Jun/21 17:13
Start Date: 09/Jun/21 17:13
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r648509685
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -914,6 +914,7 @@ private boolean isViewTable(String catName, String dbName,
String tblName) throw
long queryTime = doTrace ? System.nanoTime() : 0;
MetastoreDirectSqlUtils.timingTrace(doTrace, queryText, start, queryTime);
if (sqlResult.isEmpty()) {
+ query.closeAll();
return Collections.emptyList(); // no partitions, bail early.
}
Review comment:
Can you just remove this entire block? This will allow for a single
return statement at the bottom and also allows the cleanup to happen.
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -1453,6 +1454,7 @@ public ColumnStatistics getTableStats(final String
catName, final String dbName,
};
List list = Batchable.runBatched(batchSize, colNames, b);
if (list.isEmpty()) {
+ b.closeAllQueries();
return null;
}
Review comment:
Same here
```java
List list = Batchable.runBatched(batchSize, colNames, b);
final ColumnStatistics result;
if (list.isEmpty()) {
result = null;
} else {
ColumnStatisticsDesc csd = new ColumnStatisticsDesc(true, dbName,
tableName);
csd.setCatName(catName);
result = makeColumnStats(list, csd, 0, engine);
}
b.closeAllQueries();
return result;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 609266)
Time Spent: 4h 50m (was: 4h 40m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=610093&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-610093 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 14/Jun/21 07:16 Start Date: 14/Jun/21 07:16 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #2344: URL: https://github.com/apache/hive/pull/2344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 610093) Time Spent: 5h (was: 4h 50m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 5h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=610140&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-610140
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 14/Jun/21 07:21
Start Date: 14/Jun/21 07:21
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r64951
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##
@@ -1453,6 +1454,7 @@ public ColumnStatistics getTableStats(final String
catName, final String dbName,
};
List list = Batchable.runBatched(batchSize, colNames, b);
if (list.isEmpty()) {
+ b.closeAllQueries();
return null;
}
Review comment:
Appreciate that, done. Thank you very much for the review!
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -10049,9 +10053,6 @@ public boolean deletePartitionColumnStatistics(String
catName, String dbName, St
}
}
ret = commitTransaction();
-} catch (NoSuchObjectException e) {
- rollbackTransaction();
- throw e;
Review comment:
It's not related to the query closing, here just to cleanup some
unnecessary codes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 610140)
Time Spent: 5h 10m (was: 5h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=610385&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-610385 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 14/Jun/21 07:45 Start Date: 14/Jun/21 07:45 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #2344: URL: https://github.com/apache/hive/pull/2344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 610385) Time Spent: 5h 20m (was: 5h 10m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 5h 20m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=610479&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-610479
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 14/Jun/21 07:56
Start Date: 14/Jun/21 07:56
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #2344:
URL: https://github.com/apache/hive/pull/2344#discussion_r650418080
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -10049,9 +10053,6 @@ public boolean deletePartitionColumnStatistics(String
catName, String dbName, St
}
}
ret = commitTransaction();
-} catch (NoSuchObjectException e) {
- rollbackTransaction();
- throw e;
Review comment:
Hey, how is this related to closing query objects?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 610479)
Time Spent: 5.5h (was: 5h 20m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23633.01.patch
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=611362&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-611362 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 15/Jun/21 14:14 Start Date: 15/Jun/21 14:14 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #2344: URL: https://github.com/apache/hive/pull/2344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 611362) Time Spent: 5h 40m (was: 5.5h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=611363&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-611363 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 15/Jun/21 14:16 Start Date: 15/Jun/21 14:16 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #2344: URL: https://github.com/apache/hive/pull/2344 …properly ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 611363) Time Spent: 5h 50m (was: 5h 40m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=611686&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-611686 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 16/Jun/21 01:09 Start Date: 16/Jun/21 01:09 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #2344: URL: https://github.com/apache/hive/pull/2344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 611686) Time Spent: 6h (was: 5h 50m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 6h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=611687&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-611687 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 16/Jun/21 01:11 Start Date: 16/Jun/21 01:11 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #2344: URL: https://github.com/apache/hive/pull/2344 …properly ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 611687) Time Spent: 6h 10m (was: 6h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 6h 10m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=612064&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612064 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 18/Jun/21 21:15 Start Date: 18/Jun/21 21:15 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #2344: URL: https://github.com/apache/hive/pull/2344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 612064) Time Spent: 6h 20m (was: 6h 10m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 6h 20m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=612091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612091 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 18/Jun/21 21:19 Start Date: 18/Jun/21 21:19 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #2344: URL: https://github.com/apache/hive/pull/2344 …properly ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 612091) Time Spent: 6.5h (was: 6h 20m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 6.5h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=612257&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612257 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 19/Jun/21 04:31 Start Date: 19/Jun/21 04:31 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #2344: URL: https://github.com/apache/hive/pull/2344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 612257) Time Spent: 6h 40m (was: 6.5h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 6h 40m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=612258&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612258 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 19/Jun/21 04:33 Start Date: 19/Jun/21 04:33 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #2344: URL: https://github.com/apache/hive/pull/2344 …properly ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 612258) Time Spent: 6h 50m (was: 6h 40m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 6h 50m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=614043&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-614043 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 23/Jun/21 14:25 Start Date: 23/Jun/21 14:25 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-866884966 Tests passed. Hey @belugabehr @kgyrtkirk could you please take a look? :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 614043) Time Spent: 7h (was: 6h 50m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 7h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=620461&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-620461 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 08/Jul/21 11:26 Start Date: 08/Jul/21 11:26 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-876358994 Hi @kgyrtkirk could you please take a look if have secs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 620461) Time Spent: 7h 10m (was: 7h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 7h 10m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=620462&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-620462 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 08/Jul/21 11:27 Start Date: 08/Jul/21 11:27 Worklog Time Spent: 10m Work Description: dengzhhu653 removed a comment on pull request #2344: URL: https://github.com/apache/hive/pull/2344#issuecomment-866884966 Tests passed. Hey @belugabehr @kgyrtkirk could you please take a look? :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 620462) Time Spent: 7h 20m (was: 7h 10m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 7h 20m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=442623&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-442623 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 08/Jun/20 09:38 Start Date: 08/Jun/20 09:38 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #1071: URL: https://github.com/apache/hive/pull/1071 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 442623) Remaining Estimate: 0h Time Spent: 10m > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444093&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444093 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 11/Jun/20 02:43 Start Date: 11/Jun/20 02:43 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #1071: URL: https://github.com/apache/hive/pull/1071 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444093) Time Spent: 20m (was: 10m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444095&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444095 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 11/Jun/20 02:43 Start Date: 11/Jun/20 02:43 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #1071: URL: https://github.com/apache/hive/pull/1071 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444095) Time Spent: 0.5h (was: 20m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444162&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444162
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 07:43
Start Date: 11/Jun/20 07:43
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r438601029
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13272,19 +13283,18 @@ public int deleteScheduledExecutions(int
maxRetainSecs) {
return 0;
}
boolean committed = false;
+Query q = null;
try {
openTransaction();
int maxCreateTime = (int) (System.currentTimeMillis() / 1000) -
maxRetainSecs;
- Query q = pm.newQuery(MScheduledExecution.class);
+ q = pm.newQuery(MScheduledExecution.class);
Review comment:
The `Query` and `QueryWrapper` classes are `AutoCloseable` - which might
be better than closing it by hand (`Query#close` calls `closeAll` so it does
the same)
so I think it would be better to use try-with-resources to close these
objects...at least in cases when there is already a `try` there...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444162)
Time Spent: 40m (was: 0.5h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444237&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444237
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 11:55
Start Date: 11/Jun/20 11:55
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r438729662
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13272,19 +13283,18 @@ public int deleteScheduledExecutions(int
maxRetainSecs) {
return 0;
}
boolean committed = false;
+Query q = null;
try {
openTransaction();
int maxCreateTime = (int) (System.currentTimeMillis() / 1000) -
maxRetainSecs;
- Query q = pm.newQuery(MScheduledExecution.class);
+ q = pm.newQuery(MScheduledExecution.class);
Review comment:
Done. Thanks for the review, @kgyrtkirk. The try-with-resources
exception handling written here may not be beautiful.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444237)
Time Spent: 50m (was: 40m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444242&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444242 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 11/Jun/20 12:07 Start Date: 11/Jun/20 12:07 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #1071: URL: https://github.com/apache/hive/pull/1071 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444242) Time Spent: 1h (was: 50m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444243&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444243 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 11/Jun/20 12:07 Start Date: 11/Jun/20 12:07 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #1071: URL: https://github.com/apache/hive/pull/1071 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444243) Time Spent: 1h 10m (was: 1h) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444272&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444272 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 11/Jun/20 14:06 Start Date: 11/Jun/20 14:06 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #1071: URL: https://github.com/apache/hive/pull/1071 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444272) Time Spent: 1.5h (was: 1h 20m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444270&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444270 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 11/Jun/20 14:06 Start Date: 11/Jun/20 14:06 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #1071: URL: https://github.com/apache/hive/pull/1071 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444270) Time Spent: 1h 20m (was: 1h 10m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444273&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444273
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 14:09
Start Date: 11/Jun/20 14:09
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r438729662
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13272,19 +13283,18 @@ public int deleteScheduledExecutions(int
maxRetainSecs) {
return 0;
}
boolean committed = false;
+Query q = null;
try {
openTransaction();
int maxCreateTime = (int) (System.currentTimeMillis() / 1000) -
maxRetainSecs;
- Query q = pm.newQuery(MScheduledExecution.class);
+ q = pm.newQuery(MScheduledExecution.class);
Review comment:
Done. Thanks for the review, @kgyrtkirk.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444273)
Time Spent: 1h 40m (was: 1.5h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444369&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444369
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 16:13
Start Date: 11/Jun/20 16:13
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r438901224
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13317,6 +13325,10 @@ public int markScheduledExecutionsTimedOut(int
timeoutSecs) throws InvalidOperat
recoverInvalidScheduledQueryState(timeoutSecs);
committed = commitTransaction();
return results.size();
+} catch (InvalidOperationException | MetaException e) {
+ throw e;
+} catch (Exception e) {
+ throw new MetaException(e.getMessage());
Review comment:
are these catches needed?
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -12872,15 +12879,16 @@ public int deleteRuntimeStats(int maxRetainSecs)
throws MetaException {
return 0;
}
boolean committed = false;
-try {
+try (Query q = pm.newQuery(MRuntimeStat.class)) {
openTransaction();
int maxCreateTime = (int) (System.currentTimeMillis() / 1000) -
maxRetainSecs;
- Query q = pm.newQuery(MRuntimeStat.class);
q.setFilter("createTime <= maxCreateTime");
q.declareParameters("int maxCreateTime");
long deleted = q.deletePersistentAll(maxCreateTime);
committed = commitTransaction();
return (int) deleted;
+} catch (Exception e) {
+ throw new MetaException(e.getMessage());
Review comment:
is this catch needed?
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13272,15 +13280,16 @@ public int deleteScheduledExecutions(int
maxRetainSecs) {
return 0;
}
boolean committed = false;
-try {
+try (Query q = pm.newQuery(MScheduledExecution.class)) {
openTransaction();
int maxCreateTime = (int) (System.currentTimeMillis() / 1000) -
maxRetainSecs;
- Query q = pm.newQuery(MScheduledExecution.class);
q.setFilter("startTime <= maxCreateTime");
q.declareParameters("int maxCreateTime");
long deleted = q.deletePersistentAll(maxCreateTime);
committed = commitTransaction();
return (int) deleted;
+} catch (Exception e) {
Review comment:
is this catch needed?
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13339,5 +13351,6 @@ private void recoverInvalidScheduledQueryState(int
timeoutSecs) {
pm.makePersistent(e);
}
}
+q.closeAll();
Review comment:
use try-with-resources in this method
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444369)
Time Spent: 1h 50m (was: 1h 40m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444603&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444603
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 22:32
Start Date: 11/Jun/20 22:32
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439107031
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13317,6 +13325,10 @@ public int markScheduledExecutionsTimedOut(int
timeoutSecs) throws InvalidOperat
recoverInvalidScheduledQueryState(timeoutSecs);
committed = commitTransaction();
return results.size();
+} catch (InvalidOperationException | MetaException e) {
+ throw e;
+} catch (Exception e) {
+ throw new MetaException(e.getMessage());
Review comment:
yes, in my ide shows ```Unhandled exception from auto-closeable
resource: java.lang.Exception``` without the catch clause ```catch (Exception
e)``` and fails compilation. ```try (QueryWrapper wrapper = new
QueryWrapper())``` works ok here, but ```QueryWrapper``` may be going to be
removed later. I trying to replace with ```QueryWrapper```.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444603)
Time Spent: 2h (was: 1h 50m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444606&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444606
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 22:35
Start Date: 11/Jun/20 22:35
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439107695
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13272,15 +13280,16 @@ public int deleteScheduledExecutions(int
maxRetainSecs) {
return 0;
}
boolean committed = false;
-try {
+try (Query q = pm.newQuery(MScheduledExecution.class)) {
openTransaction();
int maxCreateTime = (int) (System.currentTimeMillis() / 1000) -
maxRetainSecs;
- Query q = pm.newQuery(MScheduledExecution.class);
q.setFilter("startTime <= maxCreateTime");
q.declareParameters("int maxCreateTime");
long deleted = q.deletePersistentAll(maxCreateTime);
committed = commitTransaction();
return (int) deleted;
+} catch (Exception e) {
Review comment:
The ```try (Query q = pm.newQuery(MScheduledExecution.class)) {```
throws a checked exception that can't be ignored.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444606)
Time Spent: 2h 20m (was: 2h 10m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444605&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444605
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 22:35
Start Date: 11/Jun/20 22:35
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439107695
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13272,15 +13280,16 @@ public int deleteScheduledExecutions(int
maxRetainSecs) {
return 0;
}
boolean committed = false;
-try {
+try (Query q = pm.newQuery(MScheduledExecution.class)) {
openTransaction();
int maxCreateTime = (int) (System.currentTimeMillis() / 1000) -
maxRetainSecs;
- Query q = pm.newQuery(MScheduledExecution.class);
q.setFilter("startTime <= maxCreateTime");
q.declareParameters("int maxCreateTime");
long deleted = q.deletePersistentAll(maxCreateTime);
committed = commitTransaction();
return (int) deleted;
+} catch (Exception e) {
Review comment:
```try (Query q = pm.newQuery(MScheduledExecution.class)) {``` throws a
checked exception that can't be ignored.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444605)
Time Spent: 2h 10m (was: 2h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444609&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444609
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 22:36
Start Date: 11/Jun/20 22:36
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439108243
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -12872,15 +12879,16 @@ public int deleteRuntimeStats(int maxRetainSecs)
throws MetaException {
return 0;
}
boolean committed = false;
-try {
+try (Query q = pm.newQuery(MRuntimeStat.class)) {
openTransaction();
int maxCreateTime = (int) (System.currentTimeMillis() / 1000) -
maxRetainSecs;
- Query q = pm.newQuery(MRuntimeStat.class);
q.setFilter("createTime <= maxCreateTime");
q.declareParameters("int maxCreateTime");
long deleted = q.deletePersistentAll(maxCreateTime);
committed = commitTransaction();
return (int) deleted;
+} catch (Exception e) {
+ throw new MetaException(e.getMessage());
Review comment:
A checked exception should be handled by try-with-resources
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444609)
Time Spent: 2.5h (was: 2h 20m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444611&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444611
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 22:53
Start Date: 11/Jun/20 22:53
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439113859
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13339,5 +13351,6 @@ private void recoverInvalidScheduledQueryState(int
timeoutSecs) {
pm.makePersistent(e);
}
}
+q.closeAll();
Review comment:
the method ```recoverInvalidScheduledQueryState``` originally has not a
try clause, so use ```q.closeAll()``` directly, maybe another jira is needed
to address the problem?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444611)
Time Spent: 2h 40m (was: 2.5h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444612&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444612
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 22:57
Start Date: 11/Jun/20 22:57
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439114998
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -12872,15 +12879,16 @@ public int deleteRuntimeStats(int maxRetainSecs)
throws MetaException {
return 0;
}
boolean committed = false;
-try {
+try (Query q = pm.newQuery(MRuntimeStat.class)) {
openTransaction();
int maxCreateTime = (int) (System.currentTimeMillis() / 1000) -
maxRetainSecs;
- Query q = pm.newQuery(MRuntimeStat.class);
q.setFilter("createTime <= maxCreateTime");
q.declareParameters("int maxCreateTime");
long deleted = q.deletePersistentAll(maxCreateTime);
committed = commitTransaction();
return (int) deleted;
+} catch (Exception e) {
+ throw new MetaException(e.getMessage());
Review comment:
```try (Query q = pm.newQuery(MRuntimeStat.class))``` throws a checked
exception, will use ```try (QueryWrapper wrapper = new QueryWrapper())``` to
eliminate the exception handling.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444612)
Time Spent: 2h 50m (was: 2h 40m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444613&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444613
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 23:01
Start Date: 11/Jun/20 23:01
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439107031
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13317,6 +13325,10 @@ public int markScheduledExecutionsTimedOut(int
timeoutSecs) throws InvalidOperat
recoverInvalidScheduledQueryState(timeoutSecs);
committed = commitTransaction();
return results.size();
+} catch (InvalidOperationException | MetaException e) {
+ throw e;
+} catch (Exception e) {
+ throw new MetaException(e.getMessage());
Review comment:
yes, in my ide shows ```Unhandled exception from auto-closeable
resource: java.lang.Exception``` without the catch clause ```catch (Exception
e)``` and fails compilation. ```try (QueryWrapper wrapper = new
QueryWrapper())``` works ok here, but ```QueryWrapper``` may be going to be
removed later. I trying to replace with ```QueryWrapper`` instead`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444613)
Time Spent: 3h (was: 2h 50m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444614&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444614
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 23:01
Start Date: 11/Jun/20 23:01
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439107031
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13317,6 +13325,10 @@ public int markScheduledExecutionsTimedOut(int
timeoutSecs) throws InvalidOperat
recoverInvalidScheduledQueryState(timeoutSecs);
committed = commitTransaction();
return results.size();
+} catch (InvalidOperationException | MetaException e) {
+ throw e;
+} catch (Exception e) {
+ throw new MetaException(e.getMessage());
Review comment:
yes, in my ide shows ```Unhandled exception from auto-closeable
resource: java.lang.Exception``` without the catch clause ```catch (Exception
e)``` and fails compilation. ```try (QueryWrapper wrapper = new
QueryWrapper())``` works ok here, but ```QueryWrapper``` may be going to be
removed later. I trying to replace with ```QueryWrapper``` instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444614)
Time Spent: 3h 10m (was: 3h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444620&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444620
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 23:17
Start Date: 11/Jun/20 23:17
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439113859
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13339,5 +13351,6 @@ private void recoverInvalidScheduledQueryState(int
timeoutSecs) {
pm.makePersistent(e);
}
}
+q.closeAll();
Review comment:
the method ```recoverInvalidScheduledQueryState``` originally doesn't
have a try clause, so use ```q.closeAll()``` directly, or maybe another jira
is needed to address the problem?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444620)
Time Spent: 3.5h (was: 3h 20m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444619&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444619
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 23:17
Start Date: 11/Jun/20 23:17
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439113859
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13339,5 +13351,6 @@ private void recoverInvalidScheduledQueryState(int
timeoutSecs) {
pm.makePersistent(e);
}
}
+q.closeAll();
Review comment:
the method ```recoverInvalidScheduledQueryState``` originally has not a
try clause, so use ```q.closeAll()``` directly, or maybe another jira is
needed to address the problem?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444619)
Time Spent: 3h 20m (was: 3h 10m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444692&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444692
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 12/Jun/20 02:42
Start Date: 12/Jun/20 02:42
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439113859
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13339,5 +13351,6 @@ private void recoverInvalidScheduledQueryState(int
timeoutSecs) {
pm.makePersistent(e);
}
}
+q.closeAll();
Review comment:
the method ```recoverInvalidScheduledQueryState``` originally doesn't
have a try clause, so use ```q.closeAll()``` directly, or maybe another jira
is needed to improve this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444692)
Time Spent: 3h 40m (was: 3.5h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444767&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444767
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 12/Jun/20 08:41
Start Date: 12/Jun/20 08:41
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439284688
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13317,6 +13325,10 @@ public int markScheduledExecutionsTimedOut(int
timeoutSecs) throws InvalidOperat
recoverInvalidScheduledQueryState(timeoutSecs);
committed = commitTransaction();
return results.size();
+} catch (InvalidOperationException | MetaException e) {
+ throw e;
+} catch (Exception e) {
+ throw new MetaException(e.getMessage());
Review comment:
I don't know what kind of IDE you are using; but mine(eclipse) and maven
is perfectly fine with this:
```
public static void main(String[] args) {
try (aa x = new aa()) {
System.out.println("body");
} finally {
System.out.println("final");
}
}
static class aa implements AutoCloseable {
public aa() {
throw new RuntimeException();
}
@Override
public void close() {
}
}
```
note that `aa#close` doesn't throw `Exception` while `AutoCloseable#close`
may.
EDIT: however... `Query#close` is not overriden - only the `JDOQuery`
implementation doesn't throw exception...so it will force the exception to be
catched...
I think then it would be better to go back to to the try-with-resources free
version; the repackaging of the exceptions is one thing(which happens too much;
and sometimes incorrectly around objectstore) - but now that try-with-resources
is used the the "opentransaction" is also inside the try block - if the try
fails the finally will still run; and may make the "counter" off by one... -
the best would be to rewrite that as well to use try-with-resources but I'm
aftraid that will be start a cascade of changesit seems there is already
some places which are using `openTransaction` in an interesting fashion...
sorry for the detour; but its hard to make this part better..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444767)
Time Spent: 4h (was: 3h 50m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444765&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444765
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 12/Jun/20 08:36
Start Date: 12/Jun/20 08:36
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439284688
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13317,6 +13325,10 @@ public int markScheduledExecutionsTimedOut(int
timeoutSecs) throws InvalidOperat
recoverInvalidScheduledQueryState(timeoutSecs);
committed = commitTransaction();
return results.size();
+} catch (InvalidOperationException | MetaException e) {
+ throw e;
+} catch (Exception e) {
+ throw new MetaException(e.getMessage());
Review comment:
I don't know what kind of IDE you are using; but mine(eclipse) and maven
is perfectly fine with this:
```
public static void main(String[] args) {
try (aa x = new aa()) {
System.out.println("body");
} finally {
System.out.println("final");
}
}
static class aa implements AutoCloseable {
public aa() {
throw new RuntimeException();
}
@Override
public void close() {
}
}
```
note that `aa#close` doesn't throw `Exception` while `AutoCloseable#close`
may.
but anyway - let's go back to to the try-with-resources free version; the
repackaging of the exceptions is one thing(which happens too much; and
sometimes incorrectly around objectstore) - but now that try-with-resources is
used the the "opentransaction" is also inside the try block - if the try fails
the finally will still run; and may make the "counter" off by one... - the best
would be to rewrite that as well to use try-with-resources but I'm aftraid that
will be start a cascade of changesit seems there is already some places
which are using `openTransaction` in an interesting fashion...
sorry for the detour; but its hard to make this part better..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444765)
Time Spent: 3h 50m (was: 3h 40m)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[
https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=444814&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444814
]
ASF GitHub Bot logged work on HIVE-23633:
-
Author: ASF GitHub Bot
Created on: 12/Jun/20 11:40
Start Date: 12/Jun/20 11:40
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1071:
URL: https://github.com/apache/hive/pull/1071#discussion_r439368853
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -13317,6 +13325,10 @@ public int markScheduledExecutionsTimedOut(int
timeoutSecs) throws InvalidOperat
recoverInvalidScheduledQueryState(timeoutSecs);
committed = commitTransaction();
return results.size();
+} catch (InvalidOperationException | MetaException e) {
+ throw e;
+} catch (Exception e) {
+ throw new MetaException(e.getMessage());
Review comment:
Agreed, I will update the codes lately. Thanks so much for your
feedback, @kgyrtkirk!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444814)
Time Spent: 4h 10m (was: 4h)
> Metastore some JDO query objects do not close properly
> --
>
> Key: HIVE-23633
> URL: https://issues.apache.org/jira/browse/HIVE-23633
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895],
> The metastore still has seen a memory leak on db resources: many
> StatementImpls left unclosed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=445742&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445742 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 15/Jun/20 08:57 Start Date: 15/Jun/20 08:57 Worklog Time Spent: 10m Work Description: kgyrtkirk closed pull request #1071: URL: https://github.com/apache/hive/pull/1071 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 445742) Time Spent: 4h 20m (was: 4h 10m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23633.01.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
