[jira] [Commented] (HIVE-25203) HiveQueryResultSet and client operation are not expected to be closed twice
[
https://issues.apache.org/jira/browse/HIVE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358014#comment-17358014
]
László Bodor commented on HIVE-25203:
-
FYI: this was a debug logging while we investigated:
{code}
1 row selected (14.303 seconds)
21/06/04 04:55:04 [main]: INFO jdbc.HiveConnection: HiveQueryResultSet.close():
org.apache.hive.jdbc.HiveStatement@523424b5
java.lang.RuntimeException
at
org.apache.hive.jdbc.HiveQueryResultSet.close(HiveQueryResultSet.java:310)
at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1041)
at org.apache.hive.beeline.Commands.execute(Commands.java:1217)
at org.apache.hive.beeline.Commands.sql(Commands.java:1146)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1497)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1355)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1134)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082)
at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
21/06/04 04:55:04 [main]: INFO jdbc.HiveConnection: HiveStatement
closeClientOperation()
21/06/04 04:55:04 [main]: INFO jdbc.HiveConnection: HiveStatement
closeStatementIfNeeded() stmtHandle:
TOperationHandle(operationId:THandleIdentifier(guid:46 7F D7 60 2F 38 4C F5 AC
6F 08 C3 AC FA 13 A2, secret:A8 C7 B8 FF C4 88 4B C0 84 27 57 C1 C5 8C 44 F3),
operationType:EXECUTE_STATEMENT, hasResultSet:true)
21/06/04 04:55:04 [main]: INFO jdbc.HiveConnection: HiveStatement
CloseOperation returned: TStatus(statusCode:SUCCESS_STATUS)
21/06/04 04:55:04 [main]: INFO jdbc.HiveConnection: HiveStatement close()
isClosed: false
21/06/04 04:55:04 [main]: INFO jdbc.HiveConnection: HiveStatement
closeClientOperation()
21/06/04 04:55:04 [main]: INFO jdbc.HiveConnection: HiveStatement
closeStatementIfNeeded() stmtHandle: null
21/06/04 04:55:04 [main]: INFO jdbc.HiveConnection: HiveQueryResultSet.close():
org.apache.hive.jdbc.HiveStatement@523424b5
java.lang.RuntimeException
at
org.apache.hive.jdbc.HiveQueryResultSet.close(HiveQueryResultSet.java:310)
at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:252)
at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1067)
at org.apache.hive.beeline.Commands.execute(Commands.java:1217)
at org.apache.hive.beeline.Commands.sql(Commands.java:1146)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1497)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1355)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1134)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082)
at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
{code}
> HiveQueryResultSet and client operation are not expected to be closed twice
> ---
>
> Key: HIVE-25203
> URL: https://issues.apache.org/jira/browse/HIVE-25203
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While testing retry scenarios of HIVE-24786, we found that
> HiveQueryResultSet.close() is called twice, which is not expected. There are
> 2 different issues here:
> 1. ResultSet should not handle Statement as in HiveQueryResultSet:
> {code}
> if (this.statement != null && (this.statement instanceof HiveStatement)) {
> HiveStatement s = (HiveStatement) this.statement;
> s.closeClientOperation();
> {code}
> The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
> ResultSet(HiveQueryResultSet) should be respected in a sense that
[jira] [Work logged] (HIVE-25203) HiveQueryResultSet and client operation are not expected to be closed twice
[
https://issues.apache.org/jira/browse/HIVE-25203?focusedWorklogId=607473=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-607473
]
ASF GitHub Bot logged work on HIVE-25203:
-
Author: ASF GitHub Bot
Created on: 06/Jun/21 04:11
Start Date: 06/Jun/21 04:11
Worklog Time Spent: 10m
Work Description: abstractdog opened a new pull request #2353:
URL: https://github.com/apache/hive/pull/2353
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 607473)
Remaining Estimate: 0h
Time Spent: 10m
> HiveQueryResultSet and client operation are not expected to be closed twice
> ---
>
> Key: HIVE-25203
> URL: https://issues.apache.org/jira/browse/HIVE-25203
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While testing retry scenarios of HIVE-24786, we found that
> HiveQueryResultSet.close() is called twice, which is not expected. There are
> 2 different issues here:
> 1. ResultSet should not handle Statement as in HiveQueryResultSet:
> {code}
> if (this.statement != null && (this.statement instanceof HiveStatement)) {
> HiveStatement s = (HiveStatement) this.statement;
> s.closeClientOperation();
> {code}
> The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
> ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
> can handle child but not the opposite way, only except a single case, where
> the state of the result set has an effect on statement's state, which is
> [Statement.closeOnCompletion|https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#closeOnCompletion()],
> which was introduced by HIVE-22698.
> The above logic was introduced by
> [HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
> Its intention was to make children able to return their parents, but that
> doesn't mean they should handle their parents' lifecycle.
> 2. Also, HiveStatement should close HiveQueryResultSet only if it's not
> already closed, so it would make sense to check ResultSet.isClosed() before
> closing. This is for the very same reason as another change above, to avoid
> duplicated close logic.
> Background: under normal circumstances, a close operation is idempotent, we
> should not worry about any side effects of calling it twice, but while
> testing HIVE-24786, we found strange issues where in case of a
> SocketTimeoutException, such code path was hit in the jdbc client, that made
> HiveStatement.closeClientOperation() to be called twice, and it led to a
> WARNING on HS2 side. This is not expected as the operation close is protected
> by stmtHandle != null check, but yet it ran twice. To avoid situations like
> this, cleaning up duplicated close calls would help.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25203) HiveQueryResultSet and client operation are not expected to be closed twice
[
https://issues.apache.org/jira/browse/HIVE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25203:
--
Labels: pull-request-available (was: )
> HiveQueryResultSet and client operation are not expected to be closed twice
> ---
>
> Key: HIVE-25203
> URL: https://issues.apache.org/jira/browse/HIVE-25203
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While testing retry scenarios of HIVE-24786, we found that
> HiveQueryResultSet.close() is called twice, which is not expected. There are
> 2 different issues here:
> 1. ResultSet should not handle Statement as in HiveQueryResultSet:
> {code}
> if (this.statement != null && (this.statement instanceof HiveStatement)) {
> HiveStatement s = (HiveStatement) this.statement;
> s.closeClientOperation();
> {code}
> The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
> ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
> can handle child but not the opposite way, only except a single case, where
> the state of the result set has an effect on statement's state, which is
> [Statement.closeOnCompletion|https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#closeOnCompletion()],
> which was introduced by HIVE-22698.
> The above logic was introduced by
> [HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
> Its intention was to make children able to return their parents, but that
> doesn't mean they should handle their parents' lifecycle.
> 2. Also, HiveStatement should close HiveQueryResultSet only if it's not
> already closed, so it would make sense to check ResultSet.isClosed() before
> closing. This is for the very same reason as another change above, to avoid
> duplicated close logic.
> Background: under normal circumstances, a close operation is idempotent, we
> should not worry about any side effects of calling it twice, but while
> testing HIVE-24786, we found strange issues where in case of a
> SocketTimeoutException, such code path was hit in the jdbc client, that made
> HiveStatement.closeClientOperation() to be called twice, and it led to a
> WARNING on HS2 side. This is not expected as the operation close is protected
> by stmtHandle != null check, but yet it ran twice. To avoid situations like
> this, cleaning up duplicated close calls would help.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25203) HiveQueryResultSet and client operation are not expected to be closed twice
[
https://issues.apache.org/jira/browse/HIVE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-25203:
Summary: HiveQueryResultSet and client operation are not expected to be
closed twice (was: HiveQueryResultSet and client operations are not expected
to be closed twice)
> HiveQueryResultSet and client operation are not expected to be closed twice
> ---
>
> Key: HIVE-25203
> URL: https://issues.apache.org/jira/browse/HIVE-25203
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> While testing retry scenarios of HIVE-24786, we found that
> HiveQueryResultSet.close() is called twice, which is not expected. There are
> 2 different issues here:
> 1. ResultSet should not handle Statement as in HiveQueryResultSet:
> {code}
> if (this.statement != null && (this.statement instanceof HiveStatement)) {
> HiveStatement s = (HiveStatement) this.statement;
> s.closeClientOperation();
> {code}
> The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
> ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
> can handle child but not the opposite way, only except a single case, where
> the state of the result set has an effect on statement's state, which is
> [Statement.closeOnCompletion|https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#closeOnCompletion()],
> which was introduced by HIVE-22698.
> The above logic was introduced by
> [HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
> Its intention was to make children able to return their parents, but that
> doesn't mean they should handle their parents' lifecycle.
> 2. Also, HiveStatement should close HiveQueryResultSet only if it's not
> already closed, so it would make sense to check ResultSet.isClosed() before
> closing. This is for the very same reason as another change above, to avoid
> duplicated close logic.
> Background: under normal circumstances, a close operation is idempotent, we
> should not worry about any side effects of calling it twice, but while
> testing HIVE-24786, we found strange issues where in case of a
> SocketTimeoutException, such code path was hit in the jdbc client, that made
> HiveStatement.closeClientOperation() to be called twice, and it led to a
> WARNING on HS2 side. This is not expected as the operation close is protected
> by stmtHandle != null check, but yet it ran twice. To avoid situations like
> this, cleaning up duplicated close calls would help.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25203) HiveQueryResultSet and client operations are not expected to be closed twice
[
https://issues.apache.org/jira/browse/HIVE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-25203:
Summary: HiveQueryResultSet and client operations are not expected to be
closed twice (was: HiveQueryResultSet is not expected to be closed twice)
> HiveQueryResultSet and client operations are not expected to be closed twice
>
>
> Key: HIVE-25203
> URL: https://issues.apache.org/jira/browse/HIVE-25203
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> While testing retry scenarios of HIVE-24786, we found that
> HiveQueryResultSet.close() is called twice, which is not expected. There are
> 2 different issues here:
> 1. ResultSet should not handle Statement as in HiveQueryResultSet:
> {code}
> if (this.statement != null && (this.statement instanceof HiveStatement)) {
> HiveStatement s = (HiveStatement) this.statement;
> s.closeClientOperation();
> {code}
> The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
> ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
> can handle child but not the opposite way, only except a single case, where
> the state of the result set has an effect on statement's state, which is
> [Statement.closeOnCompletion|https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#closeOnCompletion()],
> which was introduced by HIVE-22698.
> The above logic was introduced by
> [HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
> Its intention was to make children able to return their parents, but that
> doesn't mean they should handle their parents' lifecycle.
> 2. Also, HiveStatement should close HiveQueryResultSet only if it's not
> already closed, so it would make sense to check ResultSet.isClosed() before
> closing. This is for the very same reason as another change above, to avoid
> duplicated close logic.
> Background: under normal circumstances, a close operation is idempotent, we
> should not worry about any side effects of calling it twice, but while
> testing HIVE-24786, we found strange issues where in case of a
> SocketTimeoutException, such code path was hit in the jdbc client, that made
> HiveStatement.closeClientOperation() to be called twice, and it led to a
> WARNING on HS2 side. This is not expected as the operation close is protected
> by stmtHandle != null check, but yet it ran twice. To avoid situations like
> this, cleaning up duplicated close calls would help.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25203) HiveQueryResultSet is not expected to be closed twice
[
https://issues.apache.org/jira/browse/HIVE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-25203:
Description:
While testing retry scenarios of HIVE-24786, we found that
HiveQueryResultSet.close() is called twice, which is not expected. There are 2
different issues here:
1. ResultSet should not handle Statement as in HiveQueryResultSet:
{code}
if (this.statement != null && (this.statement instanceof HiveStatement)) {
HiveStatement s = (HiveStatement) this.statement;
s.closeClientOperation();
{code}
The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
can handle child but not the opposite way, only except a single case, where the
state of the result set has an effect on statement's state, which is
[Statement.closeOnCompletion|https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#closeOnCompletion()],
which was introduced by HIVE-22698.
The above logic was introduced by
[HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
Its intention was to make children able to return their parents, but that
doesn't mean they should handle their parents' lifecycle.
2. Also, HiveStatement should close HiveQueryResultSet only if it's not already
closed, so it would make sense to check ResultSet.isClosed() before closing.
This is for the very same reason as another change above, to avoid duplicated
close logic.
Background: under normal circumstances, a close operation is idempotent, we
should not worry about any side effects of calling it twice, but while testing
HIVE-24786, we found strange issues where in case of a SocketTimeoutException,
such code path was hit in the jdbc client, that made
HiveStatement.closeClientOperation() to be called twice, and it led to a
WARNING on HS2 side. This is not expected as the operation close is protected
by stmtHandle != null check, but yet it ran twice. To avoid situations like
this, cleaning up duplicated close calls would help.
was:
While testing retry scenarios of HIVE-24786, we found that
HiveQueryResultSet.close() is called twice, which is not expected. There are 2
different issues here:
1. ResultSet should not handle Statement as in HiveQueryResultSet:
{code}
if (this.statement != null && (this.statement instanceof HiveStatement)) {
HiveStatement s = (HiveStatement) this.statement;
s.closeClientOperation();
{code}
The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
can handle child but not the opposite way, only except a single case, where the
state of the result set has an effect of statement's state, which is
[Statement.closeOnCompletion|https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#closeOnCompletion()],
which was introduced by HIVE-22698.
The above logic was introduced by
[HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
Its intention was to make children able to return their parents, but that
doesn't mean they should handle their parents' lifecycle.
2. Also, HiveStatement should close HiveQueryResultSet only if it's not already
closed, so it would make sense to check ResultSet.isClosed() before closing.
This is for the very same reason as another change above, to avoid duplicated
close logic.
Background: under normal circumstances, a close operation is idempotent, we
should not worry about any side effects of calling it twice, but while testing
HIVE-24786, we found strange issues where in case of a SocketTimeoutException,
such code path was hit in the jdbc client, that made
HiveStatement.closeClientOperation() to be called twice, and it led to a
WARNING on HS2 side. This is not expected as the operation close is protected
by stmtHandle != null check, but yet it ran twice. To avoid situations like
this, cleaning up duplicated close calls would help.
> HiveQueryResultSet is not expected to be closed twice
> -
>
> Key: HIVE-25203
> URL: https://issues.apache.org/jira/browse/HIVE-25203
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> While testing retry scenarios of HIVE-24786, we found that
> HiveQueryResultSet.close() is called twice, which is not expected. There are
> 2 different issues here:
> 1. ResultSet should not handle Statement as in HiveQueryResultSet:
> {code}
> if (this.statement != null && (this.statement instanceof HiveStatement)) {
> HiveStatement s = (HiveStatement) this.statement;
> s.closeClientOperation();
>
[jira] [Updated] (HIVE-25203) HiveQueryResultSet is not expected to be closed twice
[
https://issues.apache.org/jira/browse/HIVE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-25203:
Description:
While testing retry scenarios of HIVE-24786, we found that
HiveQueryResultSet.close() is called twice, which is not expected. There are 2
different issues here:
1. ResultSet should not handle Statement as in HiveQueryResultSet:
{code}
if (this.statement != null && (this.statement instanceof HiveStatement)) {
HiveStatement s = (HiveStatement) this.statement;
s.closeClientOperation();
{code}
The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
can handle child but not the opposite way, only except a single case, where the
state of the result set has an effect of statement's state, which is
[Statement.closeOnCompletion|https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#closeOnCompletion()],
which was introduced by HIVE-22698.
The above logic was introduced by
[HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
Its intention was to make children able to return their parents, but that
doesn't mean they should handle their parents' lifecycle.
2. Also, HiveStatement should close HiveQueryResultSet only if it's not already
closed, so it would make sense to check ResultSet.isClosed() before closing.
This is for the very same reason as another change above, to avoid duplicated
close logic.
Background: under normal circumstances, a close operation is idempotent, we
should not worry about any side effects of calling it twice, but while testing
HIVE-24786, we found strange issues where in case of a SocketTimeoutException,
such code path was hit in the jdbc client, that made
HiveStatement.closeClientOperation() to be called twice, and it led to a
WARNING on HS2 side. This is not expected as the operation close is protected
by stmtHandle != null check, but yet it ran twice. To avoid situations like
this, cleaning up duplicated close calls would help.
was:
While testing retry scenarios of HIVE-24786, we found that
HiveQueryResultSet.close() is called twice, which is not expected. There are 2
different issues here:
1. ResultSet should not handle Statement as in HiveQueryResultSet:
{code}
if (this.statement != null && (this.statement instanceof HiveStatement)) {
HiveStatement s = (HiveStatement) this.statement;
s.closeClientOperation();
{code}
The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
can handle child but not the opposite way.
The above logic was introduced by
[HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
Its intention was to make children able to return their parents, but that
doesn't mean they should handle their parents' lifecycle.
2. Also, HiveStatement should close HiveQueryResultSet only if it's not already
closed, so it would make sense to check ResultSet.isClosed() before closing.
This is for the very same reason as another change above, to avoid duplicated
close logic.
Background: under normal circumstances, a close operation is idempotent, we
should not worry about any side effects of calling it twice, but while testing
HIVE-24786, we found strange issues where in case of a SocketTimeoutException,
such code path was hit in the jdbc client, that made
HiveStatement.closeClientOperation() to be called twice, and it led to a
WARNING on HS2 side. This is not expected as the operation close is protected
by stmtHandle != null check, but yet it ran twice. To avoid situations like
this, cleaning up duplicated close calls would help.
> HiveQueryResultSet is not expected to be closed twice
> -
>
> Key: HIVE-25203
> URL: https://issues.apache.org/jira/browse/HIVE-25203
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> While testing retry scenarios of HIVE-24786, we found that
> HiveQueryResultSet.close() is called twice, which is not expected. There are
> 2 different issues here:
> 1. ResultSet should not handle Statement as in HiveQueryResultSet:
> {code}
> if (this.statement != null && (this.statement instanceof HiveStatement)) {
> HiveStatement s = (HiveStatement) this.statement;
> s.closeClientOperation();
> {code}
> The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
> ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
> can handle child but not the opposite way, only except a single case, where
> the state of the
[jira] [Updated] (HIVE-25203) HiveQueryResultSet is not expected to be closed twice
[
https://issues.apache.org/jira/browse/HIVE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-25203:
Description:
While testing retry scenarios of HIVE-24786, we found that
HiveQueryResultSet.close() is called twice, which is not expected. There are 2
different issues here:
1. ResultSet should not handle Statement as in HiveQueryResultSet:
{code}
if (this.statement != null && (this.statement instanceof HiveStatement)) {
HiveStatement s = (HiveStatement) this.statement;
s.closeClientOperation();
{code}
The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
can handle child but not the opposite way.
The above logic was introduced by
[HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
Its intention was to make children able to return their parents, but that
doesn't mean they should handle their parents' lifecycle.
2. Also, HiveStatement should close HiveQueryResultSet only if it's not already
closed, so it would make sense to check ResultSet.isClosed() before closing.
This is for the very same reason as another change above, to avoid duplicated
close logic.
Background: under normal circumstances, a close operation is idempotent, we
should not worry about any side effects of calling it twice, but while testing
HIVE-24786, we found strange issues where in case of a SocketTimeoutException,
such code path was hit in the jdbc client, that made
HiveStatement.closeClientOperation() to be called twice, and it led to a
WARNING on HS2 side. This is not expected as the operation close is protected
by stmtHandle != null check, but yet it ran twice. To avoid situations like
this, cleaning up duplicated close calls would help.
was:
While testing retry scenarios of HIVE-24786, we found that
HiveQueryResultSet.close() is called twice, which is not expected. There are 2
different issues here:
1. ResultSet should not handle Statement as in HiveQueryResultSet:
{code}
if (this.statement != null && (this.statement instanceof HiveStatement)) {
HiveStatement s = (HiveStatement) this.statement;
s.closeClientOperation();
{code}
The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
can handle child but not to opposite way.
The above logic was introduced by
[HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
Its intention was to make children able to return their parents, but that
doesn't mean they should handle their parents' lifecycle.
2. Also, HiveStatement should close HiveQueryResultSet only if it's not already
closed, so it would make sense to check ResultSet.isClosed() before closing.
This is for the very same reason as another change above, to avoid duplicated
close logic.
Background: under normal circumstances, a close operation is idempotent, we
should not worry about any side effects of calling it twice, but while testing
HIVE-24786, we found strange issues where in case of a SocketTimeoutException,
such code path was hit in the jdbc client, that made
HiveStatement.closeClientOperation() to be called twice, and it led to a
WARNING on HS2 side. This is not expected as the operation close is protected
by stmtHandle != null check, but yet it ran twice. To avoid situations like
this, cleaning up duplicated close calls would help.
> HiveQueryResultSet is not expected to be closed twice
> -
>
> Key: HIVE-25203
> URL: https://issues.apache.org/jira/browse/HIVE-25203
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> While testing retry scenarios of HIVE-24786, we found that
> HiveQueryResultSet.close() is called twice, which is not expected. There are
> 2 different issues here:
> 1. ResultSet should not handle Statement as in HiveQueryResultSet:
> {code}
> if (this.statement != null && (this.statement instanceof HiveStatement)) {
> HiveStatement s = (HiveStatement) this.statement;
> s.closeClientOperation();
> {code}
> The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
> ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
> can handle child but not the opposite way.
> The above logic was introduced by
> [HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
> Its intention was to make children able to return their parents, but that
> doesn't mean they should handle their parents' lifecycle.
> 2. Also,
[jira] [Updated] (HIVE-25203) HiveQueryResultSet is not expected to be closed twice
[
https://issues.apache.org/jira/browse/HIVE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-25203:
Description:
While testing retry scenarios of HIVE-24786, we found that
HiveQueryResultSet.close() is called twice, which is not expected. There are 2
different issues here:
1. ResultSet should not handle Statement as in HiveQueryResultSet:
{code}
if (this.statement != null && (this.statement instanceof HiveStatement)) {
HiveStatement s = (HiveStatement) this.statement;
s.closeClientOperation();
{code}
The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
can handle child but not to opposite way.
The above logic was introduced by
[HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
Its intention was to make children able to return their parents, but that
doesn't mean they should handle their parents' lifecycle.
2. Also, HiveStatement should close HiveQueryResultSet only if it's not already
closed, so it would make sense to check ResultSet.isClosed() before closing.
This is for the very same reason as another change above, to avoid duplicated
close logic.
Background: under normal circumstances, a close operation is idempotent, we
should not worry about any side effects of calling it twice, but while testing
HIVE-24786, we found strange issues where in case of a SocketTimeoutException,
such code path was hit in the jdbc client, that made
HiveStatement.closeClientOperation() to be called twice, and it led to a
WARNING on HS2 side. This is not expected as the operation close is protected
by stmtHandle != null check, but yet it ran twice. To avoid situations like
this, cleaning up duplicated close calls would help.
was:
While testing retry scenarios of HIVE-24786, we found that
HiveQueryResultSet.close() is called twice, which is not expected. There is 2
different issues here:
1. ResultSet should not handle Statement as in HiveQueryResultSet:
{code}
if (this.statement != null && (this.statement instanceof HiveStatement)) {
HiveStatement s = (HiveStatement) this.statement;
s.closeClientOperation();
{code}
The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
can handle child but not to opposite way.
The above logic was introduced by
[HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
Its intention was to make children able to return their parents, but that
doesn't mean they should handle their parents' lifecycle.
2. Also, HiveStatement should close HiveQueryResultSet only if it's not already
closed, so it would make sense to check ResultSet.isClosed() before closing.
This is for the very same reason as another change above, to avoid duplicated
close logic.
Background: under normal circumstances, a close operation is idempotent, we
should not worry about any side effects of calling it twice, but while testing
HIVE-24786, we found strange issues where in case of a SocketTimeoutException,
such code path was hit in the jdbc client, that made
HiveStatement.closeClientOperation() to be called twice, and it led to a
WARNING on HS2 side. This is not expected as the operation close is protected
by stmtHandle != null check, but yet it ran twice. To avoid situations like
this, cleaning up duplicated close calls would help.
> HiveQueryResultSet is not expected to be closed twice
> -
>
> Key: HIVE-25203
> URL: https://issues.apache.org/jira/browse/HIVE-25203
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> While testing retry scenarios of HIVE-24786, we found that
> HiveQueryResultSet.close() is called twice, which is not expected. There are
> 2 different issues here:
> 1. ResultSet should not handle Statement as in HiveQueryResultSet:
> {code}
> if (this.statement != null && (this.statement instanceof HiveStatement)) {
> HiveStatement s = (HiveStatement) this.statement;
> s.closeClientOperation();
> {code}
> The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
> ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
> can handle child but not to opposite way.
> The above logic was introduced by
> [HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
> Its intention was to make children able to return their parents, but that
> doesn't mean they should handle their parents' lifecycle.
> 2. Also,
[jira] [Updated] (HIVE-25203) HiveQueryResultSet is not expected to be closed twice
[
https://issues.apache.org/jira/browse/HIVE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-25203:
Description:
While testing retry scenarios of HIVE-24786, we found that
HiveQueryResultSet.close() is called twice, which is not expected. There is 2
different issues here:
1. ResultSet should not handle Statement as in HiveQueryResultSet:
{code}
if (this.statement != null && (this.statement instanceof HiveStatement)) {
HiveStatement s = (HiveStatement) this.statement;
s.closeClientOperation();
{code}
The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
can handle child but not to opposite way.
The above logic was introduced by
[HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
Its intention was to make children able to return their parents, but that
doesn't mean they should handle their parents' lifecycle.
2. Also, HiveStatement should close HiveQueryResultSet only if it's not already
closed, so it would make sense to check ResultSet.isClosed() before closing.
This is for the very same reason as another change above, to avoid duplicated
close logic.
Background: under normal circumstances, a close operation is idempotent, we
should not worry about any side effects of calling it twice, but while testing
HIVE-24786, we found strange issues where in case of a SocketTimeoutException,
such code path was hit in the jdbc client, that made
HiveStatement.closeClientOperation() to be called twice, and it led to a
WARNING on HS2 side. This is not expected as the operation close is protected
by stmtHandle != null check, but yet it ran twice. To avoid situations like
this, cleaning up duplicated close calls would help.
> HiveQueryResultSet is not expected to be closed twice
> -
>
> Key: HIVE-25203
> URL: https://issues.apache.org/jira/browse/HIVE-25203
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> While testing retry scenarios of HIVE-24786, we found that
> HiveQueryResultSet.close() is called twice, which is not expected. There is 2
> different issues here:
> 1. ResultSet should not handle Statement as in HiveQueryResultSet:
> {code}
> if (this.statement != null && (this.statement instanceof HiveStatement)) {
> HiveStatement s = (HiveStatement) this.statement;
> s.closeClientOperation();
> {code}
> The hiearchy of Connection(HiveConnection) -> Statement(HiveStatement) ->
> ResultSet(HiveQueryResultSet) should be respected in a sense that the parent
> can handle child but not to opposite way.
> The above logic was introduced by
> [HIVE-4974|https://github.com/apache/hive/blame/master/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L276].
> Its intention was to make children able to return their parents, but that
> doesn't mean they should handle their parents' lifecycle.
> 2. Also, HiveStatement should close HiveQueryResultSet only if it's not
> already closed, so it would make sense to check ResultSet.isClosed() before
> closing. This is for the very same reason as another change above, to avoid
> duplicated close logic.
> Background: under normal circumstances, a close operation is idempotent, we
> should not worry about any side effects of calling it twice, but while
> testing HIVE-24786, we found strange issues where in case of a
> SocketTimeoutException, such code path was hit in the jdbc client, that made
> HiveStatement.closeClientOperation() to be called twice, and it led to a
> WARNING on HS2 side. This is not expected as the operation close is protected
> by stmtHandle != null check, but yet it ran twice. To avoid situations like
> this, cleaning up duplicated close calls would help.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25203) HiveQueryResultSet is not expected to be closed twice
[ https://issues.apache.org/jira/browse/HIVE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned HIVE-25203: --- Assignee: László Bodor > HiveQueryResultSet is not expected to be closed twice > - > > Key: HIVE-25203 > URL: https://issues.apache.org/jira/browse/HIVE-25203 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22038) Fix memory related sideeffects of opening/closing sessions
[ https://issues.apache.org/jira/browse/HIVE-22038?focusedWorklogId=607465=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-607465 ] ASF GitHub Bot logged work on HIVE-22038: - Author: ASF GitHub Bot Created on: 06/Jun/21 00:18 Start Date: 06/Jun/21 00:18 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2078: URL: https://github.com/apache/hive/pull/2078#issuecomment-855314869 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 607465) Time Spent: 20m (was: 10m) > Fix memory related sideeffects of opening/closing sessions > -- > > Key: HIVE-22038 > URL: https://issues.apache.org/jira/browse/HIVE-22038 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22038.01.patch, HIVE-22038.02.patch, > HIVE-22038.02.patch, HIVE-22038.02.patch, HIVE-22038.02.patch, > HIVE-22038.02.patch, HIVE-22038.02.patch, HIVE-22038.02.patch, > HIVE-22038.02.patch, HIVE-22038.02.patch > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24765) ClassCastException with AND or OR condition
[
https://issues.apache.org/jira/browse/HIVE-24765?focusedWorklogId=607461=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-607461
]
ASF GitHub Bot logged work on HIVE-24765:
-
Author: ASF GitHub Bot
Created on: 06/Jun/21 00:18
Start Date: 06/Jun/21 00:18
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #2147:
URL: https://github.com/apache/hive/pull/2147#issuecomment-855314846
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the d...@hive.apache.org list if the patch is in
need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 607461)
Time Spent: 40m (was: 0.5h)
> ClassCastException with AND or OR condition
> ---
>
> Key: HIVE-24765
> URL: https://issues.apache.org/jira/browse/HIVE-24765
> Project: Hive
> Issue Type: Bug
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I previously created the following ticket:
> https://issues.apache.org/jira/browse/HIVE-11708
> However, it turns out that an error occurs under the following conditions:
> {code:java}
> CREATE TABLE tbl(
> id int,
> flg int
> );
> set hive.cbo.enable=true;
> SELECT * FROM tbl
> WHERE id >= 100 AND flg = TRUE;
> java.io.IOException: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to
> org.apache.hadoop.io.BooleanWritable
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:165)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2204)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:820)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:760)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:687)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.lazy.LazyInteger cannot be cast to
> org.apache.hadoop.io.BooleanWritable
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableBooleanObjectInspector.get(WritableBooleanObjectInspector.java:36)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:69)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:112)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:434)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:426)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
> ... 13 more
> {code}
> I know this is a cast issue as well as the previous issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24861) Hive JDBC driver doesn't consider the value of 'hasMoreRows'
[ https://issues.apache.org/jira/browse/HIVE-24861?focusedWorklogId=607467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-607467 ] ASF GitHub Bot logged work on HIVE-24861: - Author: ASF GitHub Bot Created on: 06/Jun/21 00:18 Start Date: 06/Jun/21 00:18 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2070: URL: https://github.com/apache/hive/pull/2070#issuecomment-855314871 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 607467) Time Spent: 50m (was: 40m) > Hive JDBC driver doesn't consider the value of 'hasMoreRows' > > > Key: HIVE-24861 > URL: https://issues.apache.org/jira/browse/HIVE-24861 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Zoltán Borók-Nagy >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > TCLIService's FetchResults might return an empty result set, but with > hasMoreRows=true. In that case the driver ignores the flag hasMoreRows and > thinks it is the end of the result stream, causing data loss. > I've seen this when the Hive JDBC driver was used to connect to Impala. > IMPALA-7312 introduced a timeout on FetchResults(). If Impala cannot produce > rows in the given timeout then it returns an empty result set, but setting > hasMoreRows=true. However, the Hive JDBC driver interprets it as the end of > the result stream and closes the operation. > I think if hasMoreRows=true then the Hive JDBC driver should issue > FetchResults() again. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24940) PartitionPruner may reject partitionfilter expressions evaluating to unknown if the filter is safe
[
https://issues.apache.org/jira/browse/HIVE-24940?focusedWorklogId=607463=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-607463
]
ASF GitHub Bot logged work on HIVE-24940:
-
Author: ASF GitHub Bot
Created on: 06/Jun/21 00:18
Start Date: 06/Jun/21 00:18
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #2131:
URL: https://github.com/apache/hive/pull/2131#issuecomment-855314855
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the d...@hive.apache.org list if the patch is in
need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 607463)
Time Spent: 20m (was: 10m)
> PartitionPruner may reject partitionfilter expressions evaluating to unknown
> if the filter is safe
> --
>
> Key: HIVE-24940
> URL: https://issues.apache.org/jira/browse/HIVE-24940
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code}
> CREATE TABLE t1pstring (col1 string) PARTITIONED BY (p1 string);
> INSERT INTO t1pstring PARTITION (p1) VALUES
> ("2020","2020"),("2021","2021"),("2021_backup","2021_backup"),('',''),('9','9'),('_a','_a'),('a','a');
> explain extended SELECT count(*) FROM t1pstring WHERE p1=;
> [...]
> | Truncated Path -> Alias: |
> | /t1pstring/p1=2021_backup [t1pstring] |
> | /t1pstring/p1= [t1pstring] |
> | /t1pstring/p1=_a [t1pstring] |
> | /t1pstring/p1=a [t1pstring] |
> [...]
> {code}
> note:
> * for all values which are interpretable as integers - the equals is
> evaluated and the result is false
> * for {{NULL}} values and for non-integer values ({{a}}) the comparision
> results a {{NULL}} which is retained
> * {{NULL}} -s are retained because there is a preprocessing step which
> removes functions which are dependant of non-partition columns as well - and
> replaces them with a {{NULL}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24927) Alter table rename moves temporary folders as part of the rename
[ https://issues.apache.org/jira/browse/HIVE-24927?focusedWorklogId=607464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-607464 ] ASF GitHub Bot logged work on HIVE-24927: - Author: ASF GitHub Bot Created on: 06/Jun/21 00:18 Start Date: 06/Jun/21 00:18 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2110: URL: https://github.com/apache/hive/pull/2110#issuecomment-855314863 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 607464) Time Spent: 20m (was: 10m) > Alter table rename moves temporary folders as part of the rename > > > Key: HIVE-24927 > URL: https://issues.apache.org/jira/browse/HIVE-24927 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Alter table rename moves temporary folders as part of the rename -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24948) Enhancing performance of OrcInputFormat.getSplits with bucket pruning
[
https://issues.apache.org/jira/browse/HIVE-24948?focusedWorklogId=607462=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-607462
]
ASF GitHub Bot logged work on HIVE-24948:
-
Author: ASF GitHub Bot
Created on: 06/Jun/21 00:18
Start Date: 06/Jun/21 00:18
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #2126:
URL: https://github.com/apache/hive/pull/2126#issuecomment-855314858
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the d...@hive.apache.org list if the patch is in
need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 607462)
Time Spent: 40m (was: 0.5h)
> Enhancing performance of OrcInputFormat.getSplits with bucket pruning
> -
>
> Key: HIVE-24948
> URL: https://issues.apache.org/jira/browse/HIVE-24948
> Project: Hive
> Issue Type: Bug
> Components: ORC, Query Processor, Tez
>Reporter: Eugene Chung
>Assignee: Eugene Chung
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-24948_3.1.2.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The summarized flow of generating input splits at Tez AM is like below; (by
> calling HiveSplitGenerator.initialize())
> # Perform dynamic partition pruning
> # Get the list of InputSplit by calling InputFormat.getSplits()
>
> [https://github.com/apache/hive/blob/624f62aadc08577cafaa299cfcf17c71fa6cdb3a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java#L260-L260]
> # Perform bucket pruning with the list above if it's possible
>
> [https://github.com/apache/hive/blob/624f62aadc08577cafaa299cfcf17c71fa6cdb3a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java#L299-L301]
> But I observed that the action 2, getting the list of InputSplit, can make
> big overhead when the inputs are ORC files in HDFS.
> For example, there is a ORC table T partitioned by 'log_date' and each
> partition is bucketed by a column 'q'. There are 240 buckets in each
> partition and the size of each bucket(ORC file) is, let's say, 100MB.
> The SQL is like this.
> {noformat}
> set hive.tez.bucket.pruning=true;
> select count(*) from T
> where log_date between '2020-01-01' and '2020-06-30'
> and q = 'foobar';{noformat}
> It means there are 240 * 183(days) = 43680 ORC files in the input paths, but
> thanks to bucket pruning, only 183 files should be processed.
> In my company's environment, the whole processing time of the SQL was roughly
> 5 minutes. However, I've checked that it took more than 3 minutes to make the
> list of OrcSplit for 43680 ORC files. The logs with tez.am.log.level=DEBUG
> showed like below;
> {noformat}
> 2021-03-25 01:21:31,850 [DEBUG] [InputInitializer {Map 1} #0]
> |orc.OrcInputFormat|: getSplits started
> ...
> 2021-03-25 01:24:51,435 [DEBUG] [InputInitializer {Map 1} #0]
> |orc.OrcInputFormat|: getSplits finished
> 2021-03-25 01:24:51,444 [INFO] [InputInitializer {Map 1} #0]
> |io.HiveInputFormat|: number of splits 43680
> 2021-03-25 01:24:51,444 [DEBUG] [InputInitializer {Map 1} #0]
> |log.PerfLogger|: /PERFLOG method=getSplits start=1616602891776
> end=1616602891776 duration=199668
> from=org.apache.hadoop.hive.ql.io.HiveInputFormat
> ...
> 2021-03-25 01:26:03,385 [INFO] [Dispatcher thread {Central}]
> |app.DAGAppMaster|: DAG completed, dagId=dag_1615862187190_731117_1,
> dagState=SUCCEEDED {noformat}
> 43680 - 183 = 43497 InputSplits which consume about 60% of entire processing
> time are just simply discarded by the action 3, pruneBuckets().
>
> With bucket pruning, I think making the whole list of ORC InputSplit is not
> necessary.
> Therefore, I suggest that the flow would be like this;
> # Perform dynamic partition pruning
> # Get the list of InputSplit by calling InputFormat.getSplits()
> ## OrcInputFormat.getSplits() returns the bucket-pruned list if BitSet from
> FixedBucketPruningOptimizer exists
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24915) Distribute by with sort by clause when used with constant parameter for sort produces wrong result.
[
https://issues.apache.org/jira/browse/HIVE-24915?focusedWorklogId=607466=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-607466
]
ASF GitHub Bot logged work on HIVE-24915:
-
Author: ASF GitHub Bot
Created on: 06/Jun/21 00:18
Start Date: 06/Jun/21 00:18
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #2093:
URL: https://github.com/apache/hive/pull/2093#issuecomment-855314867
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the d...@hive.apache.org list if the patch is in
need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 607466)
Time Spent: 20m (was: 10m)
> Distribute by with sort by clause when used with constant parameter for sort
> produces wrong result.
> ---
>
> Key: HIVE-24915
> URL: https://issues.apache.org/jira/browse/HIVE-24915
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: Suprith Chandrashekharachar
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Distribute by with sort by clause when used with constant parameter for sort
> produces wrong result.
> Example:
> {code:java}
> SELECT
> t.time,
> 'a' as const
> FROM
> (SELECT 1591819264 as time
> UNION ALL
> SELECT 1591819265 as time) t
> DISTRIBUTE by const
> sort by const, t.time
> {code}
> Produces
>
> |{color:#00}*time*{color}|{color:#00}*const*{color}|
> | NULL|{color:#00}a{color}|
> | NULL|{color:#00}a{color}|
> Instead it should produce(Hive 0.13 produces this):
> |{color:#00}*time*{color}|{color:#00}*const*{color}|
> |{color:#00}*1591819264*{color}|{color:#00}a{color}|
> |{color:#00}*1591819265*{color}|{color:#00}a{color}|
> Incorrect sort columns are used while creating ReduceSink here
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L9066]
> With constant propagation optimizer enabled, due to incorrect constant
> operator folding, incorrect results will be produced.
>
> More examples for incorrect behavior:
> {code:java}
> SELECT
> t.time,
> 'a' as const,
> t.id
> FROM
> (SELECT 1591819264 as time, 1 as id
> UNION ALL
> SELECT 1591819265 as time, 2 as id) t
> DISTRIBUTE by t.time
> sort by t.time, const, t.id
> {code}
> produces
> |{color:#00}*time*{color}|{color:#00}*const*{color}|{color:#00}*id*{color}|
> |{color:#00}*1591819264*{color}|{color:#00}a{color}|NULL |
> |{color:#00}*1591819265*{color}|{color:#00}a{color}| NULL|
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-23489) dynamic partition failed use insert overwrite
[
https://issues.apache.org/jira/browse/HIVE-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robbie Zhang resolved HIVE-23489.
-
Resolution: Resolved
> dynamic partition failed use insert overwrite
> -
>
> Key: HIVE-23489
> URL: https://issues.apache.org/jira/browse/HIVE-23489
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.1
>Reporter: shining
>Assignee: Robbie Zhang
>Priority: Major
>
> SQL: insert *overwrite *table dzdz_fpxx_dzfp partition(nian) select * from
> test.dzdz_fpxx_dzfp
> {noformat}
> create table dzdz_fpxx_dzfp (
> FPDM string,
> FPHM string,
> KPRQ timestamp,
> FPLY string)
> partitioned by (nian string)
> stored as parquet;
>
>
> create table test.dzdz_fpxx_dzfp (
> FPDM string,
> FPHM string,
> KPRQ timestamp,
> FPLY string,
> nian string)
> stored as textfile
> location "/origin/data/dzfp_origin/"
> {noformat}
> The execute insert sql:
> SQL: insert overwrite table dzdz_fpxx_dzfp partition(nian) select * from
> test.dzdz_fpxx_dzfp
> {noformat}
> INFO : Compiling
> command(queryId=hive_20200519100412_12f2b39c-45f4-4f4c-9261-32ca86fa28db):
> insert overwrite table dzdz_fpxx_dzfp partition(nian) select * from
> test.dzdz_fpxx_dzfp
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:dzdz_fpxx_dzfp.fpdm, type:string,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.fphm, type:string,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.kprq, type:timestamp,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.tspz_dm, type:string,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.fpzt_bz, type:string,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.fply, type:string,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.nian, type:string,
> comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200519100412_12f2b39c-45f4-4f4c-9261-32ca86fa28db);
> Time taken: 1.719 seconds
> INFO : Executing
> command(queryId=hive_20200519100412_12f2b39c-45f4-4f4c-9261-32ca86fa28db):
> insert overwrite table dzdz_fpxx_dzfp partition(nian) select * from
> test.dzdz_fpxx_dzfp
> WARN :
> INFO : Query ID = hive_20200519100412_12f2b39c-45f4-4f4c-9261-32ca86fa28db
> INFO : Total jobs = 3
> INFO : Launching Job 1 out of 3
> INFO : Starting task [Stage-1:MAPRED] in serial mode
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : number of splits:1
> INFO : Submitting tokens for job: job_1589451904439_0049
> INFO : Executing with tokens: []
> INFO : The url to track the job:
> http://qcj37.hde.com:8088/proxy/application_1589451904439_0049/
> INFO : Starting Job = job_1589451904439_0049, Tracking URL =
> http://qcj37.hde.com:8088/proxy/application_1589451904439_0049/
> INFO : Kill Command = /usr/hdp/current/hadoop-client/bin/hadoop job -kill
> job_1589451904439_0049
> INFO : Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> INFO : 2020-05-19 10:06:56,823 Stage-1 map = 0%, reduce = 0%
> INFO : 2020-05-19 10:07:06,317 Stage-1 map = 100%, reduce = 0%, Cumulative
> CPU 4.3 sec
> INFO : MapReduce Total cumulative CPU time: 4 seconds 300 msec
> INFO : Ended Job = job_1589451904439_0049
> INFO : Starting task [Stage-7:CONDITIONAL] in serial mode
> INFO : Stage-4 is selected by condition resolver.
> INFO : Stage-3 is filtered out by condition resolver.
> INFO : Stage-5 is filtered out by condition resolver.
> INFO : Starting task [Stage-4:MOVE] in serial mode
> INFO : Moving data to directory
> hdfs://mycluster/warehouse/tablespace/managed/hive/dzdz_fpxx_dzfp/.hive-staging_hive_2020-05-19_10-04-12_468_7695595367555279265-3/-ext-1
> from
> hdfs://mycluster/warehouse/tablespace/managed/hive/dzdz_fpxx_dzfp/.hive-staging_hive_2020-05-19_10-04-12_468_7695595367555279265-3/-ext-10002
> INFO : Starting task [Stage-0:MOVE] in serial mode
> INFO : Loading data to table default.dzdz_fpxx_dzfp partition (nian=null)
> from
> hdfs://mycluster/warehouse/tablespace/managed/hive/dzdz_fpxx_dzfp/.hive-staging_hive_2020-05-19_10-04-12_468_7695595367555279265-3/-ext-1
> INFO :
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask. Exception when loading 2 in table
> dzdz_fpxx_dzfp with
> loadPath=hdfs://mycluster/warehouse/tablespace/managed/hive/dzdz_fpxx_dzfp/.hive-staging_hive_2020-05-19_10-04-12_468_7695595367555279265-3/-ext-1
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-1: Map: 1 Cumulative CPU: 4.3 sec HDFS Read: 8015
> HDFS Write: 2511 SUCCESS
>
[jira] [Commented] (HIVE-23489) dynamic partition failed use insert overwrite
[
https://issues.apache.org/jira/browse/HIVE-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357764#comment-17357764
]
Robbie Zhang commented on HIVE-23489:
-
I can reproduce this issue on the same version by setting
hive.metastore.dml.events to true. Insert Overwrites to a new partition should
not capture new files as part of insert event. Otherwise, HMS will report this
error because the new partition doesn't exist. This bug is fixed by HIVE-15642.
> dynamic partition failed use insert overwrite
> -
>
> Key: HIVE-23489
> URL: https://issues.apache.org/jira/browse/HIVE-23489
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.1
>Reporter: shining
>Assignee: Robbie Zhang
>Priority: Major
>
> SQL: insert *overwrite *table dzdz_fpxx_dzfp partition(nian) select * from
> test.dzdz_fpxx_dzfp
> {noformat}
> create table dzdz_fpxx_dzfp (
> FPDM string,
> FPHM string,
> KPRQ timestamp,
> FPLY string)
> partitioned by (nian string)
> stored as parquet;
>
>
> create table test.dzdz_fpxx_dzfp (
> FPDM string,
> FPHM string,
> KPRQ timestamp,
> FPLY string,
> nian string)
> stored as textfile
> location "/origin/data/dzfp_origin/"
> {noformat}
> The execute insert sql:
> SQL: insert overwrite table dzdz_fpxx_dzfp partition(nian) select * from
> test.dzdz_fpxx_dzfp
> {noformat}
> INFO : Compiling
> command(queryId=hive_20200519100412_12f2b39c-45f4-4f4c-9261-32ca86fa28db):
> insert overwrite table dzdz_fpxx_dzfp partition(nian) select * from
> test.dzdz_fpxx_dzfp
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:dzdz_fpxx_dzfp.fpdm, type:string,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.fphm, type:string,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.kprq, type:timestamp,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.tspz_dm, type:string,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.fpzt_bz, type:string,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.fply, type:string,
> comment:null), FieldSchema(name:dzdz_fpxx_dzfp.nian, type:string,
> comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200519100412_12f2b39c-45f4-4f4c-9261-32ca86fa28db);
> Time taken: 1.719 seconds
> INFO : Executing
> command(queryId=hive_20200519100412_12f2b39c-45f4-4f4c-9261-32ca86fa28db):
> insert overwrite table dzdz_fpxx_dzfp partition(nian) select * from
> test.dzdz_fpxx_dzfp
> WARN :
> INFO : Query ID = hive_20200519100412_12f2b39c-45f4-4f4c-9261-32ca86fa28db
> INFO : Total jobs = 3
> INFO : Launching Job 1 out of 3
> INFO : Starting task [Stage-1:MAPRED] in serial mode
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : number of splits:1
> INFO : Submitting tokens for job: job_1589451904439_0049
> INFO : Executing with tokens: []
> INFO : The url to track the job:
> http://qcj37.hde.com:8088/proxy/application_1589451904439_0049/
> INFO : Starting Job = job_1589451904439_0049, Tracking URL =
> http://qcj37.hde.com:8088/proxy/application_1589451904439_0049/
> INFO : Kill Command = /usr/hdp/current/hadoop-client/bin/hadoop job -kill
> job_1589451904439_0049
> INFO : Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> INFO : 2020-05-19 10:06:56,823 Stage-1 map = 0%, reduce = 0%
> INFO : 2020-05-19 10:07:06,317 Stage-1 map = 100%, reduce = 0%, Cumulative
> CPU 4.3 sec
> INFO : MapReduce Total cumulative CPU time: 4 seconds 300 msec
> INFO : Ended Job = job_1589451904439_0049
> INFO : Starting task [Stage-7:CONDITIONAL] in serial mode
> INFO : Stage-4 is selected by condition resolver.
> INFO : Stage-3 is filtered out by condition resolver.
> INFO : Stage-5 is filtered out by condition resolver.
> INFO : Starting task [Stage-4:MOVE] in serial mode
> INFO : Moving data to directory
> hdfs://mycluster/warehouse/tablespace/managed/hive/dzdz_fpxx_dzfp/.hive-staging_hive_2020-05-19_10-04-12_468_7695595367555279265-3/-ext-1
> from
> hdfs://mycluster/warehouse/tablespace/managed/hive/dzdz_fpxx_dzfp/.hive-staging_hive_2020-05-19_10-04-12_468_7695595367555279265-3/-ext-10002
> INFO : Starting task [Stage-0:MOVE] in serial mode
> INFO : Loading data to table default.dzdz_fpxx_dzfp partition (nian=null)
> from
> hdfs://mycluster/warehouse/tablespace/managed/hive/dzdz_fpxx_dzfp/.hive-staging_hive_2020-05-19_10-04-12_468_7695595367555279265-3/-ext-1
> INFO :
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask. Exception when loading 2 in table
