[jira] [Updated] (HIVE-23607) Permission Issue: Create view on another view succeeds but alter view fails
[
https://issues.apache.org/jira/browse/HIVE-23607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naresh P R updated HIVE-23607:
--
Attachment: HIVE-23607.patch
Status: Patch Available (was: Open)
> Permission Issue: Create view on another view succeeds but alter view fails
> -
>
> Key: HIVE-23607
> URL: https://issues.apache.org/jira/browse/HIVE-23607
> Project: Hive
> Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Major
> Attachments: HIVE-23607.patch
>
>
> create table test_table (id int);
> create view test_view as select * from test_table;
>
> {code:java}
> -- user "naresh" as read access on test_view
> -- Create view succeeds
> create view test_view_1 as select * from test_view;
> -- Alter view fails
> alter view test_view_1 as select * from test_view
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [naresh] does not have [SELECT] privilege on
> [test/test_table] (state=42000,code=4)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23607) Permission Issue: Create view on another view succeeds but alter view fails
[
https://issues.apache.org/jira/browse/HIVE-23607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naresh P R reassigned HIVE-23607:
-
> Permission Issue: Create view on another view succeeds but alter view fails
> -
>
> Key: HIVE-23607
> URL: https://issues.apache.org/jira/browse/HIVE-23607
> Project: Hive
> Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Major
>
> create table test_table (id int);
> create view test_view as select * from test_table;
>
> {code:java}
> -- user "naresh" as read access on test_view
> -- Create view succeeds
> create view test_view_1 as select * from test_view;
> -- Alter view fails
> alter view test_view_1 as select * from test_view
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [naresh] does not have [SELECT] privilege on
> [test/test_table] (state=42000,code=4)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23606) LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl
[ https://issues.apache.org/jira/browse/HIVE-23606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125557#comment-17125557 ] Gopal Vijayaraghavan commented on HIVE-23606: - +1 tests pending. > LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl > -- > > Key: HIVE-23606 > URL: https://issues.apache.org/jira/browse/HIVE-23606 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23606.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > DirectByteBuffler are only cleaned up when there is Full GC or manually > invoked cleaner method of DirectByteBuffer, Since full GC may take some time > to kick in, In the meanwhile the native memory usage of LLAP daemon process > might shoot up and this will force the YARN pmem monitor to kill the > container running the daemon. > HIVE-16180 tried to solve this problem, but the code structure got messed up > after HIVE-15665 > The IdentityHashMap (toRelease) is initialized in > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L409 > , but it is getting re-initialized inside the method > getDataFromCacheAndDisk() > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L633 > which makes it local to that method hence the original toRelease > IdentityHashMap remains empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23606) LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl
[ https://issues.apache.org/jira/browse/HIVE-23606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-23606: - Component/s: llap > LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl > -- > > Key: HIVE-23606 > URL: https://issues.apache.org/jira/browse/HIVE-23606 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23606.01.patch > > > DirectByteBuffler are only cleaned up when there is Full GC or manually > invoked cleaner method of DirectByteBuffer, Since full GC may take some time > to kick in, In the meanwhile the native memory usage of LLAP daemon process > might shoot up and this will force the YARN pmem monitor to kill the > container running the daemon. > HIVE-16180 tried to solve this problem, but the code structure got messed up > after HIVE-15665 > The IdentityHashMap (toRelease) is initialized in > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L409 > , but it is getting re-initialized inside the method > getDataFromCacheAndDisk() > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L633 > which makes it local to that method hence the original toRelease > IdentityHashMap remains empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23606) LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl
[ https://issues.apache.org/jira/browse/HIVE-23606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23606: -- Labels: pull-request-available (was: ) > LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl > -- > > Key: HIVE-23606 > URL: https://issues.apache.org/jira/browse/HIVE-23606 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23606.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > DirectByteBuffler are only cleaned up when there is Full GC or manually > invoked cleaner method of DirectByteBuffer, Since full GC may take some time > to kick in, In the meanwhile the native memory usage of LLAP daemon process > might shoot up and this will force the YARN pmem monitor to kill the > container running the daemon. > HIVE-16180 tried to solve this problem, but the code structure got messed up > after HIVE-15665 > The IdentityHashMap (toRelease) is initialized in > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L409 > , but it is getting re-initialized inside the method > getDataFromCacheAndDisk() > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L633 > which makes it local to that method hence the original toRelease > IdentityHashMap remains empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23606) LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl
[ https://issues.apache.org/jira/browse/HIVE-23606?focusedWorklogId=441105=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441105 ] ASF GitHub Bot logged work on HIVE-23606: - Author: ASF GitHub Bot Created on: 04/Jun/20 05:28 Start Date: 04/Jun/20 05:28 Worklog Time Spent: 10m Work Description: shameersss1 opened a new pull request #1057: URL: https://github.com/apache/hive/pull/1057 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441105) Remaining Estimate: 0h Time Spent: 10m > LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl > -- > > Key: HIVE-23606 > URL: https://issues.apache.org/jira/browse/HIVE-23606 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23606.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > DirectByteBuffler are only cleaned up when there is Full GC or manually > invoked cleaner method of DirectByteBuffer, Since full GC may take some time > to kick in, In the meanwhile the native memory usage of LLAP daemon process > might shoot up and this will force the YARN pmem monitor to kill the > container running the daemon. > HIVE-16180 tried to solve this problem, but the code structure got messed up > after HIVE-15665 > The IdentityHashMap (toRelease) is initialized in > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L409 > , but it is getting re-initialized inside the method > getDataFromCacheAndDisk() > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L633 > which makes it local to that method hence the original toRelease > IdentityHashMap remains empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23606) LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl
[ https://issues.apache.org/jira/browse/HIVE-23606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-23606: - Attachment: HIVE-23606.01.patch Status: Patch Available (was: Open) > LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl > -- > > Key: HIVE-23606 > URL: https://issues.apache.org/jira/browse/HIVE-23606 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23606.01.patch > > > DirectByteBuffler are only cleaned up when there is Full GC or manually > invoked cleaner method of DirectByteBuffer, Since full GC may take some time > to kick in, In the meanwhile the native memory usage of LLAP daemon process > might shoot up and this will force the YARN pmem monitor to kill the > container running the daemon. > HIVE-16180 tried to solve this problem, but the code structure got messed up > after HIVE-15665 > The IdentityHashMap (toRelease) is initialized in > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L409 > , but it is getting re-initialized inside the method > getDataFromCacheAndDisk() > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L633 > which makes it local to that method hence the original toRelease > IdentityHashMap remains empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23606) LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl
[ https://issues.apache.org/jira/browse/HIVE-23606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman reassigned HIVE-23606: > LLAP: Delay In DirectByteBuffer Clean Up For EncodedReaderImpl > -- > > Key: HIVE-23606 > URL: https://issues.apache.org/jira/browse/HIVE-23606 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Fix For: 4.0.0 > > > DirectByteBuffler are only cleaned up when there is Full GC or manually > invoked cleaner method of DirectByteBuffer, Since full GC may take some time > to kick in, In the meanwhile the native memory usage of LLAP daemon process > might shoot up and this will force the YARN pmem monitor to kill the > container running the daemon. > HIVE-16180 tried to solve this problem, but the code structure got messed up > after HIVE-15665 > The IdentityHashMap (toRelease) is initialized in > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L409 > , but it is getting re-initialized inside the method > getDataFromCacheAndDisk() > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java#L633 > which makes it local to that method hence the original toRelease > IdentityHashMap remains empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23277) HiveProtoLogger should carry out JSON conversion in its own thread
[
https://issues.apache.org/jira/browse/HIVE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125528#comment-17125528
]
Rajesh Balamohan commented on HIVE-23277:
-
This is to avoid JSON serialization being in the hotpath of compiler thread. In
short queries, where subsecond latency matters, this becomes an issue along
with the query complexity. BTW, its \{getExplainPlan(plan, conf, hookContext)}
also needs to be moved out of the in-thread processing.
> HiveProtoLogger should carry out JSON conversion in its own thread
> --
>
> Key: HIVE-23277
> URL: https://issues.apache.org/jira/browse/HIVE-23277
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Minor
> Attachments: HIVE-23277.1.patch, Screenshot 2020-04-23 at 11.27.42
> AM.png
>
>
> !Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=441073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441073 ] ASF GitHub Bot logged work on HIVE-23526: - Author: ASF GitHub Bot Created on: 04/Jun/20 02:27 Start Date: 04/Jun/20 02:27 Worklog Time Spent: 10m Work Description: belugabehr edited a comment on pull request #1029: URL: https://github.com/apache/hive/pull/1029#issuecomment-638563394 I think the best thing would be to modify this: https://github.com/apache/hive/blob/2795d3253f5ff09c23b12e5ff980cb14268d37ab/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java#L189-L202 Thrift does not have sub-classes for Exception, but Exceptions generally carry a type. In this case, the type is `TApplicationException.BAD_SEQUENCE_ID` : https://github.com/apache/thrift/blob/af7ecd6a2b15efe5c6b742cf4a9ccb31bcc1f362/lib/java/src/org/apache/thrift/TServiceClient.java#L84-L87 I think the best course of action here is to catch the Exception in the `HiveStatement`, check the 'type', and if it is `BAD_SEQUENCE_ID`, wrap the Exception in a `SQLException` and provide a better error message that is more clear. That `Commands` method is a bit hairy, but all that needs to happen as a quick fix is to wrap the `Statement#close()` with its own try-catch block. There is no need to close the result set. Closing the Statement should close all associated result sets. > When a Statement object is closed, its current ResultSet object, if one exists, is also closed. https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#close() This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441073) Time Spent: 3.5h (was: 3h 20m) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Improvement > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 3.5h > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=441071=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441071 ] ASF GitHub Bot logged work on HIVE-23526: - Author: ASF GitHub Bot Created on: 04/Jun/20 02:26 Start Date: 04/Jun/20 02:26 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #1029: URL: https://github.com/apache/hive/pull/1029#issuecomment-638563394 I think the best thing would be to modify this: https://github.com/apache/hive/blob/2795d3253f5ff09c23b12e5ff980cb14268d37ab/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java#L189-L202 Thrift does not have sub-classes for Exception, but Exceptions generally carry a type. In this case, the type is `TApplicationException.BAD_SEQUENCE_ID` : https://github.com/apache/thrift/blob/af7ecd6a2b15efe5c6b742cf4a9ccb31bcc1f362/lib/java/src/org/apache/thrift/TServiceClient.java#L84-L87 I think the best course of action here is to catch the Exception in the `HiveStatement`, check the 'type', and if it is `BAD_SEQUENCE_ID, wrap the Exception in a `SQLException` and provide a better error message that is more clear. That `Commands` method is a bit hairy, but all that needs to happen as a quick fix is to wrap the `Statement#close()` with its own try-catch block. There is no need to close the result set. Closing the Statement should close all associated result sets. > When a Statement object is closed, its current ResultSet object, if one exists, is also closed. https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#close() This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441071) Time Spent: 3h 20m (was: 3h 10m) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Improvement > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 3h 20m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[
https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=441070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441070
]
ASF GitHub Bot logged work on HIVE-23526:
-
Author: ASF GitHub Bot
Created on: 04/Jun/20 02:16
Start Date: 04/Jun/20 02:16
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #1029:
URL: https://github.com/apache/hive/pull/1029#discussion_r434953975
##
File path: beeline/src/java/org/apache/hive/beeline/Commands.java
##
@@ -1063,13 +1057,17 @@ private boolean executeInternal(String sql, boolean
call) {
showRemainingLogsIfAny(stmnt);
}
}
+if (rs != null) {
+ rs.close();
Review comment:
Closing the result set can throw a `SQLException` in which case it will
skip over the `Statement#close()` method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 441070)
Time Spent: 3h 10m (was: 3h)
> Beeline may throw the misleading exception
> --
>
> Key: HIVE-23526
> URL: https://issues.apache.org/jira/browse/HIVE-23526
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
> Environment: Hive 1.2.2
>Reporter: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch,
> HIVE-23526.patch, outofsequence.log
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Sometimes we can see 'out of sequence response' message in beeline, for
> example:
> Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of
> sequence response (state=08S01,code=0)
> java.sql.SQLException: org.apache.thrift.TApplicationException:
> CloseOperation failed: out of sequence response
> at
> org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198)
> at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217)
> at org.apache.hive.beeline.Commands.execute(Commands.java:891)
> at org.apache.hive.beeline.Commands.sql(Commands.java:713)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470)
> and there is no other usage message to figured it out, even with --verbose,
> this makes problem puzzled as beeline does not have concurrency problem on
> underlying thrift transport.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23605) 'Wrong FS' error during _external_tables_info creation when staging location is remote
[
https://issues.apache.org/jira/browse/HIVE-23605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125469#comment-17125469
]
Hive QA commented on HIVE-23605:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004764/HIVE-23605.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17102 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22738/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22738/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22738/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004764 - PreCommit-HIVE-Build
> 'Wrong FS' error during _external_tables_info creation when staging location
> is remote
> --
>
> Key: HIVE-23605
> URL: https://issues.apache.org/jira/browse/HIVE-23605
> Project: Hive
> Issue Type: Bug
>Reporter: Pravin Sinha
>Assignee: Pravin Sinha
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23605.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When staging location is on target cluster, Repl Dump fails to create
> _external_tables_info file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125420#comment-17125420 ] Zhihua Deng edited comment on HIVE-23526 at 6/4/20, 1:22 AM: - [~belugabehr] you are right. With this, the oom error will show before the "CloseOperation failed: out of sequence response"(when closing driver in finally), beeline users can easily know what happened on their client. was (Author: dengzh): [~belugabehr] you are right. With this, the oom error will show before the "CloseOperation failed: out of sequence response"(when closing driver in finally), beeline users can easily know what happened. > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Improvement > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 3h > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23605) 'Wrong FS' error during _external_tables_info creation when staging location is remote
[
https://issues.apache.org/jira/browse/HIVE-23605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125452#comment-17125452
]
Hive QA commented on HIVE-23605:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 1523 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22738/dev-support/hive-personality.sh
|
| git revision | master / 2795d32 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22738/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> 'Wrong FS' error during _external_tables_info creation when staging location
> is remote
> --
>
> Key: HIVE-23605
> URL: https://issues.apache.org/jira/browse/HIVE-23605
> Project: Hive
> Issue Type: Bug
>Reporter: Pravin Sinha
>Assignee: Pravin Sinha
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23605.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When staging location is on target cluster, Repl Dump fails to create
> _external_tables_info file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23605) 'Wrong FS' error during _external_tables_info creation when staging location is remote
[ https://issues.apache.org/jira/browse/HIVE-23605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha updated HIVE-23605: Attachment: HIVE-23605.01.patch > 'Wrong FS' error during _external_tables_info creation when staging location > is remote > -- > > Key: HIVE-23605 > URL: https://issues.apache.org/jira/browse/HIVE-23605 > Project: Hive > Issue Type: Bug >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23605.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > When staging location is on target cluster, Repl Dump fails to create > _external_tables_info file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23605) 'Wrong FS' error during _external_tables_info creation when staging location is remote
[ https://issues.apache.org/jira/browse/HIVE-23605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha updated HIVE-23605: Status: Patch Available (was: Open) > 'Wrong FS' error during _external_tables_info creation when staging location > is remote > -- > > Key: HIVE-23605 > URL: https://issues.apache.org/jira/browse/HIVE-23605 > Project: Hive > Issue Type: Bug >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23605.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > When staging location is on target cluster, Repl Dump fails to create > _external_tables_info file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23605) Wrong FS error during _external_tables_info creation when staging location is remote
[ https://issues.apache.org/jira/browse/HIVE-23605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23605: -- Labels: pull-request-available (was: ) > Wrong FS error during _external_tables_info creation when staging location is > remote > > > Key: HIVE-23605 > URL: https://issues.apache.org/jira/browse/HIVE-23605 > Project: Hive > Issue Type: Bug >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When staging location is on target cluster, Repl Dump fails to create > _external_tables_info file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23605) Wrong FS error during _external_tables_info creation when staging location is remote
[ https://issues.apache.org/jira/browse/HIVE-23605?focusedWorklogId=441042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441042 ] ASF GitHub Bot logged work on HIVE-23605: - Author: ASF GitHub Bot Created on: 04/Jun/20 00:21 Start Date: 04/Jun/20 00:21 Worklog Time Spent: 10m Work Description: pkumarsinha opened a new pull request #1056: URL: https://github.com/apache/hive/pull/1056 …staging location is remote This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441042) Remaining Estimate: 0h Time Spent: 10m > Wrong FS error during _external_tables_info creation when staging location is > remote > > > Key: HIVE-23605 > URL: https://issues.apache.org/jira/browse/HIVE-23605 > Project: Hive > Issue Type: Bug >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > When staging location is on target cluster, Repl Dump fails to create > _external_tables_info file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23605) 'Wrong FS' error during _external_tables_info creation when staging location is remote
[ https://issues.apache.org/jira/browse/HIVE-23605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha updated HIVE-23605: Summary: 'Wrong FS' error during _external_tables_info creation when staging location is remote (was: Wrong FS error during _external_tables_info creation when staging location is remote) > 'Wrong FS' error during _external_tables_info creation when staging location > is remote > -- > > Key: HIVE-23605 > URL: https://issues.apache.org/jira/browse/HIVE-23605 > Project: Hive > Issue Type: Bug >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When staging location is on target cluster, Repl Dump fails to create > _external_tables_info file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125420#comment-17125420 ] Zhihua Deng commented on HIVE-23526: [~belugabehr] you are right. With this, the oom error will show before the "CloseOperation failed: out of sequence response"(when closing driver in finally), beeline users can easily know what happened. > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Improvement > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 3h > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23605) Wrong FS error during _external_tables_info creation when staging location is remote
[ https://issues.apache.org/jira/browse/HIVE-23605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha reassigned HIVE-23605: --- > Wrong FS error during _external_tables_info creation when staging location is > remote > > > Key: HIVE-23605 > URL: https://issues.apache.org/jira/browse/HIVE-23605 > Project: Hive > Issue Type: Bug >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > > When staging location is on target cluster, Repl Dump fails to create > _external_tables_info file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23526: --- Issue Type: Improvement (was: Bug) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Improvement > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 3h > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=441031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441031 ] ASF GitHub Bot logged work on HIVE-23526: - Author: ASF GitHub Bot Created on: 03/Jun/20 23:48 Start Date: 03/Jun/20 23:48 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #1029: URL: https://github.com/apache/hive/pull/1029#issuecomment-638518722 > As I understand it, the code calling the JDBC Driver captures and handles the OOM error and then closes the JDBC Driver. When it closes, it throws another error " CloseOperation failed: out of sequence response". You would like for the close to be silent here. Is that correct? > > If so, there's not a ton that can be done here (other than perhaps wrap the OOM error in a Thrift Exception). > > In this particular case, Thrift client is reading the size of the string off the wire, then trying to create a buffer large enough to read the string. This is where it fails. When it fails here, it loses its place in the stream. The data is still in the stream, but the size is lost since that was read before the OOM error and there is no way to push the size back into the stream. At this point, Thrift client is hosed and it can't read anything else. > > When the JDBC Driver is closed, it sends a "close" request to HS2 to gracefully close the Hive session on the server. However, it is unable to read the response properly because, again, it lost its place and does not know how to find the next message. I'm not sure what the fix is here, but I don't think this is it. Yes, that's the truth, somehow a socket read timeout can also make the same trouble. The fix handles the root cause before beeline throwing the puzzling exception, make the beeline user have a way to fix the problem quickly. Maybe this is not a bug fix but just an improvement. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 441031) Time Spent: 3h (was: 2h 50m) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 3h > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23526: --- Priority: Minor (was: Major) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 3h > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve
[
https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=441014=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441014
]
ASF GitHub Bot logged work on HIVE-19261:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 22:45
Start Date: 03/Jun/20 22:45
Worklog Time Spent: 10m
Work Description: mythrocks commented on a change in pull request #807:
URL: https://github.com/apache/hive/pull/807#discussion_r434898293
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
##
@@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
- public synchronized Instance retrieve(SeedObject hv,
- Set seenSchemas) throws AvroSerdeException {
+ public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
if(cache.containsKey(hv)) {
Review comment:
Ok, I think I have it.
[`InstanceCache::retrieve()`](https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java#L66)
modifies the cache here:
```java
Instance instance = makeInstance(hv, seenSchemas);
cache.put(hv, instance);
return instance;
```
We would like to use `computeIfAbsent()` instead:
```java
cache.computeIfAbsent(hv, ()->makeInstance(hv, seenSchemas));
```
If `makeInstance()` ends up modifying `cache`, (e.g. by calling
`retrieve()`), that would cause undefined behaviour in`computeIfAbsent()`.
[`SchemaToTypeInfo`](https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java#L116)
implements `InstanceCache::makeInstance()` as follows:
```java
static InstanceCache typeInfoCache = new
InstanceCache() {
@Override
protected TypeInfo makeInstance(Schema s,
Set seenSchemas)
throws AvroSerdeException {
return generateTypeInfoWorker(s,
seenSchemas); // < HERE!
}
};
```
Please note that
[`SchemaToTypeInfo::generateTypeInfo()`](https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java#L186)
happens to call `InstanceCache::retrieve()`:
```java
public static TypeInfo generateTypeInfo(Schema schema,
Set seenSchemas) throws AvroSerdeException {
// ...
return typeInfoCache.retrieve(schema, seenSchemas);
}
```
Hence, the recursive call. :/
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 441014)
Time Spent: 1h 40m (was: 1.5h)
> Avro SerDe's InstanceCache should not be synchronized on retrieve
> -
>
> Key: HIVE-19261
> URL: https://issues.apache.org/jira/browse/HIVE-19261
> Project: Hive
> Issue Type: Improvement
>Reporter: Fangshi Li
>Assignee: Alexey Diomin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19261.1.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> In HIVE-16175, upstream made a patch to fix the thread safety issue in
> AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache
> synchronized. While it should make InstanceCache thread-safe, making retrieve
> synchronized for the cache can be expensive in highly concurrent environment
> like Spark, as multiple threads need to be synchronized on entering the
> entire retrieve method.
> We are proposing another way to fix this thread safety issue by making the
> underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use
> atomic computeIfAbsent in the retrieve method to avoid synchronizing the
> entire method.
> While computeIfAbsent is only available on java 8 and java 7 is still
> supported in Hive,
> we use a pattern to simulate the behavior of computeIfAbsent. In the future,
> we should move to
[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve
[
https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=441006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-441006
]
ASF GitHub Bot logged work on HIVE-19261:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 22:25
Start Date: 03/Jun/20 22:25
Worklog Time Spent: 10m
Work Description: mythrocks commented on a change in pull request #807:
URL: https://github.com/apache/hive/pull/807#discussion_r434891408
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
##
@@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
- public synchronized Instance retrieve(SeedObject hv,
- Set seenSchemas) throws AvroSerdeException {
+ public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
if(cache.containsKey(hv)) {
Review comment:
> I poked around a bit, nothing is obviously reentrant.
Sorry for the delay. I remember this being convoluted... Let me see if I can
recall this. Just a moment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 441006)
Time Spent: 1.5h (was: 1h 20m)
> Avro SerDe's InstanceCache should not be synchronized on retrieve
> -
>
> Key: HIVE-19261
> URL: https://issues.apache.org/jira/browse/HIVE-19261
> Project: Hive
> Issue Type: Improvement
>Reporter: Fangshi Li
>Assignee: Alexey Diomin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19261.1.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In HIVE-16175, upstream made a patch to fix the thread safety issue in
> AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache
> synchronized. While it should make InstanceCache thread-safe, making retrieve
> synchronized for the cache can be expensive in highly concurrent environment
> like Spark, as multiple threads need to be synchronized on entering the
> entire retrieve method.
> We are proposing another way to fix this thread safety issue by making the
> underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use
> atomic computeIfAbsent in the retrieve method to avoid synchronizing the
> entire method.
> While computeIfAbsent is only available on java 8 and java 7 is still
> supported in Hive,
> we use a pattern to simulate the behavior of computeIfAbsent. In the future,
> we should move to computeIfAbsent when Hive requires java 8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22681) Replace Base64 in hcatalog-webhcat Package
[
https://issues.apache.org/jira/browse/HIVE-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125338#comment-17125338
]
Hive QA commented on HIVE-22681:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004745/HIVE-22681.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17102 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22737/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22737/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22737/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004745 - PreCommit-HIVE-Build
> Replace Base64 in hcatalog-webhcat Package
> --
>
> Key: HIVE-22681
> URL: https://issues.apache.org/jira/browse/HIVE-22681
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22681.1.patch, HIVE-22681.1.patch,
> HIVE-22681.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22681) Replace Base64 in hcatalog-webhcat Package
[
https://issues.apache.org/jira/browse/HIVE-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125304#comment-17125304
]
Hive QA commented on HIVE-22681:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/webhcat/java-client in master has 3 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22737/dev-support/hive-personality.sh
|
| git revision | master / 2795d32 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: hcatalog/webhcat/java-client U: hcatalog/webhcat/java-client |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22737/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Replace Base64 in hcatalog-webhcat Package
> --
>
> Key: HIVE-22681
> URL: https://issues.apache.org/jira/browse/HIVE-22681
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22681.1.patch, HIVE-22681.1.patch,
> HIVE-22681.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23516) Store hive replication policy execution metrics in the relational DB
[
https://issues.apache.org/jira/browse/HIVE-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125288#comment-17125288
]
Hive QA commented on HIVE-23516:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004746/HIVE-23516.06.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 17109 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.repl.TestAtlasDumpTask.testAtlasDumpMetrics
(batchId=293)
org.apache.hadoop.hive.ql.exec.repl.TestAtlasLoadTask.testAtlasLoadMetrics
(batchId=293)
org.apache.hadoop.hive.ql.exec.repl.TestReplDumpTask.removeDBPropertyToPreventRenameWhenBootstrapDumpOfTableFails
(batchId=293)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testRetryAcidTablesBootstrapFromDifferentDump
(batchId=183)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testIncrementalDumpEmptyDumpDirectory
(batchId=192)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testIncrementalMetadataReplication
(batchId=192)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testRetryAcidTablesBootstrapFromDifferentDump
(batchId=180)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testIncrementalDumpEmptyDumpDirectory
(batchId=196)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testIncrementalMetadataReplication
(batchId=196)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTables.externalTableIncrementalReplication
(batchId=191)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTables.retryBootstrapExternalTablesFromDifferentDump
(batchId=191)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTables.testExtTableBootstrapDuringIncrementalWithoutAnyEvents
(batchId=191)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTables.testIncrementalDumpEmptyDumpDirectory
(batchId=191)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTablesMetaDataOnly.externalTableIncrementalReplication
(batchId=206)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTablesMetaDataOnly.testIncrementalDumpEmptyDumpDirectory
(batchId=206)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22736/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22736/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22736/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004746 - PreCommit-HIVE-Build
> Store hive replication policy execution metrics in the relational DB
>
>
> Key: HIVE-23516
> URL: https://issues.apache.org/jira/browse/HIVE-23516
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23516.01.patch, HIVE-23516.02.patch,
> HIVE-23516.03.patch, HIVE-23516.04.patch, HIVE-23516.05.patch,
> HIVE-23516.06.patch, Replication Metrics.pdf
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Details documented in the attached doc
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23604) LLAP does not have correct version of guava after HIVE-22126
[
https://issues.apache.org/jira/browse/HIVE-23604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanhao Lu updated HIVE-23604:
--
Description:
This JIRA https://issues.apache.org/jira/browse/HIVE-22126 shaded guava in
hive-exec. There is a issue that LLAP is also using this guava, so after
shading, it cannot use hive-exec guava and may turn to `./tez/guava-11.0.2.jar`
which will cause following error
{code:java}
Status: Running (Executing on YARN cluster with App id
application_1591081923777_0005)
Map 1: -/- Reducer 2: 0/1
Map 1: 0/11 Reducer 2: 0/1
Map 1: 0(+11,-11)/11Reducer 2: 0/1
Map 1: 0(+0,-32)/11 Reducer 2: 0/1
Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1591081923777_0005_1_00,
diagnostics=[Task failed, taskId=task_1591081923777_0005_1_00_02,
diagnostics=[TaskAttempt 0 failed,
info=[org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
com.google.common.base.Stopwatch.createUnstarted()Lcom/google/common/base/Stopwatch;
at
org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.(TaskRunnerCallable.java:122)
at
org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl.submitWork(ContainerRunnerImpl.java:274)
at
org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.submitWork(LlapDaemon.java:558)
at
org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl.submitWork(LlapProtocolServerImpl.java:104)
at
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$LlapDaemonProtocol$2.callBlockingMethod(LlapDaemonProtocolProtos.java:19020)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
{code}
This can be solved by adding `com.google.common.base.Stopwatch.class` to this
section in hive 3.1.2 release
[https://github.com/apache/hive/blob/release-3.1.2-rc0/llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java#L385-L409]
and probably to this file for master branch
https://github.com/apache/hive/blob/2385361ef33ff81760115cfee4de31ec18afe2f3/llap-server/src/java/org/apache/hadoop/hive/llap/cli/service/AsyncTaskCopyLocalJars.java
was:
This JIRA https://issues.apache.org/jira/browse/HIVE-22126 shaded guava in
hive-exec. There is a issue that LLAP is also using this guava, so after
shading, it cannot use hive-exec guava and may turn to `./tez/guava-11.0.2.jar`
which will cause following error
{code:java}
Status: Running (Executing on YARN cluster with App id
application_1591081923777_0005)
Map 1: -/- Reducer 2: 0/1
Map 1: 0/11 Reducer 2: 0/1
Map 1: 0(+11,-11)/11Reducer 2: 0/1
Map 1: 0(+0,-32)/11 Reducer 2: 0/1
Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1591081923777_0005_1_00,
diagnostics=[Task failed, taskId=task_1591081923777_0005_1_00_02,
diagnostics=[TaskAttempt 0 failed,
info=[org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
com.google.common.base.Stopwatch.createUnstarted()Lcom/google/common/base/Stopwatch;
at
org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.(TaskRunnerCallable.java:122)
at
org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl.submitWork(ContainerRunnerImpl.java:274)
at
org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.submitWork(LlapDaemon.java:558)
at
org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl.submitWork(LlapProtocolServerImpl.java:104)
at
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$LlapDaemonProtocol$2.callBlockingMethod(LlapDaemonProtocolProtos.java:19020)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
{code}
This can be solved by adding `com.google.common.base.Stopwatch.class` to this
section
[jira] [Commented] (HIVE-23516) Store hive replication policy execution metrics in the relational DB
[
https://issues.apache.org/jira/browse/HIVE-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125271#comment-17125271
]
Hive QA commented on HIVE-23516:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-common in master failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 1523 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
52s{color} | {color:red} ql: The patch generated 1 new + 84 unchanged - 21
fixed = 85 total (was 105) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} ql generated 0 new + 1522 unchanged - 1 fixed = 1522
total (was 1523) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22736/dev-support/hive-personality.sh
|
| git revision | master / 2795d32 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22736/yetus/branch-findbugs-standalone-metastore_metastore-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22736/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22736/yetus/patch-findbugs-standalone-metastore_metastore-common.txt
|
| modules | C: standalone-metastore/metastore-common ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22736/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Store hive replication policy execution metrics in the relational DB
>
>
> Key: HIVE-23516
> URL: https://issues.apache.org/jira/browse/HIVE-23516
> Project: Hive
>
[jira] [Assigned] (HIVE-23603) transformDatabase() should work with changes from HIVE-22995
[ https://issues.apache.org/jira/browse/HIVE-23603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam reassigned HIVE-23603: > transformDatabase() should work with changes from HIVE-22995 > > > Key: HIVE-23603 > URL: https://issues.apache.org/jira/browse/HIVE-23603 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Fix For: 4.0.0 > > > The translation layer alters the locationUri on Database based on the > capabilities of the client. Now that we have separate locations for managed > and external for database, the implementation should be adjusted to work with > both locations. locationUri could already be external location. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22674) Replace Base64 in serde Package
[
https://issues.apache.org/jira/browse/HIVE-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125242#comment-17125242
]
Hive QA commented on HIVE-22674:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004744/HIVE-22674.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17102 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[compute_stats_binary]
(batchId=75)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.allocateNextWriteIdRetriesAfterDetectingConflictingConcurrentInsert
(batchId=245)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22735/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22735/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22735/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004744 - PreCommit-HIVE-Build
> Replace Base64 in serde Package
> ---
>
> Key: HIVE-22674
> URL: https://issues.apache.org/jira/browse/HIVE-22674
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22674.1.patch, HIVE-22674.2.patch,
> HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22674) Replace Base64 in serde Package
[
https://issues.apache.org/jira/browse/HIVE-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125212#comment-17125212
]
Hive QA commented on HIVE-22674:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} serde: The patch generated 1 new + 33 unchanged - 1
fixed = 34 total (was 34) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22735/dev-support/hive-personality.sh
|
| git revision | master / 2795d32 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22735/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22735/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Replace Base64 in serde Package
> ---
>
> Key: HIVE-22674
> URL: https://issues.apache.org/jira/browse/HIVE-22674
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22674.1.patch, HIVE-22674.2.patch,
> HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve
[
https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=440911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440911
]
ASF GitHub Bot logged work on HIVE-19261:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 18:13
Start Date: 03/Jun/20 18:13
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #807:
URL: https://github.com/apache/hive/pull/807#discussion_r434761769
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
##
@@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
- public synchronized Instance retrieve(SeedObject hv,
- Set seenSchemas) throws AvroSerdeException {
+ public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
if(cache.containsKey(hv)) {
Review comment:
I poked around a bit, nothing is obviously reentrant. Do you know where
this is happening?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440911)
Time Spent: 1h 20m (was: 1h 10m)
> Avro SerDe's InstanceCache should not be synchronized on retrieve
> -
>
> Key: HIVE-19261
> URL: https://issues.apache.org/jira/browse/HIVE-19261
> Project: Hive
> Issue Type: Improvement
>Reporter: Fangshi Li
>Assignee: Alexey Diomin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19261.1.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In HIVE-16175, upstream made a patch to fix the thread safety issue in
> AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache
> synchronized. While it should make InstanceCache thread-safe, making retrieve
> synchronized for the cache can be expensive in highly concurrent environment
> like Spark, as multiple threads need to be synchronized on entering the
> entire retrieve method.
> We are proposing another way to fix this thread safety issue by making the
> underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use
> atomic computeIfAbsent in the retrieve method to avoid synchronizing the
> entire method.
> While computeIfAbsent is only available on java 8 and java 7 is still
> supported in Hive,
> we use a pattern to simulate the behavior of computeIfAbsent. In the future,
> we should move to computeIfAbsent when Hive requires java 8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21895) Kafka Storage handler uses deprecated Kafka client methods
[
https://issues.apache.org/jira/browse/HIVE-21895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125187#comment-17125187
]
Hive QA commented on HIVE-21895:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972262/HIVE-21895.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22734/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22734/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22734/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:02:58.860
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22734/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:02:58.863
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 2795d32 HIVE-23595: Do not query task guaranteed status when wlm
off (Mustafa Iman via Gopal Vijayaraghavan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 2795d32 HIVE-23595: Do not query task guaranteed status when wlm
off (Mustafa Iman via Gopal Vijayaraghavan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:02:59.910
+ rm -rf ../yetus_PreCommit-HIVE-Build-22734
+ mkdir ../yetus_PreCommit-HIVE-Build-22734
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22734
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22734/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed:
kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java:46
Falling back to three-way merge...
Applied patch to
'kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java'
with conflicts.
Going to apply patch with: git apply -p0
error: patch failed:
kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java:46
Falling back to three-way merge...
Applied patch to
'kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java'
with conflicts.
U
kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-22734
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972262 - PreCommit-HIVE-Build
> Kafka Storage handler uses deprecated Kafka client methods
> --
>
> Key: HIVE-21895
> URL: https://issues.apache.org/jira/browse/HIVE-21895
> Project: Hive
> Issue Type: Improvement
> Components: kafka integration
>Affects Versions: 4.0.0
>Reporter: Kristopher Kane
>Assignee: Kristopher Kane
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21895.1.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Kafka client version is 2.2 and there are deprecated methods used like
> {code:java}
> producer.close(0, TimeUnit){code}
> in SimpleKafkaWriter
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21790) Bump Java to 1.8
[
https://issues.apache.org/jira/browse/HIVE-21790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125184#comment-17125184
]
Hive QA commented on HIVE-21790:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12969644/0001-HIVE-21790-Update-to-Java-1.8.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22733/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22733/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22733/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:01:39.195
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22733/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:01:39.198
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
847673a..2795d32 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 847673a HIVE-22017: Keep HMS interfaces backward compatible with
changes for HIVE-21637 (Kishen Das, reviewed by Vihang Karajgaonkar)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 2795d32 HIVE-23595: Do not query task guaranteed status when wlm
off (Mustafa Iman via Gopal Vijayaraghavan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:01:40.649
+ rm -rf ../yetus_PreCommit-HIVE-Build-22733
+ mkdir ../yetus_PreCommit-HIVE-Build-22733
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22733
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22733/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/README.md: does not exist in index
error: a/pom.xml: does not exist in index
Trying to apply the patch with -p1
error: patch failed: README.md:75
Falling back to three-way merge...
Applied patch to 'README.md' with conflicts.
error: patch failed: pom.xml:804
Falling back to three-way merge...
Applied patch to 'pom.xml' with conflicts.
Going to apply patch with: git apply -p1
error: patch failed: README.md:75
Falling back to three-way merge...
Applied patch to 'README.md' with conflicts.
error: patch failed: pom.xml:804
Falling back to three-way merge...
Applied patch to 'pom.xml' with conflicts.
U README.md
U pom.xml
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-22733
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12969644 - PreCommit-HIVE-Build
> Bump Java to 1.8
>
>
> Key: HIVE-21790
> URL: https://issues.apache.org/jira/browse/HIVE-21790
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.3.5
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Labels: pull-request-available
> Attachments: 0001-HIVE-21790-Update-to-Java-1.8.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> We're using Hive for reading Parquet files, but we would like to move from
> gzip to zstandard compression. Currently, the Parquet support of Hive is old
> because we can't upgrade since Parquet is Java 1.8+. Therefore it is a good
> idea to upgrade Hive as well.
> GA support of Java 1.7 is also almost over:
>
[jira] [Commented] (HIVE-23277) HiveProtoLogger should carry out JSON conversion in its own thread
[
https://issues.apache.org/jira/browse/HIVE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125182#comment-17125182
]
Hive QA commented on HIVE-23277:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004724/HIVE-23277.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17103 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22732/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22732/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22732/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004724 - PreCommit-HIVE-Build
> HiveProtoLogger should carry out JSON conversion in its own thread
> --
>
> Key: HIVE-23277
> URL: https://issues.apache.org/jira/browse/HIVE-23277
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Minor
> Attachments: HIVE-23277.1.patch, Screenshot 2020-04-23 at 11.27.42
> AM.png
>
>
> !Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23569) [RawStore] RawStore changes to facilitate HMS cache consistency
[ https://issues.apache.org/jira/browse/HIVE-23569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das reassigned HIVE-23569: - Assignee: (was: Kishen Das) > [RawStore] RawStore changes to facilitate HMS cache consistency > > > Key: HIVE-23569 > URL: https://issues.apache.org/jira/browse/HIVE-23569 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Priority: Major > > ObjectStore should use additional fields tableId and validWriteIdList fields > for all read methods to compare with cached ValidWriteIdList. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work stopped] (HIVE-23569) [RawStore] RawStore changes to facilitate HMS cache consistency
[ https://issues.apache.org/jira/browse/HIVE-23569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23569 stopped by Kishen Das. - > [RawStore] RawStore changes to facilitate HMS cache consistency > > > Key: HIVE-23569 > URL: https://issues.apache.org/jira/browse/HIVE-23569 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > > ObjectStore should use additional fields tableId and validWriteIdList fields > for all read methods to compare with cached ValidWriteIdList. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-22018) [ Interface changes ] Add table id to HMS get methods
[ https://issues.apache.org/jira/browse/HIVE-22018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das resolved HIVE-22018. --- Resolution: Duplicate This was done part of https://issues.apache.org/jira/browse/HIVE-22017 . > [ Interface changes ] Add table id to HMS get methods > - > > Key: HIVE-22018 > URL: https://issues.apache.org/jira/browse/HIVE-22018 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > > It is possible we remove a table and immediately move another table to occupy > the same name. CachedStore may retrieve the wrong table in this case. We > shall add tableid in every get_(table/partition) api, so we can compare the > one stored in TBLS (tableid is part of Table object) and check if the same > id, if not, HMS shall fail the read request. The initial table id can be > retrieved along with writeid (in DbTxnManager.getValidWriteIds call, to join > the TBLS table) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve
[
https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=440899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440899
]
ASF GitHub Bot logged work on HIVE-19261:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 17:35
Start Date: 03/Jun/20 17:35
Worklog Time Spent: 10m
Work Description: mythrocks commented on a change in pull request #807:
URL: https://github.com/apache/hive/pull/807#discussion_r434739477
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
##
@@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
- public synchronized Instance retrieve(SeedObject hv,
- Set seenSchemas) throws AvroSerdeException {
+ public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
if(cache.containsKey(hv)) {
Review comment:
I had left [a comment about this on
HIVE-19261](https://issues.apache.org/jira/browse/HIVE-19261?focusedCommentId=16929610=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16929610).
I think `makeInstance()` makes reentrant calls into
`InstanceCache::retrieve()`.
`computeIfAbsent()` expects that `cache` not be modified in the lambda.
Calling calling `makeInstance()` from `computeIfAbsent()` might lead to a
hang. I ran into this when testing an older version of this patch, some months
ago.
It's possible that the reentrant call no longer occurs. I'd check this
before using `computeIfAbsent()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440899)
Time Spent: 1h 10m (was: 1h)
> Avro SerDe's InstanceCache should not be synchronized on retrieve
> -
>
> Key: HIVE-19261
> URL: https://issues.apache.org/jira/browse/HIVE-19261
> Project: Hive
> Issue Type: Improvement
>Reporter: Fangshi Li
>Assignee: Alexey Diomin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19261.1.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In HIVE-16175, upstream made a patch to fix the thread safety issue in
> AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache
> synchronized. While it should make InstanceCache thread-safe, making retrieve
> synchronized for the cache can be expensive in highly concurrent environment
> like Spark, as multiple threads need to be synchronized on entering the
> entire retrieve method.
> We are proposing another way to fix this thread safety issue by making the
> underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use
> atomic computeIfAbsent in the retrieve method to avoid synchronizing the
> entire method.
> While computeIfAbsent is only available on java 8 and java 7 is still
> supported in Hive,
> we use a pattern to simulate the behavior of computeIfAbsent. In the future,
> we should move to computeIfAbsent when Hive requires java 8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23277) HiveProtoLogger should carry out JSON conversion in its own thread
[
https://issues.apache.org/jira/browse/HIVE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125152#comment-17125152
]
Hive QA commented on HIVE-23277:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 1523 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 18 new + 10 unchanged - 0
fixed = 28 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22732/dev-support/hive-personality.sh
|
| git revision | master / 64b8c0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22732/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22732/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveProtoLogger should carry out JSON conversion in its own thread
> --
>
> Key: HIVE-23277
> URL: https://issues.apache.org/jira/browse/HIVE-23277
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Minor
> Attachments: HIVE-23277.1.patch, Screenshot 2020-04-23 at 11.27.42
> AM.png
>
>
> !Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23596) Encode guaranteed task information in containerId
[ https://issues.apache.org/jira/browse/HIVE-23596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125151#comment-17125151 ] Gopal Vijayaraghavan commented on HIVE-23596: - LGTM - +1 Will add extra comments in the LlapTaskScheduler, since this takes advantage of arbitrary container-ids which have a full 64 bit range & do not get used outside of the runtime of the query (unlike say, task-ids, which show up in filenames). The LLAP daemon side doesn't care about a container-id, since we use a 1:1 correspondence between containers & tasks in LLAP. So a "tagged pointer" or "colored pointer" equivalent implementation is safe. > Encode guaranteed task information in containerId > - > > Key: HIVE-23596 > URL: https://issues.apache.org/jira/browse/HIVE-23596 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We should avoid calling LlapTaskScheduler to get initial isguaranteed flag > for all the tasks. It causes arbitrary delays in sending tasks out. Since > communicator is a single thread, any blocking there delays all the tasks. > There are [https://jira.apache.org/jira/browse/TEZ-4192] and > [https://jira.apache.org/jira/browse/HIVE-23589] for a proper solution to > this. However, that requires a Tez release which seems far right now. We can > replace the current hack with another hack that does not require locking. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23595) Do not query task guaranteed status when wlm off
[ https://issues.apache.org/jira/browse/HIVE-23595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23595: Resolution: Fixed Status: Resolved (was: Patch Available) > Do not query task guaranteed status when wlm off > > > Key: HIVE-23595 > URL: https://issues.apache.org/jira/browse/HIVE-23595 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23595.patch > > > LlapTaskCommunicator queries scheduler for every task guaranteed status. When > workload management is off it is always false. There is no need for the > synchronous check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23595) Do not query task guaranteed status when wlm off
[ https://issues.apache.org/jira/browse/HIVE-23595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23595: Fix Version/s: 4.0.0 > Do not query task guaranteed status when wlm off > > > Key: HIVE-23595 > URL: https://issues.apache.org/jira/browse/HIVE-23595 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23595.patch > > > LlapTaskCommunicator queries scheduler for every task guaranteed status. When > workload management is off it is always false. There is no need for the > synchronous check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23595) Do not query task guaranteed status when wlm off
[ https://issues.apache.org/jira/browse/HIVE-23595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125146#comment-17125146 ] László Bodor commented on HIVE-23595: - pushed to master, thanks [~mustafaiman] for the patch and [~gopalv] for the review! > Do not query task guaranteed status when wlm off > > > Key: HIVE-23595 > URL: https://issues.apache.org/jira/browse/HIVE-23595 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Attachments: HIVE-23595.patch > > > LlapTaskCommunicator queries scheduler for every task guaranteed status. When > workload management is off it is always false. There is no need for the > synchronous check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23358) MSCK REPAIR should remove all insignificant zeroes from partition values (for numeric datatypes) before creating the partitions
[
https://issues.apache.org/jira/browse/HIVE-23358?focusedWorklogId=440891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440891
]
ASF GitHub Bot logged work on HIVE-23358:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 17:05
Start Date: 03/Jun/20 17:05
Worklog Time Spent: 10m
Work Description: adesh-rao commented on pull request #1050:
URL: https://github.com/apache/hive/pull/1050#issuecomment-638329732
@sankarh Can you take a look at the PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440891)
Time Spent: 20m (was: 10m)
> MSCK REPAIR should remove all insignificant zeroes from partition values (for
> numeric datatypes) before creating the partitions
> ---
>
> Key: HIVE-23358
> URL: https://issues.apache.org/jira/browse/HIVE-23358
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For the following scenario
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1;
> {noformat}
> ++
> | partition |
> ++
> | year=2020/month=03/day=10 |
> | year=2020/month=03/day=11 |
> ++
> {noformat}
> 5.show table extended like 't1' partition (Year=2020, Month=03, Day=11);
> will throw an error:
> {noformat}
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10006]: Partition not found {year=2020, month=3, day=11}
> (state=42000,code=10006)
> {noformat}
> When the partition directory are created without the extra zeroes, this works
> fine.
> {noformat}
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=11
> {noformat}
> This happens because while searching for partitions, hive strips the extra
> "0" in month key and then queries the metastore
> (partSpec="year=2020/month=3/day=10") which returns no rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23516) Store hive replication policy execution metrics in the relational DB
[ https://issues.apache.org/jira/browse/HIVE-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23516: --- Status: In Progress (was: Patch Available) > Store hive replication policy execution metrics in the relational DB > > > Key: HIVE-23516 > URL: https://issues.apache.org/jira/browse/HIVE-23516 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23516.01.patch, HIVE-23516.02.patch, > HIVE-23516.03.patch, HIVE-23516.04.patch, HIVE-23516.05.patch, > HIVE-23516.06.patch, Replication Metrics.pdf > > Time Spent: 4.5h > Remaining Estimate: 0h > > Details documented in the attached doc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23516) Store hive replication policy execution metrics in the relational DB
[ https://issues.apache.org/jira/browse/HIVE-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23516: --- Attachment: HIVE-23516.06.patch Status: Patch Available (was: In Progress) > Store hive replication policy execution metrics in the relational DB > > > Key: HIVE-23516 > URL: https://issues.apache.org/jira/browse/HIVE-23516 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23516.01.patch, HIVE-23516.02.patch, > HIVE-23516.03.patch, HIVE-23516.04.patch, HIVE-23516.05.patch, > HIVE-23516.06.patch, Replication Metrics.pdf > > Time Spent: 4.5h > Remaining Estimate: 0h > > Details documented in the attached doc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22681) Replace Base64 in hcatalog-webhcat Package
[ https://issues.apache.org/jira/browse/HIVE-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22681: -- Attachment: HIVE-22681.2.patch > Replace Base64 in hcatalog-webhcat Package > -- > > Key: HIVE-22681 > URL: https://issues.apache.org/jira/browse/HIVE-22681 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22681.1.patch, HIVE-22681.1.patch, > HIVE-22681.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22681) Replace Base64 in hcatalog-webhcat Package
[ https://issues.apache.org/jira/browse/HIVE-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22681: -- Attachment: (was: HIVE-22681.1.patch) > Replace Base64 in hcatalog-webhcat Package > -- > > Key: HIVE-22681 > URL: https://issues.apache.org/jira/browse/HIVE-22681 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22681.1.patch, HIVE-22681.1.patch, > HIVE-22681.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22681) Replace Base64 in hcatalog-webhcat Package
[ https://issues.apache.org/jira/browse/HIVE-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22681: -- Attachment: (was: HIVE-22681.1.patch) > Replace Base64 in hcatalog-webhcat Package > -- > > Key: HIVE-22681 > URL: https://issues.apache.org/jira/browse/HIVE-22681 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22681.1.patch, HIVE-22681.1.patch, > HIVE-22681.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22675) Replace Base64 in hive-standalone-metastore Package
[ https://issues.apache.org/jira/browse/HIVE-22675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22675: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~ashutoshc] for the review! > Replace Base64 in hive-standalone-metastore Package > --- > > Key: HIVE-22675 > URL: https://issues.apache.org/jira/browse/HIVE-22675 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22675.1.patch, HIVE-22675.2.patch, > HIVE-22675.3.patch, HIVE-22675.3.patch, HIVE-22675.3.patch, > HIVE-22675.3.patch, HIVE-22675.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22674) Replace Base64 in serde Package
[ https://issues.apache.org/jira/browse/HIVE-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22674: -- Attachment: HIVE-22674.2.patch > Replace Base64 in serde Package > --- > > Key: HIVE-22674 > URL: https://issues.apache.org/jira/browse/HIVE-22674 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22674.1.patch, HIVE-22674.2.patch, > HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-20784) Migrate hbase.util.Base64 to java.util.Base64
[ https://issues.apache.org/jira/browse/HIVE-20784?focusedWorklogId=440887=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440887 ] ASF GitHub Bot logged work on HIVE-20784: - Author: ASF GitHub Bot Created on: 03/Jun/20 16:52 Start Date: 03/Jun/20 16:52 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #451: URL: https://github.com/apache/hive/pull/451#issuecomment-638321924 @kgyrtkirk Please assist :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440887) Time Spent: 20m (was: 10m) > Migrate hbase.util.Base64 to java.util.Base64 > - > > Key: HIVE-20784 > URL: https://issues.apache.org/jira/browse/HIVE-20784 > Project: Hive > Issue Type: Sub-task > Components: HBase Handler >Affects Versions: 3.1.0 > Environment: HBase 2.0.2 > Hive 3.1.0 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > By default Hive 3.1.0 depends on HBase 2.0.0-alpha4. HBase 2.0.2 migrated > from hbase.util.Base64 to java.util.Base64 (HBASE-20884), which causes Hive > 3.1.0 fails to build with HBase 2.0.2. > > $ cd hbase-handler > $ mvn package -DskipTests -Dhbase.version=2.0.2 > [ERROR] > .../hive/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[29,36] > cannot find symbol > [ERROR] symbol: class Base64 > [ERROR] location: package org.apache.hadoop.hbase.util > > To make Hive works with 2.0.2+ (and also older versions), we should consider > migrating Hive to java.util.Base64. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22017) [ Interface changes ] Keep HMS interfaces backward compatible with changes for HIVE-21637
[ https://issues.apache.org/jira/browse/HIVE-22017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-22017: --- Target Version/s: (was: 2.3.7) > [ Interface changes ] Keep HMS interfaces backward compatible with changes > for HIVE-21637 > - > > Key: HIVE-22017 > URL: https://issues.apache.org/jira/browse/HIVE-22017 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.3.7 >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22017.1.patch, HIVE-22017.2.patch, > HIVE-22017.3.patch, HIVE-22017.4.patch, HIVE-22017.5.patch, > HIVE-22017.6.patch, HIVE-22017.7.patch, HIVE-22017.8.patch > > > As part of HIVE-21637 we would have to introduce ValidWriteIdList in several > methods. Also, in the long term, we should deprecate and remove all the > methods that take direct arguments, as the service definition keeps changing > whenever we add/remove arguments, making it hard to maintain backward > compatibility. So, instead of adding writeId in bunch of get_xxx calls that > take direct arguments, we will create new set of methods that take Request > object and return Response object. We shall mark those deprecated and remove > in future version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[
https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125130#comment-17125130
]
Hive QA commented on HIVE-23242:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004711/HIVE-23242.5.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17096 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22731/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22731/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22731/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004711 - PreCommit-HIVE-Build
> Fix flaky tests testHouseKeepingThreadExistence in
> TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
> -
>
> Key: HIVE-23242
> URL: https://issues.apache.org/jira/browse/HIVE-23242
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch,
> HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch
>
>
> Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for
> details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22017) [ Interface changes ] Keep HMS interfaces backward compatible with changes for HIVE-21637
[ https://issues.apache.org/jira/browse/HIVE-22017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-22017: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > [ Interface changes ] Keep HMS interfaces backward compatible with changes > for HIVE-21637 > - > > Key: HIVE-22017 > URL: https://issues.apache.org/jira/browse/HIVE-22017 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.3.7 >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22017.1.patch, HIVE-22017.2.patch, > HIVE-22017.3.patch, HIVE-22017.4.patch, HIVE-22017.5.patch, > HIVE-22017.6.patch, HIVE-22017.7.patch, HIVE-22017.8.patch > > > As part of HIVE-21637 we would have to introduce ValidWriteIdList in several > methods. Also, in the long term, we should deprecate and remove all the > methods that take direct arguments, as the service definition keeps changing > whenever we add/remove arguments, making it hard to maintain backward > compatibility. So, instead of adding writeId in bunch of get_xxx calls that > take direct arguments, we will create new set of methods that take Request > object and return Response object. We shall mark those deprecated and remove > in future version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22017) [ Interface changes ] Keep HMS interfaces backward compatible with changes for HIVE-21637
[ https://issues.apache.org/jira/browse/HIVE-22017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125128#comment-17125128 ] Vihang Karajgaonkar commented on HIVE-22017: Patch merged into master. Thanks for your contribution [~kishendas] > [ Interface changes ] Keep HMS interfaces backward compatible with changes > for HIVE-21637 > - > > Key: HIVE-22017 > URL: https://issues.apache.org/jira/browse/HIVE-22017 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.3.7 >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Attachments: HIVE-22017.1.patch, HIVE-22017.2.patch, > HIVE-22017.3.patch, HIVE-22017.4.patch, HIVE-22017.5.patch, > HIVE-22017.6.patch, HIVE-22017.7.patch, HIVE-22017.8.patch > > > As part of HIVE-21637 we would have to introduce ValidWriteIdList in several > methods. Also, in the long term, we should deprecate and remove all the > methods that take direct arguments, as the service definition keeps changing > whenever we add/remove arguments, making it hard to maintain backward > compatibility. So, instead of adding writeId in bunch of get_xxx calls that > take direct arguments, we will create new set of methods that take Request > object and return Response object. We shall mark those deprecated and remove > in future version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-20784) Migrate hbase.util.Base64 to java.util.Base64
[ https://issues.apache.org/jira/browse/HIVE-20784?focusedWorklogId=440886=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440886 ] ASF GitHub Bot logged work on HIVE-20784: - Author: ASF GitHub Bot Created on: 03/Jun/20 16:51 Start Date: 03/Jun/20 16:51 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #451: URL: https://github.com/apache/hive/pull/451#issuecomment-638321777 Nice fix. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440886) Remaining Estimate: 0h Time Spent: 10m > Migrate hbase.util.Base64 to java.util.Base64 > - > > Key: HIVE-20784 > URL: https://issues.apache.org/jira/browse/HIVE-20784 > Project: Hive > Issue Type: Sub-task > Components: HBase Handler >Affects Versions: 3.1.0 > Environment: HBase 2.0.2 > Hive 3.1.0 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Critical > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > By default Hive 3.1.0 depends on HBase 2.0.0-alpha4. HBase 2.0.2 migrated > from hbase.util.Base64 to java.util.Base64 (HBASE-20884), which causes Hive > 3.1.0 fails to build with HBase 2.0.2. > > $ cd hbase-handler > $ mvn package -DskipTests -Dhbase.version=2.0.2 > [ERROR] > .../hive/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[29,36] > cannot find symbol > [ERROR] symbol: class Base64 > [ERROR] location: package org.apache.hadoop.hbase.util > > To make Hive works with 2.0.2+ (and also older versions), we should consider > migrating Hive to java.util.Base64. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20784) Migrate hbase.util.Base64 to java.util.Base64
[ https://issues.apache.org/jira/browse/HIVE-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-20784: -- Parent: HIVE-22611 Issue Type: Sub-task (was: Improvement) > Migrate hbase.util.Base64 to java.util.Base64 > - > > Key: HIVE-20784 > URL: https://issues.apache.org/jira/browse/HIVE-20784 > Project: Hive > Issue Type: Sub-task > Components: HBase Handler >Affects Versions: 3.1.0 > Environment: HBase 2.0.2 > Hive 3.1.0 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Critical > Labels: pull-request-available > > By default Hive 3.1.0 depends on HBase 2.0.0-alpha4. HBase 2.0.2 migrated > from hbase.util.Base64 to java.util.Base64 (HBASE-20884), which causes Hive > 3.1.0 fails to build with HBase 2.0.2. > > $ cd hbase-handler > $ mvn package -DskipTests -Dhbase.version=2.0.2 > [ERROR] > .../hive/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[29,36] > cannot find symbol > [ERROR] symbol: class Base64 > [ERROR] location: package org.apache.hadoop.hbase.util > > To make Hive works with 2.0.2+ (and also older versions), we should consider > migrating Hive to java.util.Base64. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21141) Fix some spell errors in Hive
[ https://issues.apache.org/jira/browse/HIVE-21141?focusedWorklogId=440884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440884 ] ASF GitHub Bot logged work on HIVE-21141: - Author: ASF GitHub Bot Created on: 03/Jun/20 16:47 Start Date: 03/Jun/20 16:47 Worklog Time Spent: 10m Work Description: belugabehr commented on a change in pull request #519: URL: https://github.com/apache/hive/pull/519#discussion_r434711070 ## File path: hcatalog/src/docs/src/documentation/skinconf.xml ## @@ -38,11 +38,11 @@ which will be used to configure the chosen Forrest skin. false true - + true > Key: HIVE-21141 > URL: https://issues.apache.org/jira/browse/HIVE-21141 > Project: Hive > Issue Type: Bug >Reporter: Bo Xu >Assignee: Bo Xu >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21141.1.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Fix som spell errors in Hive -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21895) Kafka Storage handler uses deprecated Kafka client methods
[
https://issues.apache.org/jira/browse/HIVE-21895?focusedWorklogId=440873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440873
]
ASF GitHub Bot logged work on HIVE-21895:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 16:38
Start Date: 03/Jun/20 16:38
Worklog Time Spent: 10m
Work Description: belugabehr commented on pull request #680:
URL: https://github.com/apache/hive/pull/680#issuecomment-638314315
Can you please take another crack at this?
Also, be mindful of this situation:
`
LOG.info("Flushing Kafka Producer with writerId [{}]", writerId);
producer.flush();
LOG.info("Closing WriterId [{}]", writerId);
producer.close();
`
I think the 'flush' should be wrapped in a try-finally block with the close.
Right now, if the flush fails, the close is skipped.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440873)
Time Spent: 20m (was: 10m)
> Kafka Storage handler uses deprecated Kafka client methods
> --
>
> Key: HIVE-21895
> URL: https://issues.apache.org/jira/browse/HIVE-21895
> Project: Hive
> Issue Type: Improvement
> Components: kafka integration
>Affects Versions: 4.0.0
>Reporter: Kristopher Kane
>Assignee: Kristopher Kane
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21895.1.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Kafka client version is 2.2 and there are deprecated methods used like
> {code:java}
> producer.close(0, TimeUnit){code}
> in SimpleKafkaWriter
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23520) REPL: repl dump could add support for immutable dataset
[ https://issues.apache.org/jira/browse/HIVE-23520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125092#comment-17125092 ] Aasha Medhi commented on HIVE-23520: Is this both for managed and external tables? There is a Metadata only config for external tables which dumps partition, stats and metadata without dumping any data. Will that be useful? Can you please add a pull request where I can add the review comments > REPL: repl dump could add support for immutable dataset > --- > > Key: HIVE-23520 > URL: https://issues.apache.org/jira/browse/HIVE-23520 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23520.1.patch > > > Currently, "REPL DUMP" ends up copying entire dataset along with partition > information, stats etc in its dump folder. However, there are cases (e.g > large reference datasets), where we need a way to just retain metadata along > with partition information & stats. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23499) REPL: repl load should honor "hive.repl.dump.metadata.only=true"
[
https://issues.apache.org/jira/browse/HIVE-23499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123745#comment-17123745
]
Aasha Medhi edited comment on HIVE-23499 at 6/3/20, 4:19 PM:
-
hive.repl.dump.metadata.only=true is a dump side configuration. Should we reuse
the same for load? You can use the immutable property for this.
Can you please add a pull request for the same to add the review comments.
was (Author: aasha):
hive.repl.dump.metadata.only=true is a dump side configuration. Should we reuse
the same for load?
Can you please add a pull request for the same to add the review comments.
> REPL: repl load should honor "hive.repl.dump.metadata.only=true"
>
>
> Key: HIVE-23499
> URL: https://issues.apache.org/jira/browse/HIVE-23499
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23499.1.patch
>
>
> "{{hive.repl.dump.metadata.only=true"}} is not currently honored during
> "{{repl load"}}. Currently, it ends up copying all files even if this option
> is specified in "repl load". E.g
> {noformat}
> repl load airline_ontime_orc into another_airline_ontime_orc with
> ('hive.repl.rootdir'='s3a://blah/', 'hive.repl.dump.metadata.only'='true');
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21790) Bump Java to 1.8
[ https://issues.apache.org/jira/browse/HIVE-21790?focusedWorklogId=440855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440855 ] ASF GitHub Bot logged work on HIVE-21790: - Author: ASF GitHub Bot Created on: 03/Jun/20 16:14 Start Date: 03/Jun/20 16:14 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #692: URL: https://github.com/apache/hive/pull/692#issuecomment-638300946 @Fokko Are you still interested in doing this? Hive 3.x support Java 8 and we plan on supporting Java 11 on Hive 4.x This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440855) Time Spent: 40m (was: 0.5h) > Bump Java to 1.8 > > > Key: HIVE-21790 > URL: https://issues.apache.org/jira/browse/HIVE-21790 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.5 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21790-Update-to-Java-1.8.patch > > Time Spent: 40m > Remaining Estimate: 0h > > We're using Hive for reading Parquet files, but we would like to move from > gzip to zstandard compression. Currently, the Parquet support of Hive is old > because we can't upgrade since Parquet is Java 1.8+. Therefore it is a good > idea to upgrade Hive as well. > GA support of Java 1.7 is also almost over: > https://www.oracle.com/technetwork/java/java-se-support-roadmap.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[
https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125081#comment-17125081
]
Hive QA commented on HIVE-23242:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} metastore-server in master failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 427 unchanged - 2 fixed = 427 total (was 429) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22731/dev-support/hive-personality.sh
|
| git revision | master / c886653 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22731/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22731/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22731/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix flaky tests testHouseKeepingThreadExistence in
> TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
> -
>
> Key: HIVE-23242
> URL: https://issues.apache.org/jira/browse/HIVE-23242
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch,
> HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch
>
>
>
[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve
[
https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=440851=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440851
]
ASF GitHub Bot logged work on HIVE-19261:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 16:01
Start Date: 03/Jun/20 16:01
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #807:
URL: https://github.com/apache/hive/pull/807#discussion_r434678396
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
##
@@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
- public synchronized Instance retrieve(SeedObject hv,
- Set seenSchemas) throws AvroSerdeException {
+ public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
Review comment:
Please change this (and the rest) to:
`
LOG.debug("Checking for hv: {}", hv);
`
http://www.slf4j.org/faq.html#logging_performance
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
##
@@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
- public synchronized Instance retrieve(SeedObject hv,
- Set seenSchemas) throws AvroSerdeException {
+ public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
if(cache.containsKey(hv)) {
Review comment:
Please leverage `Map#computeIfAbsent` here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440851)
Time Spent: 50m (was: 40m)
> Avro SerDe's InstanceCache should not be synchronized on retrieve
> -
>
> Key: HIVE-19261
> URL: https://issues.apache.org/jira/browse/HIVE-19261
> Project: Hive
> Issue Type: Improvement
>Reporter: Fangshi Li
>Assignee: Alexey Diomin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19261.1.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In HIVE-16175, upstream made a patch to fix the thread safety issue in
> AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache
> synchronized. While it should make InstanceCache thread-safe, making retrieve
> synchronized for the cache can be expensive in highly concurrent environment
> like Spark, as multiple threads need to be synchronized on entering the
> entire retrieve method.
> We are proposing another way to fix this thread safety issue by making the
> underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use
> atomic computeIfAbsent in the retrieve method to avoid synchronizing the
> entire method.
> While computeIfAbsent is only available on java 8 and java 7 is still
> supported in Hive,
> we use a pattern to simulate the behavior of computeIfAbsent. In the future,
> we should move to computeIfAbsent when Hive requires java 8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve
[ https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=440852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440852 ] ASF GitHub Bot logged work on HIVE-19261: - Author: ASF GitHub Bot Created on: 03/Jun/20 16:01 Start Date: 03/Jun/20 16:01 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #807: URL: https://github.com/apache/hive/pull/807#issuecomment-638292070 Hive project is testing out using GitHub now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440852) Time Spent: 1h (was: 50m) > Avro SerDe's InstanceCache should not be synchronized on retrieve > - > > Key: HIVE-19261 > URL: https://issues.apache.org/jira/browse/HIVE-19261 > Project: Hive > Issue Type: Improvement >Reporter: Fangshi Li >Assignee: Alexey Diomin >Priority: Major > Labels: pull-request-available > Attachments: HIVE-19261.1.patch > > Time Spent: 1h > Remaining Estimate: 0h > > In HIVE-16175, upstream made a patch to fix the thread safety issue in > AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache > synchronized. While it should make InstanceCache thread-safe, making retrieve > synchronized for the cache can be expensive in highly concurrent environment > like Spark, as multiple threads need to be synchronized on entering the > entire retrieve method. > We are proposing another way to fix this thread safety issue by making the > underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use > atomic computeIfAbsent in the retrieve method to avoid synchronizing the > entire method. > While computeIfAbsent is only available on java 8 and java 7 is still > supported in Hive, > we use a pattern to simulate the behavior of computeIfAbsent. In the future, > we should move to computeIfAbsent when Hive requires java 8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22360) MultiDelimitSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
[
https://issues.apache.org/jira/browse/HIVE-22360?focusedWorklogId=440849=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440849
]
ASF GitHub Bot logged work on HIVE-22360:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:55
Start Date: 03/Jun/20 15:55
Worklog Time Spent: 10m
Work Description: belugabehr commented on pull request #823:
URL: https://github.com/apache/hive/pull/823#issuecomment-638288379
Take a look at my related PR #815
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440849)
Time Spent: 20m (was: 10m)
> MultiDelimitSerDe returns wrong results in last column when the loaded file
> has more columns than those in table schema
> ---
>
> Key: HIVE-22360
> URL: https://issues.apache.org/jira/browse/HIVE-22360
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22360.1.patch, HIVE-22360.2.patch,
> HIVE-22360.3.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Repro steps:
> Input file:
> {code}
> 1^,1^,^,0^,0^,0
> 2^,1^,^,0^,1^,0
> 3^,1^,^,0^,0^,0
> 4^,1^,^,0^,1^,0
> {code}
> Queries:
> {code}
> CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE
> smallint) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.MultiDelimitSerDe'
> WITH SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv'
> OVERWRITE INTO TABLE n2;
> select * from n2;
> // wrong last column results here.
> +--+--+--+--+--+
> | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole |
> +--+--+--+--+--+
> | 1| 1| NULL | 0| NULL |
> | 2| 1| NULL | 0| NULL |
> | 3| 1| NULL | 0| NULL |
> | 4| 1| NULL | 0| NULL |
> +--+--+--+--+--+
> {code}
> Cause:
> In multi-serde parsing, the total length calculation here:
> https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308
> does not take extra fields into account.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22771) Partition location incorrectly formed in FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-22771?focusedWorklogId=440844=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440844
]
ASF GitHub Bot logged work on HIVE-22771:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:50
Start Date: 03/Jun/20 15:50
Worklog Time Spent: 10m
Work Description: Shivamohan07 closed pull request #889:
URL: https://github.com/apache/hive/pull/889
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440844)
Time Spent: 40m (was: 0.5h)
> Partition location incorrectly formed in FileOutputCommitterContainer
> -
>
> Key: HIVE-22771
> URL: https://issues.apache.org/jira/browse/HIVE-22771
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1
>Reporter: Shivam
>Assignee: Shivam
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22771.2.patch, HIVE-22771.3.patch,
> HIVE-22771.4.patch, HIVE-22771.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Class _HCatOutputFormat_ in package _org.apache.hive.hcatalog.mapreduce_ uses
> function _setOutput_ to generate _idHash_ using below statement:
> *+In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java+*
> *line 116: idHash = String.valueOf(Math.random());*
> The output of idHash can be similar to values like this : 7.145347157239135E-4
>
> And, in class _FileOutputCommitterContainer_ in package
> _org.apache.hive.hcatalog.mapreduce;_
> Uses below statement to compute final partition path:
> +*In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java*+
> *line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR +
> SCRATCH_DIR_NAME + "{color:#ff}\\d
> .?
> d+"{color},"");*
> *line 367: partPath = new Path(finalLocn);*
>
> Regex used here is incorrect, since it will only remove integers after the
> *SCRATCH_DIR_NAME,* and hence will append 'E-4' (for the above example) in
> the final partition location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22771) Partition location incorrectly formed in FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-22771?focusedWorklogId=440845=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440845
]
ASF GitHub Bot logged work on HIVE-22771:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:50
Start Date: 03/Jun/20 15:50
Worklog Time Spent: 10m
Work Description: Shivamohan07 commented on pull request #889:
URL: https://github.com/apache/hive/pull/889#issuecomment-638285159
Thanks
> Please close. Committed to Hive master through JIRA. Good fix!
Thanks!, Closing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440845)
Time Spent: 50m (was: 40m)
> Partition location incorrectly formed in FileOutputCommitterContainer
> -
>
> Key: HIVE-22771
> URL: https://issues.apache.org/jira/browse/HIVE-22771
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1
>Reporter: Shivam
>Assignee: Shivam
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22771.2.patch, HIVE-22771.3.patch,
> HIVE-22771.4.patch, HIVE-22771.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Class _HCatOutputFormat_ in package _org.apache.hive.hcatalog.mapreduce_ uses
> function _setOutput_ to generate _idHash_ using below statement:
> *+In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java+*
> *line 116: idHash = String.valueOf(Math.random());*
> The output of idHash can be similar to values like this : 7.145347157239135E-4
>
> And, in class _FileOutputCommitterContainer_ in package
> _org.apache.hive.hcatalog.mapreduce;_
> Uses below statement to compute final partition path:
> +*In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java*+
> *line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR +
> SCRATCH_DIR_NAME + "{color:#ff}\\d
> .?
> d+"{color},"");*
> *line 367: partPath = new Path(finalLocn);*
>
> Regex used here is incorrect, since it will only remove integers after the
> *SCRATCH_DIR_NAME,* and hence will append 'E-4' (for the above example) in
> the final partition location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22771) Partition location incorrectly formed in FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-22771?focusedWorklogId=440847=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440847
]
ASF GitHub Bot logged work on HIVE-22771:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:50
Start Date: 03/Jun/20 15:50
Worklog Time Spent: 10m
Work Description: Shivamohan07 edited a comment on pull request #889:
URL: https://github.com/apache/hive/pull/889#issuecomment-638285159
> Please close. Committed to Hive master through JIRA. Good fix!
Thanks!, Closing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440847)
Time Spent: 1h (was: 50m)
> Partition location incorrectly formed in FileOutputCommitterContainer
> -
>
> Key: HIVE-22771
> URL: https://issues.apache.org/jira/browse/HIVE-22771
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1
>Reporter: Shivam
>Assignee: Shivam
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22771.2.patch, HIVE-22771.3.patch,
> HIVE-22771.4.patch, HIVE-22771.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Class _HCatOutputFormat_ in package _org.apache.hive.hcatalog.mapreduce_ uses
> function _setOutput_ to generate _idHash_ using below statement:
> *+In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java+*
> *line 116: idHash = String.valueOf(Math.random());*
> The output of idHash can be similar to values like this : 7.145347157239135E-4
>
> And, in class _FileOutputCommitterContainer_ in package
> _org.apache.hive.hcatalog.mapreduce;_
> Uses below statement to compute final partition path:
> +*In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java*+
> *line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR +
> SCRATCH_DIR_NAME + "{color:#ff}\\d
> .?
> d+"{color},"");*
> *line 367: partPath = new Path(finalLocn);*
>
> Regex used here is incorrect, since it will only remove integers after the
> *SCRATCH_DIR_NAME,* and hence will append 'E-4' (for the above example) in
> the final partition location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22771) Partition location incorrectly formed in FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-22771?focusedWorklogId=440840=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440840
]
ASF GitHub Bot logged work on HIVE-22771:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:46
Start Date: 03/Jun/20 15:46
Worklog Time Spent: 10m
Work Description: belugabehr edited a comment on pull request #889:
URL: https://github.com/apache/hive/pull/889#issuecomment-638282682
Please close. Committed to Hive master through JIRA. Good fix!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440840)
Time Spent: 0.5h (was: 20m)
> Partition location incorrectly formed in FileOutputCommitterContainer
> -
>
> Key: HIVE-22771
> URL: https://issues.apache.org/jira/browse/HIVE-22771
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1
>Reporter: Shivam
>Assignee: Shivam
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22771.2.patch, HIVE-22771.3.patch,
> HIVE-22771.4.patch, HIVE-22771.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Class _HCatOutputFormat_ in package _org.apache.hive.hcatalog.mapreduce_ uses
> function _setOutput_ to generate _idHash_ using below statement:
> *+In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java+*
> *line 116: idHash = String.valueOf(Math.random());*
> The output of idHash can be similar to values like this : 7.145347157239135E-4
>
> And, in class _FileOutputCommitterContainer_ in package
> _org.apache.hive.hcatalog.mapreduce;_
> Uses below statement to compute final partition path:
> +*In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java*+
> *line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR +
> SCRATCH_DIR_NAME + "{color:#ff}\\d
> .?
> d+"{color},"");*
> *line 367: partPath = new Path(finalLocn);*
>
> Regex used here is incorrect, since it will only remove integers after the
> *SCRATCH_DIR_NAME,* and hence will append 'E-4' (for the above example) in
> the final partition location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22771) Partition location incorrectly formed in FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-22771?focusedWorklogId=440838=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440838
]
ASF GitHub Bot logged work on HIVE-22771:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:45
Start Date: 03/Jun/20 15:45
Worklog Time Spent: 10m
Work Description: belugabehr commented on pull request #889:
URL: https://github.com/apache/hive/pull/889#issuecomment-638282682
Please close. Committed to Hive master through JIRA.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440838)
Time Spent: 20m (was: 10m)
> Partition location incorrectly formed in FileOutputCommitterContainer
> -
>
> Key: HIVE-22771
> URL: https://issues.apache.org/jira/browse/HIVE-22771
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1
>Reporter: Shivam
>Assignee: Shivam
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22771.2.patch, HIVE-22771.3.patch,
> HIVE-22771.4.patch, HIVE-22771.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Class _HCatOutputFormat_ in package _org.apache.hive.hcatalog.mapreduce_ uses
> function _setOutput_ to generate _idHash_ using below statement:
> *+In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java+*
> *line 116: idHash = String.valueOf(Math.random());*
> The output of idHash can be similar to values like this : 7.145347157239135E-4
>
> And, in class _FileOutputCommitterContainer_ in package
> _org.apache.hive.hcatalog.mapreduce;_
> Uses below statement to compute final partition path:
> +*In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java*+
> *line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR +
> SCRATCH_DIR_NAME + "{color:#ff}\\d
> .?
> d+"{color},"");*
> *line 367: partPath = new Path(finalLocn);*
>
> Regex used here is incorrect, since it will only remove integers after the
> *SCRATCH_DIR_NAME,* and hence will append 'E-4' (for the above example) in
> the final partition location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[
https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125061#comment-17125061
]
Hive QA commented on HIVE-23340:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004710/HIVE-23340.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17094 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22730/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22730/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22730/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004710 - PreCommit-HIVE-Build
> TxnHandler cleanup
> --
>
> Key: HIVE-23340
> URL: https://issues.apache.org/jira/browse/HIVE-23340
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch,
> HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch,
> HIVE-23340.6.patch, HIVE-23340.7.patch
>
>
> * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication
> * Remove TxnStatus character constants and use the enum values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23026) Support add a yarn application name for tez on hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-23026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23026: -- Labels: pull-request-available (was: ) > Support add a yarn application name for tez on hiveserver2 > -- > > Key: HIVE-23026 > URL: https://issues.apache.org/jira/browse/HIVE-23026 > Project: Hive > Issue Type: Improvement > Components: Tez >Affects Versions: 2.3.5 >Reporter: Jake Xie >Priority: Major > Labels: pull-request-available > Fix For: 2.3.5 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently tez on hiveServer2 cannot specify yarn application name, which is > not very convenient for locating the problem SQL, so i added a configuration > item to support setting tez job name -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23026) Support add a yarn application name for tez on hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-23026?focusedWorklogId=440833=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440833
]
ASF GitHub Bot logged work on HIVE-23026:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:21
Start Date: 03/Jun/20 15:21
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #947:
URL: https://github.com/apache/hive/pull/947#discussion_r434641680
##
File path: .travis.yml
##
@@ -42,4 +42,4 @@ before_install:
install: true
-script: travis_wait 30 mvn clean install -DskipTests -T 4 -q -Pitests
+script: mvn clean install -DskipTests -q -Pitests
Review comment:
Not related to this task. Please remove.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
##
@@ -310,7 +311,11 @@ protected void openInternal(final HiveConf conf,
Collection additionalFi
setupSessionAcls(tezConfig, conf);
-final TezClient session = TezClient.newBuilder("HIVE-" + sessionId,
tezConfig)
+String tezJobName = HiveConf.getVar(conf,ConfVars.HIVETEZJOBNAME);
+if (StringUtils.isBlank(tezJobName)){
+ tezJobName = "HIVE-" + sessionId;
Review comment:
What is the value of setting a hard coded name here?
I think it would be more useful if it would accept a Format String:
`
tezJobName = String.format("HIVE-%s", sessionId);
`
The default value of the configuration should be `HIVE-%s`
##
File path: .travis.yml
##
@@ -33,7 +33,7 @@ cache:
env:
MAVEN_SKIP_RC=true
- MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M"
Review comment:
Not related to this task. Please remove.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440833)
Remaining Estimate: 0h
Time Spent: 10m
> Support add a yarn application name for tez on hiveserver2
> --
>
> Key: HIVE-23026
> URL: https://issues.apache.org/jira/browse/HIVE-23026
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.3.5
>Reporter: Jake Xie
>Priority: Major
> Fix For: 2.3.5
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently tez on hiveServer2 cannot specify yarn application name, which is
> not very convenient for locating the problem SQL, so i added a configuration
> item to support setting tez job name
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=440819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440819 ] ASF GitHub Bot logged work on HIVE-23526: - Author: ASF GitHub Bot Created on: 03/Jun/20 15:00 Start Date: 03/Jun/20 15:00 Worklog Time Spent: 10m Work Description: belugabehr edited a comment on pull request #1029: URL: https://github.com/apache/hive/pull/1029#issuecomment-638240582 As I understand it, the code calling the JDBC Driver captures and handles the OOM error and then closes the JDBC Driver. When it closes, it throws another error " CloseOperation failed: out of sequence response". You would like for the close to be silent here. Is that correct? If so, there's not a ton that can be done here (other than perhaps wrap the OOM error in a Thrift Exception). In this particular case, Thrift client is reading the size of the string off the wire, then trying to create a buffer large enough to read the string. This is where it fails. When it fails here, it loses its place in the stream. The data is still in the stream, but the size is lost since that was read before the OOM error and there is no way to push the size back into the stream. At this point, Thrift client is hosed and it can't read anything else. When the JDBC Driver is closed, it sends a "close" request to HS2 to gracefully close the Hive session on the server. However, it is unable to read the response properly because, again, it lost its place and does not know how to find the next message. I'm not sure what the fix is here, but I don't think this is it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440819) Time Spent: 2h 50m (was: 2h 40m) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=440815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440815 ] ASF GitHub Bot logged work on HIVE-23526: - Author: ASF GitHub Bot Created on: 03/Jun/20 14:58 Start Date: 03/Jun/20 14:58 Worklog Time Spent: 10m Work Description: belugabehr edited a comment on pull request #1029: URL: https://github.com/apache/hive/pull/1029#issuecomment-638240582 As I understand it, the code calling the JDBC Driver captures and handles the OOM error and then closes the JDBC Driver. When it closes, it throws another error " CloseOperation failed: out of sequence response". You would like for the close to be silent here. Is that correct? If so, there's not a ton that can be done here (other than perhaps wrap the OOM error in a Thrift Exception). In this particular case, Thrift client is reading the size of the string off the wire, then trying to create a buffer large enough to read the string. This is where it fails. When it fails here, it loses its place in the stream. The data is still in the stream, but the size is lost since that was read before the OOM error and there is no way to push the size back into the stream. At this point, Thrift client is hosed and it can't read anything else. When the JDBC Driver is closed, it sends a "close" request to HS2 to gracefully shutdown the connection. However, it is unable to read the response properly because, again, it lost its place and does not know how to find the next message. I'm not sure what the fix is here, but I don't think this is it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440815) Time Spent: 2h 40m (was: 2.5h) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=440814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440814 ] ASF GitHub Bot logged work on HIVE-23526: - Author: ASF GitHub Bot Created on: 03/Jun/20 14:57 Start Date: 03/Jun/20 14:57 Worklog Time Spent: 10m Work Description: belugabehr edited a comment on pull request #1029: URL: https://github.com/apache/hive/pull/1029#issuecomment-638240582 As I understand it, the code calling the JDBC Driver captures and handles the OOM error and then closes the JDBC Driver. When it closes, it throws another error " CloseOperation failed: out of sequence response". You would like for the close to be silent here. Is that correct? If so, there's not a ton that can be done here (other than perhaps wrap the OOM error in a Thrift Exception). In this particular case, Thrift client is reading the size of the string off the wire, then trying to create a buffer large enough to read the string. This is where it fails. When it fails here, it loses its place in the stream. The data is still in the stream, but the size is lost since that was read before the OOM error. At this point, Thrift client is hosed and it can't read anything else. When the JDBC Driver is closed, it sends a "close" request to HS2 to gracefully shutdown the connection. However, it is unable to read the response properly because, again, it lost its place and does not know how to find the next message. I'm not sure what the fix is here, but I don't think this is it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440814) Time Spent: 2.5h (was: 2h 20m) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 2.5h > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=440813=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440813 ] ASF GitHub Bot logged work on HIVE-23526: - Author: ASF GitHub Bot Created on: 03/Jun/20 14:57 Start Date: 03/Jun/20 14:57 Worklog Time Spent: 10m Work Description: belugabehr edited a comment on pull request #1029: URL: https://github.com/apache/hive/pull/1029#issuecomment-638240582 As I understand it, the code calling the JDBC Driver captures and handles the OOM error and then closes the JDBC Driver. When it closes, it throws another error " CloseOperation failed: out of sequence response". You would like for the close to be silent here. Is that correct? If so, there's not a ton that can be done here (other than perhaps wrap the OOM error). In this particular case, Thrift client is reading the size of the string off the wire, then trying to create a buffer large enough to read the string. This is where it fails. When it fails here, it loses its place in the stream. The data is still in the stream, but the size is lost since that was read before the OOM error. At this point, Thrift client is hosed and it can't read anything else. When the JDBC Driver is closed, it sends a "close" request to HS2 to gracefully shutdown the connection. However, it is unable to read the response properly because, again, it lost its place and does not know how to find the next message. I'm not sure what the fix is here, but I don't think this is it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440813) Time Spent: 2h 20m (was: 2h 10m) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[
https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125025#comment-17125025
]
Hive QA commented on HIVE-23340:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 496 unchanged - 32 fixed = 496 total (was 528) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22730/dev-support/hive-personality.sh
|
| git revision | master / c886653 |
| Default Java | 1.8.0_111 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22730/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22730/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22730/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TxnHandler cleanup
> --
>
> Key: HIVE-23340
> URL: https://issues.apache.org/jira/browse/HIVE-23340
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch,
> HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch,
> HIVE-23340.6.patch, HIVE-23340.7.patch
>
>
> * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication
> * Remove TxnStatus character constants and use the enum values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23602) Use Java Concurrent Package for Operation Handle Set
[ https://issues.apache.org/jira/browse/HIVE-23602?focusedWorklogId=440812=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440812 ] ASF GitHub Bot logged work on HIVE-23602: - Author: ASF GitHub Bot Created on: 03/Jun/20 14:53 Start Date: 03/Jun/20 14:53 Worklog Time Spent: 10m Work Description: belugabehr opened a new pull request #1055: URL: https://github.com/apache/hive/pull/1055 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440812) Remaining Estimate: 0h Time Spent: 10m > Use Java Concurrent Package for Operation Handle Set > > > Key: HIVE-23602 > URL: https://issues.apache.org/jira/browse/HIVE-23602 > Project: Hive > Issue Type: Bug >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23602) Use Java Concurrent Package for Operation Handle Set
[ https://issues.apache.org/jira/browse/HIVE-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23602: -- Labels: pull-request-available (was: ) > Use Java Concurrent Package for Operation Handle Set > > > Key: HIVE-23602 > URL: https://issues.apache.org/jira/browse/HIVE-23602 > Project: Hive > Issue Type: Bug >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23602) Use Java Concurrent Package for Operation Handle Set
[ https://issues.apache.org/jira/browse/HIVE-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23602: -- Summary: Use Java Concurrent Package for Operation Handle Set (was: Use Java Concurrent Package in Operation Handle Set) > Use Java Concurrent Package for Operation Handle Set > > > Key: HIVE-23602 > URL: https://issues.apache.org/jira/browse/HIVE-23602 > Project: Hive > Issue Type: Bug >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23602) Use Java Concurrent Package in Operation Handle Set
[ https://issues.apache.org/jira/browse/HIVE-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23602: - > Use Java Concurrent Package in Operation Handle Set > --- > > Key: HIVE-23602 > URL: https://issues.apache.org/jira/browse/HIVE-23602 > Project: Hive > Issue Type: Bug >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=440803=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440803 ] ASF GitHub Bot logged work on HIVE-23526: - Author: ASF GitHub Bot Created on: 03/Jun/20 14:37 Start Date: 03/Jun/20 14:37 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #1029: URL: https://github.com/apache/hive/pull/1029#issuecomment-638240582 As I understand it, the code calling the JDBC Driver captures and handles the OOM error and then closes the JDBC Driver. When it closes, it throws another error " CloseOperation failed: out of sequence response". You would like for the close to be silent here. Is that correct? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440803) Time Spent: 2h 10m (was: 2h) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23601) Hive Statement Clear Statement Handle on Error
[
https://issues.apache.org/jira/browse/HIVE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-23601:
--
Labels: pull-request-available (was: )
> Hive Statement Clear Statement Handle on Error
> --
>
> Key: HIVE-23601
> URL: https://issues.apache.org/jira/browse/HIVE-23601
> Project: Hive
> Issue Type: Bug
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> private void closeStatementIfNeeded() throws SQLException {
> try {
> if (stmtHandle != null) {
> TCloseOperationReq closeReq = new TCloseOperationReq(stmtHandle);
> TCloseOperationResp closeResp = client.CloseOperation(closeReq);
> Utils.verifySuccessWithInfo(closeResp.getStatus());
> stmtHandle = null;
> }
> } catch (SQLException e) {
> throw e;
> } catch (Exception e) {
> throw new SQLException("Failed to close statement", "08S01", e);
> }
> }
> void closeClientOperation() throws SQLException {
> closeStatementIfNeeded();
> isQueryClosed = true;
> stmtHandle = null;
> }
> {code}
> {{verifySuccessWithInfo}} throws an {{Exception}} if it finds an error code
> and therefore leapfrogs over setting the statement handle to null (twice).
> Probably not what is intended since the original author(s) are tried twice to
> null it out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23601) Hive Statement Clear Statement Handle on Error
[
https://issues.apache.org/jira/browse/HIVE-23601?focusedWorklogId=440802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440802
]
ASF GitHub Bot logged work on HIVE-23601:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 14:36
Start Date: 03/Jun/20 14:36
Worklog Time Spent: 10m
Work Description: belugabehr opened a new pull request #1054:
URL: https://github.com/apache/hive/pull/1054
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440802)
Remaining Estimate: 0h
Time Spent: 10m
> Hive Statement Clear Statement Handle on Error
> --
>
> Key: HIVE-23601
> URL: https://issues.apache.org/jira/browse/HIVE-23601
> Project: Hive
> Issue Type: Bug
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> private void closeStatementIfNeeded() throws SQLException {
> try {
> if (stmtHandle != null) {
> TCloseOperationReq closeReq = new TCloseOperationReq(stmtHandle);
> TCloseOperationResp closeResp = client.CloseOperation(closeReq);
> Utils.verifySuccessWithInfo(closeResp.getStatus());
> stmtHandle = null;
> }
> } catch (SQLException e) {
> throw e;
> } catch (Exception e) {
> throw new SQLException("Failed to close statement", "08S01", e);
> }
> }
> void closeClientOperation() throws SQLException {
> closeStatementIfNeeded();
> isQueryClosed = true;
> stmtHandle = null;
> }
> {code}
> {{verifySuccessWithInfo}} throws an {{Exception}} if it finds an error code
> and therefore leapfrogs over setting the statement handle to null (twice).
> Probably not what is intended since the original author(s) are tried twice to
> null it out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20890) ACID: Allow whole table ReadLocks to skip all partition locks
[
https://issues.apache.org/jira/browse/HIVE-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125006#comment-17125006
]
Hive QA commented on HIVE-20890:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004704/HIVE-20890.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 17094 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_stats3]
(batchId=43)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_stats4]
(batchId=92)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_table_stats]
(batchId=84)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explain_locks]
(batchId=74)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=94)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=273)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather
(batchId=274)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMMNoAutogather.testForParallelBootstrapLoad
(batchId=182)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMMNoAutogather.testMetadataOnlyDump
(batchId=182)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMMNoAutogather.testNonParallelBootstrapLoad
(batchId=182)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMMNoAutogather.testRetryFailure
(batchId=182)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testAllPartitions
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testNoStatsUpdateForSimpleReplTable
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testNoStatsUpdateForTxnReplTable
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testParallelOps
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testPartitionSubset
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testPartitionsWithDifferentColsAll
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testPartitionsWithDifferentColsExistingOnly
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testQueueingWithThreads
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testTxnDynamicPartitions
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testTxnPartitions
(batchId=257)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl
(batchId=178)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation (batchId=219)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22729/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22729/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22729/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004704 - PreCommit-HIVE-Build
> ACID: Allow whole table ReadLocks to skip all partition locks
> -
>
> Key: HIVE-20890
> URL: https://issues.apache.org/jira/browse/HIVE-20890
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20890.1.patch
>
>
> HIVE-19369 proposes adding a EXCL_WRITE lock which does not wait for any
> SHARED_READ locks for read operations - in the presence of that lock, the
> insert overwrite no longer takes an exclusive lock.
> The only exclusive operation will be a schema change or drop table, which
> should take an exclusive lock on the entire table directly.
> {code}
> explain locks select * from tpcds_bin_partitioned_orc_1000.store_sales where
> ss_sold_date_sk=2452626
> ++
> | Explain |
> ++
> | LOCK INFORMATION: |
> | tpcds_bin_partitioned_orc_1000.store_sales -> SHARED_READ |
> | tpcds_bin_partitioned_orc_1000.store_sales.ss_sold_date_sk=2452626 ->
> SHARED_READ |
>
[jira] [Updated] (HIVE-23601) Hive Statement Clear Statement Handle on Error
[
https://issues.apache.org/jira/browse/HIVE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23601:
--
Summary: Hive Statement Clear Statement Handle on Error (was: Hive
Statement Does Not Clear Statement Handle on Error)
> Hive Statement Clear Statement Handle on Error
> --
>
> Key: HIVE-23601
> URL: https://issues.apache.org/jira/browse/HIVE-23601
> Project: Hive
> Issue Type: Bug
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> {code:java}
> private void closeStatementIfNeeded() throws SQLException {
> try {
> if (stmtHandle != null) {
> TCloseOperationReq closeReq = new TCloseOperationReq(stmtHandle);
> TCloseOperationResp closeResp = client.CloseOperation(closeReq);
> Utils.verifySuccessWithInfo(closeResp.getStatus());
> stmtHandle = null;
> }
> } catch (SQLException e) {
> throw e;
> } catch (Exception e) {
> throw new SQLException("Failed to close statement", "08S01", e);
> }
> }
> void closeClientOperation() throws SQLException {
> closeStatementIfNeeded();
> isQueryClosed = true;
> stmtHandle = null;
> }
> {code}
> {{verifySuccessWithInfo}} throws an {{Exception}} if it finds an error code
> and therefore leapfrogs over setting the statement handle to null (twice).
> Probably not what is intended since the original author(s) are tried twice to
> null it out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23601) Hive Statement Does Not Clear Statement Handle on Error
[
https://issues.apache.org/jira/browse/HIVE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-23601:
-
> Hive Statement Does Not Clear Statement Handle on Error
> ---
>
> Key: HIVE-23601
> URL: https://issues.apache.org/jira/browse/HIVE-23601
> Project: Hive
> Issue Type: Bug
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> {code:java}
> private void closeStatementIfNeeded() throws SQLException {
> try {
> if (stmtHandle != null) {
> TCloseOperationReq closeReq = new TCloseOperationReq(stmtHandle);
> TCloseOperationResp closeResp = client.CloseOperation(closeReq);
> Utils.verifySuccessWithInfo(closeResp.getStatus());
> stmtHandle = null;
> }
> } catch (SQLException e) {
> throw e;
> } catch (Exception e) {
> throw new SQLException("Failed to close statement", "08S01", e);
> }
> }
> void closeClientOperation() throws SQLException {
> closeStatementIfNeeded();
> isQueryClosed = true;
> stmtHandle = null;
> }
> {code}
> {{verifySuccessWithInfo}} throws an {{Exception}} if it finds an error code
> and therefore leapfrogs over setting the statement handle to null (twice).
> Probably not what is intended since the original author(s) are tried twice to
> null it out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20890) ACID: Allow whole table ReadLocks to skip all partition locks
[
https://issues.apache.org/jira/browse/HIVE-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124983#comment-17124983
]
Hive QA commented on HIVE-20890:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1523 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
51s{color} | {color:red} ql: The patch generated 11 new + 219 unchanged - 5
fixed = 230 total (was 224) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22729/dev-support/hive-personality.sh
|
| git revision | master / c886653 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22729/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22729/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Allow whole table ReadLocks to skip all partition locks
> -
>
> Key: HIVE-20890
> URL: https://issues.apache.org/jira/browse/HIVE-20890
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20890.1.patch
>
>
> HIVE-19369 proposes adding a EXCL_WRITE lock which does not wait for any
> SHARED_READ locks for read operations - in the presence of that lock, the
> insert overwrite no longer takes an exclusive lock.
> The only exclusive operation will be a schema change or drop table, which
> should take an exclusive lock on the entire table directly.
> {code}
> explain locks select * from tpcds_bin_partitioned_orc_1000.store_sales where
> ss_sold_date_sk=2452626
> ++
> | Explain |
>
[jira] [Commented] (HIVE-23598) Add option to rewrite NTILE to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124966#comment-17124966
]
Zoltan Haindrich commented on HIVE-23598:
-
I think this could be rewritten to be use 'cume_dist' and let the other rewrite
do the heavy lifting...
{code}
select id,ntile(4) over (order by id),CEIL(cume_dist() over (order by id)*4)
from sketch_input
order by id;
{code}
the NTILE udf has an easy advantage: it may return different values for the
same input value.
{code}
select id,ntile(2) over (order by id) from TABLE(VALUES(1),(1),(1),(1)) AS
t(id);
+-+-+
| id | ntile_window_0 |
+-+-+
| 1 | 1 |
| 1 | 1 |
| 1 | 2 |
| 1 | 2 |
+-+-+
{code}
> Add option to rewrite NTILE to sketch functions
> ---
>
> Key: HIVE-23598
> URL: https://issues.apache.org/jira/browse/HIVE-23598
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23277) HiveProtoLogger should carry out JSON conversion in its own thread
[
https://issues.apache.org/jira/browse/HIVE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124953#comment-17124953
]
Attila Magyar edited comment on HIVE-23277 at 6/3/20, 1:31 PM:
---
Hey [~rajesh.balamohan], I made a patch for this, where json serialization
happens on the logWriter's thread. The event is only built partially up front
with a json object (not the serialized string) and the conversion happens right
before writing out the event.
However events like this take up more space in memory as before. About twice as
much. The queue has a default max capacity of 64 so this might not be a problem.
{code:java}
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.838Z
hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local
hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060
class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log
Writer 0"] XXX size with serialized JSON: 392288
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.833Z
hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local
hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060
class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log
Writer 0"] XXX with JSON object: 779536{code}
How significant do you think the speed improvements is? Is it worth it? Based
on my own measurements the JSON serialization wasn't that slow with the queries
I used (about 10-15 ms).
was (Author: amagyar):
Hey [~rajesh.balamohan], I made a patch for this, where json serialization
happens on the logWriter's thread. The event is only built partially up front
with a json object (not the serialized string) and the conversion happens right
before writing out the event.
However events like this takes up more space in memory as before. About twice
as much. The queue has a default max capacity of 64 so this might not be a
problem.
{code:java}
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.838Z
hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local
hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060
class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log
Writer 0"] XXX size with serialized JSON: 392288
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.833Z
hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local
hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060
class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log
Writer 0"] XXX with JSON object: 779536{code}
> HiveProtoLogger should carry out JSON conversion in its own thread
> --
>
> Key: HIVE-23277
> URL: https://issues.apache.org/jira/browse/HIVE-23277
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Minor
> Attachments: HIVE-23277.1.patch, Screenshot 2020-04-23 at 11.27.42
> AM.png
>
>
> !Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124954#comment-17124954 ] David Mollitor commented on HIVE-23526: --- As I understand it, the code calling the JDBC Driver captures and handles the OOM error and then closes the JDBC Driver. When it closes, it throws another error " CloseOperation failed: out of sequence response". You would like for the close to be silent here. Is that correct? > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 2h > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23277) HiveProtoLogger should carry out JSON conversion in its own thread
[
https://issues.apache.org/jira/browse/HIVE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124953#comment-17124953
]
Attila Magyar commented on HIVE-23277:
--
Hey [~rajesh.balamohan], I made a patch for this, where json serialization
happens on the logWriter's thread. The event is only built partially up front
with a json object (not the serialized string) and the conversion happens right
before writing out the event.
However events like this takes up more space in memory as before. About twice
as much. The queue has a default max capacity of 64 so this might not be a
problem.
{code:java}
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.838Z
hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local
hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060
class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log
Writer 0"] XXX size with serialized JSON: 392288
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.833Z
hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local
hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060
class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log
Writer 0"] XXX with JSON object: 779536{code}
> HiveProtoLogger should carry out JSON conversion in its own thread
> --
>
> Key: HIVE-23277
> URL: https://issues.apache.org/jira/browse/HIVE-23277
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Minor
> Attachments: HIVE-23277.1.patch, Screenshot 2020-04-23 at 11.27.42
> AM.png
>
>
> !Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
