[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019904#comment-17019904 ] Rajesh Balamohan commented on HIVE-22753: - Yeah, that could end up closing operationlog way too soon. Will check and revise the patch. > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019885#comment-17019885 ] mahesh kumar behera edited comment on HIVE-22753 at 1/21/20 6:38 AM: - We should not clean up the operation log before close operation. User may try to query operation log using the operation handle. Anyways for failed operation, operation close is called and that should clear up the operation log. Do you think this could be the reason for the leak ? https://issues.apache.org/jira/browse/HIVE-22733 was (Author: maheshk114): We should not clean up the operation log before close operation. User may try to query operation log using the operation handle. Anyways for failed operation, operation close is called and that should clear up the operation log. > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019885#comment-17019885 ] mahesh kumar behera commented on HIVE-22753: We should not clean up the operation log before close operation. User may try to query operation log using the operation handle. Anyways for failed operation, operation close is called and that should clear up the operation log. > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22601) Some columns will be lost when a UDTF has multiple aliases in some cases
[
https://issues.apache.org/jira/browse/HIVE-22601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019884#comment-17019884
]
Makoto Yui commented on HIVE-22601:
---
{code:java}
select
posexplode(split('1,4', ',')) as (seq, col)
union all
select
posexplode(split('2,6', ',')) as (seq, col){code}
produces a wrong result as well.
_u1.col
1
4
2
6
> Some columns will be lost when a UDTF has multiple aliases in some cases
>
>
> Key: HIVE-22601
> URL: https://issues.apache.org/jira/browse/HIVE-22601
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.1, 2.2.0, 3.1.2, 2.3.6
>Reporter: okumin
>Assignee: Owen O'Malley
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22601.1.patch, HIVE-22601.2.patch, HIVE-22601.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Only one column will be retained when putting UDTFs with multiple aliases and
> a top-level UNION together.
> For example, the result of the following SQL should have three columns, c1,
> c2 and c3.
> {code:java}
> SELECT stack(1, 'a', 'b', 'c') AS (c1, c2, c3)
> UNION ALL
> SELECT stack(1, 'd', 'e', 'f') AS (c1, c2, c3);
> {code}
> However, It's only the c3 column which I can get.
> {code:java}
> +-+
> | _u1.c3 |
> +-+
> | c |
> | f |
> +-+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-22753: Assignee: Rajesh Balamohan Status: Patch Available (was: Open) > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-22753: Attachment: HIVE-22753.1.patch > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-22753: Description: In case of exception in SQLOperation, operational log does not get cleared up. This causes gradual build up of HushableRandomAccessFileAppender causing HS2 to OOM after some time. !image-2020-01-21-11-14-37-911.png|width=431,height=267! !2Q==! Prod instance mem !image-2020-01-21-11-17-59-279.png|width=698,height=209! Each HushableRandomAccessFileAppender holds internal ref to RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem leak. Related ticket: HIVE-18820 was: In case of exception in SQLOperation, operational log does not get cleared up. This causes gradual build up of HushableRandomAccessFileAppender causing HS2 to OOM after some time. !image-2020-01-21-11-14-37-911.png|width=431,height=267! !2Q==|width=487,height=204! Prod instance mem !Z|width=531,height=159! Each HushableRandomAccessFileAppender holds internal ref to RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem leak. Related ticket: HIVE-18820 > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > !2Q==! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-22753: Attachment: image-2020-01-21-11-17-59-279.png > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > !2Q==|width=487,height=204! > > Prod instance mem > !Z|width=531,height=159! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-22753: Attachment: image-2020-01-21-11-18-37-294.png > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > !2Q==! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-22753: Description: In case of exception in SQLOperation, operational log does not get cleared up. This causes gradual build up of HushableRandomAccessFileAppender causing HS2 to OOM after some time. !image-2020-01-21-11-14-37-911.png|width=431,height=267! Allocation tree !image-2020-01-21-11-18-37-294.png|width=425,height=178! Prod instance mem !image-2020-01-21-11-17-59-279.png|width=698,height=209! Each HushableRandomAccessFileAppender holds internal ref to RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem leak. Related ticket: HIVE-18820 was: In case of exception in SQLOperation, operational log does not get cleared up. This causes gradual build up of HushableRandomAccessFileAppender causing HS2 to OOM after some time. !image-2020-01-21-11-14-37-911.png|width=431,height=267! !2Q==! Prod instance mem !image-2020-01-21-11-17-59-279.png|width=698,height=209! Each HushableRandomAccessFileAppender holds internal ref to RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem leak. Related ticket: HIVE-18820 > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22685) Fix TestHiveSqlDateTimeFormatter To Work With New Year 2020
[
https://issues.apache.org/jira/browse/HIVE-22685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22685:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix TestHiveSqlDateTimeFormatter To Work With New Year 2020
> ---
>
> Key: HIVE-22685
> URL: https://issues.apache.org/jira/browse/HIVE-22685
> Project: Hive
> Issue Type: Bug
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22685.1.patch, HIVE-22685.2.patch,
> HIVE-22685.3.patch
>
>
> Unit test is now broken (n)(n):(
> {code:java}
> //Tests for these patterns would need changing every decade if done in
> the above way.
> //Thursday of the first week in an ISO year always matches the Gregorian
> year.
> checkParseTimestampIso("IY-IW-ID", "0-01-04", "iw, ", "01, " +
> thisYearString.substring(0, 3) + "0");
> checkParseTimestampIso("I-IW-ID", "0-01-04", "iw, ", "01, " +
> thisYearString.substring(0, 3) + "0");
> {code}
> {code}
> org.junit.ComparisonFailure: expected:<01, 20[1]0> but was:<01, 20[2]0>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hive.common.format.datetime.TestHiveSqlDateTimeFormatter.checkParseTimestampIso(TestHiveSqlDateTimeFormatter.java:313)
> at
> org.apache.hadoop.hive.common.format.datetime.TestHiveSqlDateTimeFormatter.testParseTimestamp(TestHiveSqlDateTimeFormatter.java:287)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22685) Fix TestHiveSqlDateTimeFormatter To Work With New Year 2020
[
https://issues.apache.org/jira/browse/HIVE-22685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019833#comment-17019833
]
David Mollitor commented on HIVE-22685:
---
[~klcopp] [~kuczoram] Thank you for all your help.
Pushed to master.
> Fix TestHiveSqlDateTimeFormatter To Work With New Year 2020
> ---
>
> Key: HIVE-22685
> URL: https://issues.apache.org/jira/browse/HIVE-22685
> Project: Hive
> Issue Type: Bug
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22685.1.patch, HIVE-22685.2.patch,
> HIVE-22685.3.patch
>
>
> Unit test is now broken (n)(n):(
> {code:java}
> //Tests for these patterns would need changing every decade if done in
> the above way.
> //Thursday of the first week in an ISO year always matches the Gregorian
> year.
> checkParseTimestampIso("IY-IW-ID", "0-01-04", "iw, ", "01, " +
> thisYearString.substring(0, 3) + "0");
> checkParseTimestampIso("I-IW-ID", "0-01-04", "iw, ", "01, " +
> thisYearString.substring(0, 3) + "0");
> {code}
> {code}
> org.junit.ComparisonFailure: expected:<01, 20[1]0> but was:<01, 20[2]0>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hive.common.format.datetime.TestHiveSqlDateTimeFormatter.checkParseTimestampIso(TestHiveSqlDateTimeFormatter.java:313)
> at
> org.apache.hadoop.hive.common.format.datetime.TestHiveSqlDateTimeFormatter.testParseTimestamp(TestHiveSqlDateTimeFormatter.java:287)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22685) Fix TestHiveSqlDateTimeFormatter To Work With New Year 2020
[
https://issues.apache.org/jira/browse/HIVE-22685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22685:
--
Summary: Fix TestHiveSqlDateTimeFormatter To Work With New Year 2020 (was:
TestHiveSqlDateTimeFormatter Now Broken with New Year 2020)
> Fix TestHiveSqlDateTimeFormatter To Work With New Year 2020
> ---
>
> Key: HIVE-22685
> URL: https://issues.apache.org/jira/browse/HIVE-22685
> Project: Hive
> Issue Type: Bug
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22685.1.patch, HIVE-22685.2.patch,
> HIVE-22685.3.patch
>
>
> Unit test is now broken (n)(n):(
> {code:java}
> //Tests for these patterns would need changing every decade if done in
> the above way.
> //Thursday of the first week in an ISO year always matches the Gregorian
> year.
> checkParseTimestampIso("IY-IW-ID", "0-01-04", "iw, ", "01, " +
> thisYearString.substring(0, 3) + "0");
> checkParseTimestampIso("I-IW-ID", "0-01-04", "iw, ", "01, " +
> thisYearString.substring(0, 3) + "0");
> {code}
> {code}
> org.junit.ComparisonFailure: expected:<01, 20[1]0> but was:<01, 20[2]0>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hive.common.format.datetime.TestHiveSqlDateTimeFormatter.checkParseTimestampIso(TestHiveSqlDateTimeFormatter.java:313)
> at
> org.apache.hadoop.hive.common.format.datetime.TestHiveSqlDateTimeFormatter.testParseTimestamp(TestHiveSqlDateTimeFormatter.java:287)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22752) HiveMetastore addWriteNotificationLog should be invoked only when listeners are enabled
[
https://issues.apache.org/jira/browse/HIVE-22752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-22752:
Description:
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L8109]
Even though listeners are turned off, it gets executed and causes load on the
system. This should be guarded by listener checks.
{noformat}
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1589)
at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1700)
at
org.apache.hadoop.hive.ql.metadata.Hive.addInsertFileInformation(Hive.java:3185)
at
org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3138)
at
org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3123)
at
org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2238){noformat}
was:
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L8109]
Even though listeners are turned off, it gets executed and causes load on the
system. This should be guarded by listener checks.
> HiveMetastore addWriteNotificationLog should be invoked only when listeners
> are enabled
> ---
>
> Key: HIVE-22752
> URL: https://issues.apache.org/jira/browse/HIVE-22752
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
>
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L8109]
>
> Even though listeners are turned off, it gets executed and causes load on the
> system. This should be guarded by listener checks.
>
> {noformat}
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1589)
> at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1700)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.addInsertFileInformation(Hive.java:3185)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3138)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3123)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2238){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22751) Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper
[
https://issues.apache.org/jira/browse/HIVE-22751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019829#comment-17019829
]
Anishek Agarwal commented on HIVE-22751:
+1 pending tests.
> Move locking in HiveServer2::isDeregisteredWithZooKeeper to
> ZooKeeperHiveHelper
> ---
>
> Key: HIVE-22751
> URL: https://issues.apache.org/jira/browse/HIVE-22751
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22751.1.patch
>
>
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/server/HiveServer2.java#L620]
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/SessionManager.java#L597]
>
> When queries are run in beeline and closed, it causes unwanted delays in
> shutting down beeline. Here is the threaddump from server side, which shows
> HiveServer2 lock contention.
>
> It would be good to move synchronization to
> "zooKeeperHelper.isDeregisteredWithZooKeeper"
>
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x7f78b0078800 nid=0x2d1c waiting on
> condition [0x7f78b968c000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xac8d5ff0> (a
> java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:187)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123)
> - locked <0xa9c5f2a8> (a java.lang.Object)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115)
> at
> org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:790)
> at
> org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:763)
> at
> org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:687)
> - locked <0xa99bd568> (a
> org.apache.hive.service.server.HiveServer2)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016)
> at
> org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294)
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> "HiveServer2-HttpHandler-Pool: Thread-50" #50 prio=5 os_prio=0
> tid=0x7f78b3e60800 nid=0x2fa7 waiting for monitor entry
> [0x7f7884edf000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hive.service.server.HiveServer2.isDeregisteredWithZooKeeper(HiveServer2.java:600)
> - waiting to lock <0xa99bd568> (a
> org.apache.hive.service.server.HiveServer2)
> at
> org.apache.hive.service.cli.session.SessionManager.closeSessionInternal(SessionManager.java:631)
> at
> org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:621)
> - locked <0xaa1970b0> (a
> org.apache.hive.service.cli.session.SessionManager)
> at
> org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:244)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.CloseSession(ThriftCLIService.java:527)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1517)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1502)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:237)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>
[jira] [Updated] (HIVE-22751) Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper
[
https://issues.apache.org/jira/browse/HIVE-22751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-22751:
Assignee: Rajesh Balamohan
Status: Patch Available (was: Open)
> Move locking in HiveServer2::isDeregisteredWithZooKeeper to
> ZooKeeperHiveHelper
> ---
>
> Key: HIVE-22751
> URL: https://issues.apache.org/jira/browse/HIVE-22751
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22751.1.patch
>
>
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/server/HiveServer2.java#L620]
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/SessionManager.java#L597]
>
> When queries are run in beeline and closed, it causes unwanted delays in
> shutting down beeline. Here is the threaddump from server side, which shows
> HiveServer2 lock contention.
>
> It would be good to move synchronization to
> "zooKeeperHelper.isDeregisteredWithZooKeeper"
>
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x7f78b0078800 nid=0x2d1c waiting on
> condition [0x7f78b968c000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xac8d5ff0> (a
> java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:187)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123)
> - locked <0xa9c5f2a8> (a java.lang.Object)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115)
> at
> org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:790)
> at
> org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:763)
> at
> org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:687)
> - locked <0xa99bd568> (a
> org.apache.hive.service.server.HiveServer2)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016)
> at
> org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294)
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> "HiveServer2-HttpHandler-Pool: Thread-50" #50 prio=5 os_prio=0
> tid=0x7f78b3e60800 nid=0x2fa7 waiting for monitor entry
> [0x7f7884edf000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hive.service.server.HiveServer2.isDeregisteredWithZooKeeper(HiveServer2.java:600)
> - waiting to lock <0xa99bd568> (a
> org.apache.hive.service.server.HiveServer2)
> at
> org.apache.hive.service.cli.session.SessionManager.closeSessionInternal(SessionManager.java:631)
> at
> org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:621)
> - locked <0xaa1970b0> (a
> org.apache.hive.service.cli.session.SessionManager)
> at
> org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:244)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.CloseSession(ThriftCLIService.java:527)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1517)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1502)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:237)
> at
[jira] [Updated] (HIVE-22751) Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper
[
https://issues.apache.org/jira/browse/HIVE-22751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-22751:
Attachment: HIVE-22751.1.patch
> Move locking in HiveServer2::isDeregisteredWithZooKeeper to
> ZooKeeperHiveHelper
> ---
>
> Key: HIVE-22751
> URL: https://issues.apache.org/jira/browse/HIVE-22751
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22751.1.patch
>
>
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/server/HiveServer2.java#L620]
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/SessionManager.java#L597]
>
> When queries are run in beeline and closed, it causes unwanted delays in
> shutting down beeline. Here is the threaddump from server side, which shows
> HiveServer2 lock contention.
>
> It would be good to move synchronization to
> "zooKeeperHelper.isDeregisteredWithZooKeeper"
>
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x7f78b0078800 nid=0x2d1c waiting on
> condition [0x7f78b968c000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xac8d5ff0> (a
> java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:187)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123)
> - locked <0xa9c5f2a8> (a java.lang.Object)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115)
> at
> org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:790)
> at
> org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:763)
> at
> org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:687)
> - locked <0xa99bd568> (a
> org.apache.hive.service.server.HiveServer2)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016)
> at
> org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294)
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> "HiveServer2-HttpHandler-Pool: Thread-50" #50 prio=5 os_prio=0
> tid=0x7f78b3e60800 nid=0x2fa7 waiting for monitor entry
> [0x7f7884edf000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hive.service.server.HiveServer2.isDeregisteredWithZooKeeper(HiveServer2.java:600)
> - waiting to lock <0xa99bd568> (a
> org.apache.hive.service.server.HiveServer2)
> at
> org.apache.hive.service.cli.session.SessionManager.closeSessionInternal(SessionManager.java:631)
> at
> org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:621)
> - locked <0xaa1970b0> (a
> org.apache.hive.service.cli.session.SessionManager)
> at
> org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:244)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.CloseSession(ThriftCLIService.java:527)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1517)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1502)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:237)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at
[jira] [Updated] (HIVE-22731) Probe MapJoin hashtables for row level filtering
[ https://issues.apache.org/jira/browse/HIVE-22731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22731: -- Issue Type: Improvement (was: Bug) > Probe MapJoin hashtables for row level filtering > > > Key: HIVE-22731 > URL: https://issues.apache.org/jira/browse/HIVE-22731 > Project: Hive > Issue Type: Improvement > Components: Hive, llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Attachments: HIVE-22731.1.patch, HIVE-22731.WIP.patch, > decode_time_bars.pdf > > > Currently, RecordReaders such as ORC support filtering at coarser-grained > levels, namely: File, Stripe (64 to 256mb), and Row group (10k row) level. > They only filter sets of rows if they can guarantee that none of the rows can > pass a filter (usually given as searchable argument). > However, a significant amount of time can be spend decoding rows with > multiple columns that are not even used in the final result. See figure where > original is what happens today and in LazyDecode we skip decoding rows that > do not match the key. > To enable a more fine-grained filtering in the particular case of a MapJoin > we could utilize the key HashTable created from the smaller table to skip > deserializing row columns at the larger table that do not match any key and > thus save CPU time. > This Jira investigates this direction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22731) Probe MapJoin hashtables for row level filtering
[ https://issues.apache.org/jira/browse/HIVE-22731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22731: -- Attachment: HIVE-22731.1.patch > Probe MapJoin hashtables for row level filtering > > > Key: HIVE-22731 > URL: https://issues.apache.org/jira/browse/HIVE-22731 > Project: Hive > Issue Type: Bug > Components: Hive, llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Attachments: HIVE-22731.1.patch, HIVE-22731.WIP.patch, > decode_time_bars.pdf > > > Currently, RecordReaders such as ORC support filtering at coarser-grained > levels, namely: File, Stripe (64 to 256mb), and Row group (10k row) level. > They only filter sets of rows if they can guarantee that none of the rows can > pass a filter (usually given as searchable argument). > However, a significant amount of time can be spend decoding rows with > multiple columns that are not even used in the final result. See figure where > original is what happens today and in LazyDecode we skip decoding rows that > do not match the key. > To enable a more fine-grained filtering in the particular case of a MapJoin > we could utilize the key HashTable created from the smaller table to skip > deserializing row columns at the larger table that do not match any key and > thus save CPU time. > This Jira investigates this direction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22731) Probe MapJoin hashtables for row level filtering
[ https://issues.apache.org/jira/browse/HIVE-22731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22731: -- Status: Patch Available (was: In Progress) > Probe MapJoin hashtables for row level filtering > > > Key: HIVE-22731 > URL: https://issues.apache.org/jira/browse/HIVE-22731 > Project: Hive > Issue Type: Bug > Components: Hive, llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Attachments: HIVE-22731.1.patch, HIVE-22731.WIP.patch, > decode_time_bars.pdf > > > Currently, RecordReaders such as ORC support filtering at coarser-grained > levels, namely: File, Stripe (64 to 256mb), and Row group (10k row) level. > They only filter sets of rows if they can guarantee that none of the rows can > pass a filter (usually given as searchable argument). > However, a significant amount of time can be spend decoding rows with > multiple columns that are not even used in the final result. See figure where > original is what happens today and in LazyDecode we skip decoding rows that > do not match the key. > To enable a more fine-grained filtering in the particular case of a MapJoin > we could utilize the key HashTable created from the smaller table to skip > deserializing row columns at the larger table that do not match any key and > thus save CPU time. > This Jira investigates this direction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019687#comment-17019687
]
Hive QA commented on HIVE-22729:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991385/HIVE-22729.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17933 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=90)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20253/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20253/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20253/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991385 - PreCommit-HIVE-Build
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22712) ReExec Driver execute submit the query in default queue irrespective of user defined queue
[ https://issues.apache.org/jira/browse/HIVE-22712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22712: -- Attachment: HIVE-22712.06.patch Status: Patch Available (was: Open) > ReExec Driver execute submit the query in default queue irrespective of user > defined queue > -- > > Key: HIVE-22712 > URL: https://issues.apache.org/jira/browse/HIVE-22712 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2 >Affects Versions: 3.1.2 > Environment: Hive-3 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22712.01.patch, HIVE-22712.02.patch, > HIVE-22712.03.patch, HIVE-22712.04.patch, HIVE-22712.05.patch, > HIVE-22712.06.patch, HIVE-22712.06.patch, HIVE-22712.patch > > > we unset the queue name intentionally in > TezSessionState#startSessionAndContainers, > as a result reexec create a new session in the default queue and create a > problem, its a cumbersome to add reexec.overlay.tez.queue.name at session > level. > I could not find a better way of setting the queue name (I am open for the > suggestion here) since it can create a conflict with the Global queue name > vs user-defined queue that's why setting while initialization of > ReExecutionOverlayPlugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22712) ReExec Driver execute submit the query in default queue irrespective of user defined queue
[ https://issues.apache.org/jira/browse/HIVE-22712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22712: -- Status: Open (was: Patch Available) > ReExec Driver execute submit the query in default queue irrespective of user > defined queue > -- > > Key: HIVE-22712 > URL: https://issues.apache.org/jira/browse/HIVE-22712 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2 >Affects Versions: 3.1.2 > Environment: Hive-3 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22712.01.patch, HIVE-22712.02.patch, > HIVE-22712.03.patch, HIVE-22712.04.patch, HIVE-22712.05.patch, > HIVE-22712.06.patch, HIVE-22712.patch > > > we unset the queue name intentionally in > TezSessionState#startSessionAndContainers, > as a result reexec create a new session in the default queue and create a > problem, its a cumbersome to add reexec.overlay.tez.queue.name at session > level. > I could not find a better way of setting the queue name (I am open for the > suggestion here) since it can create a conflict with the Global queue name > vs user-defined queue that's why setting while initialization of > ReExecutionOverlayPlugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019673#comment-17019673
]
Hive QA commented on HIVE-22729:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
37s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 8 new + 677 unchanged - 6 fixed = 685 total (was 683) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 50 unchanged - 0 fixed
= 51 total (was 50) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
36s{color} | {color:red} ql generated 3 new + 1531 unchanged - 1 fixed = 1534
total (was 1532) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | ci could be null and is guaranteed to be dereferenced in
org.apache.hadoop.hive.ql.txn.compactor.Worker.run() Method invoked at
Worker.java:is guaranteed to be dereferenced in
org.apache.hadoop.hive.ql.txn.compactor.Worker.run() Method invoked at
Worker.java:[line 118] |
| | Possible null pointer dereference of RemoteCompactorThread.msc in
org.apache.hadoop.hive.ql.txn.compactor.Worker.run() on exception path
Dereferenced at Worker.java:RemoteCompactorThread.msc in
org.apache.hadoop.hive.ql.txn.compactor.Worker.run() on exception path
Dereferenced at Worker.java:[line 247] |
| | Possible null pointer dereference of ci in
org.apache.hadoop.hive.ql.txn.compactor.Worker.run() on exception path
Dereferenced at Worker.java:ci in
org.apache.hadoop.hive.ql.txn.compactor.Worker.run() on exception path
Dereferenced at Worker.java:[line 246] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20253/dev-support/hive-personality.sh
|
| git revision | master / 3c0705e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[
https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019669#comment-17019669
]
Csaba Ringhofer commented on HIVE-21931:
Thanks, I am ok with HIVE-22554 as solution. I am closing this now, will
reopen if it doesn't work for me.
> Slow compaction for tiny tables
> ---
>
> Key: HIVE-21931
> URL: https://issues.apache.org/jira/browse/HIVE-21931
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Csaba Ringhofer
>Priority: Major
> Labels: compaction
>
> I observed the issue in Impala development environment when (major)
> compacting insert_only transactional tables in Hive. The compaction could
> take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The
> actual work was done much earlier, the new base file was correctly written to
> HDFS, and Hive seemed to wait without doing any work.
> The compactions are started manually, hive.compactor.initiator.on=false to
> avoid "surprise compaction" during tests.
> {code}
> hive.compactor.abortedtxn.threshold=1000
> hive.compactor.check.interval=300s
> hive.compactor.cleaner.run.interval=5000ms
> hive.compactor.compact.insert.only=true
> hive.compactor.crud.query.based=false
> hive.compactor.delta.num.threshold=10
> hive.compactor.delta.pct.threshold=0.1
> hive.compactor.history.reaper.interval=2m
> hive.compactor.history.retention.attempted=2
> hive.compactor.history.retention.failed=3
> hive.compactor.history.retention.succeeded=3
> hive.compactor.initiator.failed.compacts.threshold=2
> hive.compactor.initiator.on=false
> hive.compactor.max.num.delta=500
> hive.compactor.worker.threads=4
> hive.compactor.worker.timeout=86400s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-21931) Slow compaction for tiny tables
[
https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer resolved HIVE-21931.
Resolution: Duplicate
> Slow compaction for tiny tables
> ---
>
> Key: HIVE-21931
> URL: https://issues.apache.org/jira/browse/HIVE-21931
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Csaba Ringhofer
>Priority: Major
> Labels: compaction
>
> I observed the issue in Impala development environment when (major)
> compacting insert_only transactional tables in Hive. The compaction could
> take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The
> actual work was done much earlier, the new base file was correctly written to
> HDFS, and Hive seemed to wait without doing any work.
> The compactions are started manually, hive.compactor.initiator.on=false to
> avoid "surprise compaction" during tests.
> {code}
> hive.compactor.abortedtxn.threshold=1000
> hive.compactor.check.interval=300s
> hive.compactor.cleaner.run.interval=5000ms
> hive.compactor.compact.insert.only=true
> hive.compactor.crud.query.based=false
> hive.compactor.delta.num.threshold=10
> hive.compactor.delta.pct.threshold=0.1
> hive.compactor.history.reaper.interval=2m
> hive.compactor.history.retention.attempted=2
> hive.compactor.history.retention.failed=3
> hive.compactor.history.retention.succeeded=3
> hive.compactor.initiator.failed.compacts.threshold=2
> hive.compactor.initiator.on=false
> hive.compactor.max.num.delta=500
> hive.compactor.worker.threads=4
> hive.compactor.worker.timeout=86400s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22705) LLAP cache is polluted by query-based compactor
[
https://issues.apache.org/jira/browse/HIVE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019647#comment-17019647
]
Hive QA commented on HIVE-22705:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991384/HIVE-22705.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 17933 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterPartition
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterTable
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterTableCascade
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterViewParititon
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testColumnStatistics
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testComplexTable
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testComplexTypeApi
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testConcurrentMetastores
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testCreateAndGetTableWithDriver
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testCreateTableSettingId
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDBLocationChange
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDBOwner
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDBOwnerChange
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDatabase
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDatabaseLocation
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDatabaseLocationWithPermissionProblems
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDropDatabaseCascadeMVMultiDB
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDropTable
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testFilterLastPartition
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testFilterSinglePartition
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testFunctionWithResources
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetConfigValue
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetMetastoreUuid
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetPartitionsWithSpec
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetSchemaWithNoClassDefFoundError
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetTableObjects
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetUUIDInParallel
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testJDOPersistanceManagerCleanup
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testListPartitionNames
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testListPartitions
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testListPartitionsWihtLimitEnabled
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testNameMethods
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartition
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartitionFilter
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testRenamePartition
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testRetriableClientWithConnLifetime
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleFunction
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleTable
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleTypeApi
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testStatsFastTrivial
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSynchronized
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testTableDatabase
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testTableFilter
(batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testUpdatePartitionStat_doesNotUpdateStats
(batchId=228)
[jira] [Commented] (HIVE-22705) LLAP cache is polluted by query-based compactor
[
https://issues.apache.org/jira/browse/HIVE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019632#comment-17019632
]
Hive QA commented on HIVE-22705:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
11s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20252/dev-support/hive-personality.sh
|
| git revision | master / 3c0705e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20252/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP cache is polluted by query-based compactor
> ---
>
> Key: HIVE-22705
> URL: https://issues.apache.org/jira/browse/HIVE-22705
> Project: Hive
> Issue Type: Improvement
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22705.0.patch, HIVE-22705.1.patch
>
>
> One of the steps that query-based compaction does is the verification of ACID
> sort order by using the _validate_acid_sort_order_ UDF. This is a
> prerequisite before the actual compaction can happen, and is done by a [query
> that reads the whole table
> content|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java#L161-L167].
> This results in the whole table content being populated into the cache. The
> problem is that this content is not useful and will rather pollute the cache
> space, as it can never be used again: cache content binds to files (file IDs)
> that obviously will be changed in this case by compaction.
> I propose we disable LLAP caching in the session of query-based compaction's
> queries.
--
This
[jira] [Updated] (HIVE-22745) Config option to turn off read locks
[ https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22745: -- Attachment: HIVE-22745.2.patch > Config option to turn off read locks > > > Key: HIVE-22745 > URL: https://issues.apache.org/jira/browse/HIVE-22745 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Ashutosh Chauhan >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22745.2.patch, HIVE-22745.patch > > > Although its not recommended but in perf critical scenario this option may be > exercised. We have observed lock acquisition to take long time in heavily > loaded system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.02.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21215) Read Parquet INT64 timestamp
[ https://issues.apache.org/jira/browse/HIVE-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-21215: - Description: This patch enables Hive to start reading timestamps from Parquet written with the new semantics: With Parquet version 1.11, a new timestamp LogicalType with base INT64 and the following metadata is introduced: * boolean isAdjustedToUtc: marks whether the timestamp is converted to UTC (aka Instant semantics) or not (LocalDateTime semantics). * enum TimeUnit (NANOS, MICROS, MILLIS): granularity of timestamp Upon reading, the semantics of these new timestamps will be determined by their metadata, while the semantics of INT96 timestamps will continue to be deduced from the writer metadata. This feature will be behind a flag for now. was: [WIP] This patch enables Hive to start reading timestamps from Parquet written with the new semantics: With Parquet version 1.11, a new timestamp LogicalType with base INT64 and the following metadata is introduced: * boolean isAdjustedToUtc: marks whether the timestamp is converted to UTC (aka Instant semantics) or not (LocalDateTime semantics). * enum TimeUnit (NANOS, MICROS, MILLIS): granularity of timestamp Upon reading, the semantics of these new timestamps will be determined by their metadata, while the semantics of INT96 timestamps will continue to be deduced from the writer metadata. This feature will be behind a flag for now. > Read Parquet INT64 timestamp > > > Key: HIVE-21215 > URL: https://issues.apache.org/jira/browse/HIVE-21215 > Project: Hive > Issue Type: New Feature >Reporter: Karen Coppage >Assignee: Marta Kuczora >Priority: Major > > This patch enables Hive to start reading timestamps from Parquet written with > the new semantics: > With Parquet version 1.11, a new timestamp LogicalType with base INT64 and > the following metadata is introduced: > * boolean isAdjustedToUtc: marks whether the timestamp is converted to UTC > (aka Instant semantics) or not (LocalDateTime semantics). > * enum TimeUnit (NANOS, MICROS, MILLIS): granularity of timestamp > Upon reading, the semantics of these new timestamps will be determined by > their metadata, while the semantics of INT96 timestamps will continue to be > deduced from the writer metadata. > This feature will be behind a flag for now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019564#comment-17019564
]
Denys Kuzmenko commented on HIVE-19369:
---
Hi [~gopalv], if there was a schema change (i.e. EXCL_DROP) - all
blocked/dependant locks (any type, not only SHARED_READ) should fail fast,
right?
Following scenario (current logic):
{code}alter table acid add columns (b);{code} - acquires EXCLUSIVE lock;
{code}insert into table acid values (1,2,3);{code} - SHARED_READ, waits for
EXCLUSIVE to be released and later throws IndexOutOfBoundsException: Index: 3,
Size: 3
> Locks: Add new lock implementations for always zero-wait readers
>
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending,
> there's no semantic reason to wait for them to succeed before going ahead
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete
> description.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22705) LLAP cache is polluted by query-based compactor
[ https://issues.apache.org/jira/browse/HIVE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22705: -- Attachment: HIVE-22705.1.patch > LLAP cache is polluted by query-based compactor > --- > > Key: HIVE-22705 > URL: https://issues.apache.org/jira/browse/HIVE-22705 > Project: Hive > Issue Type: Improvement >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22705.0.patch, HIVE-22705.1.patch > > > One of the steps that query-based compaction does is the verification of ACID > sort order by using the _validate_acid_sort_order_ UDF. This is a > prerequisite before the actual compaction can happen, and is done by a [query > that reads the whole table > content|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java#L161-L167]. > This results in the whole table content being populated into the cache. The > problem is that this content is not useful and will rather pollute the cache > space, as it can never be used again: cache content binds to files (file IDs) > that obviously will be changed in this case by compaction. > I propose we disable LLAP caching in the session of query-based compaction's > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019560#comment-17019560
]
Laszlo Pinter commented on HIVE-21487:
--
We should add the following indexes to compaction related tables:
COMPACTION_QUEUE: cq_id and (cq_state, cq_id)
COMPLETED_COMPACTIONS: (cc_database, cc_table, cc_partition)
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21050) Use Parquet LogicalTypes
[ https://issues.apache.org/jira/browse/HIVE-21050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-21050: - Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. > Use Parquet LogicalTypes > > > Key: HIVE-21050 > URL: https://issues.apache.org/jira/browse/HIVE-21050 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: Parquet, parquet > Fix For: 4.0.0 > > Attachments: HIVE-21050.1.patch, HIVE-21050.1.patch, > HIVE-21050.1.patch, HIVE-21050.10.patch, HIVE-21050.11.patch, > HIVE-21050.11.patch, HIVE-21050.2.patch, HIVE-21050.3.patch, > HIVE-21050.4.patch, HIVE-21050.4.patch, HIVE-21050.4.patch, > HIVE-21050.5.patch, HIVE-21050.5.patch, HIVE-21050.5.patch, > HIVE-21050.6.patch, HIVE-21050.6.patch, HIVE-21050.6.patch, > HIVE-21050.6.patch.txt, HIVE-21050.7.patch, HIVE-21050.7.patch, > HIVE-21050.8.patch, HIVE-21050.9.patch > > > [WIP until Parquet community releases version 1.11.0] > The new Parquet version (1.11.0) uses > [LogicalTypes|https://github.com/apache/parquet-format/blob/master/LogicalTypes.md] > instead of OriginalTypes. > These are backwards-compatible with OriginalTypes. > Thanks to [~kuczoram] for her work on this patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22666) Introduce TopNKey operator for PTF Reduce Sink
[
https://issues.apache.org/jira/browse/HIVE-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22666:
--
Status: Open (was: Patch Available)
> Introduce TopNKey operator for PTF Reduce Sink
> --
>
> Key: HIVE-22666
> URL: https://issues.apache.org/jira/browse/HIVE-22666
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22666.1.patch, HIVE-22666.2.patch,
> HIVE-22666.3.patch, HIVE-22666.3.patch, HIVE-22666.4.patch,
> HIVE-22666.4.patch, HIVE-22666.4.patch
>
>
> {code}
> EXPLAIN EXTENDED
> SELECT s_state, ranking
> FROM (
> SELECT s_state AS s_state,
> rank() OVER (PARTITION BY s_state ORDER BY ss_net_profit) AS ranking
> FROM testtable_n1000) tmp1
> WHERE ranking <= 3;
> {code}
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: testtable_n1000
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Reduce Output Operator
> key expressions: s_state (type: string), ss_net_profit
> (type: double)
> null sort order: az
> sort order: ++
> Map-reduce partition columns: s_state (type: string)
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> tag: -1
> TopN: 4
> TopN Hash Memory Usage: 0.1
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: testtable_n1000
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> name: default.testtable_n1000
> name: default.testtable_n1000
> Truncated Path ->
[jira] [Updated] (HIVE-22666) Introduce TopNKey operator for PTF Reduce Sink
[
https://issues.apache.org/jira/browse/HIVE-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22666:
--
Status: Patch Available (was: Open)
> Introduce TopNKey operator for PTF Reduce Sink
> --
>
> Key: HIVE-22666
> URL: https://issues.apache.org/jira/browse/HIVE-22666
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22666.1.patch, HIVE-22666.2.patch,
> HIVE-22666.3.patch, HIVE-22666.3.patch, HIVE-22666.4.patch,
> HIVE-22666.4.patch, HIVE-22666.4.patch
>
>
> {code}
> EXPLAIN EXTENDED
> SELECT s_state, ranking
> FROM (
> SELECT s_state AS s_state,
> rank() OVER (PARTITION BY s_state ORDER BY ss_net_profit) AS ranking
> FROM testtable_n1000) tmp1
> WHERE ranking <= 3;
> {code}
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: testtable_n1000
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Reduce Output Operator
> key expressions: s_state (type: string), ss_net_profit
> (type: double)
> null sort order: az
> sort order: ++
> Map-reduce partition columns: s_state (type: string)
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> tag: -1
> TopN: 4
> TopN Hash Memory Usage: 0.1
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: testtable_n1000
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> name: default.testtable_n1000
> name: default.testtable_n1000
> Truncated Path ->
[jira] [Updated] (HIVE-22666) Introduce TopNKey operator for PTF Reduce Sink
[
https://issues.apache.org/jira/browse/HIVE-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22666:
--
Attachment: HIVE-22666.4.patch
> Introduce TopNKey operator for PTF Reduce Sink
> --
>
> Key: HIVE-22666
> URL: https://issues.apache.org/jira/browse/HIVE-22666
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22666.1.patch, HIVE-22666.2.patch,
> HIVE-22666.3.patch, HIVE-22666.3.patch, HIVE-22666.4.patch,
> HIVE-22666.4.patch, HIVE-22666.4.patch
>
>
> {code}
> EXPLAIN EXTENDED
> SELECT s_state, ranking
> FROM (
> SELECT s_state AS s_state,
> rank() OVER (PARTITION BY s_state ORDER BY ss_net_profit) AS ranking
> FROM testtable_n1000) tmp1
> WHERE ranking <= 3;
> {code}
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: testtable_n1000
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Reduce Output Operator
> key expressions: s_state (type: string), ss_net_profit
> (type: double)
> null sort order: az
> sort order: ++
> Map-reduce partition columns: s_state (type: string)
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> tag: -1
> TopN: 4
> TopN Hash Memory Usage: 0.1
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: testtable_n1000
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> name: default.testtable_n1000
> name: default.testtable_n1000
> Truncated Path -> Alias:
>
[jira] [Assigned] (HIVE-21487) COMPLETED_COMPACTIONS table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter reassigned HIVE-21487:
Assignee: Laszlo Pinter
> COMPLETED_COMPACTIONS table missing appropriate indexes
> ---
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21487:
-
Summary: COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing
appropriate indexes (was: COMPLETED_COMPACTIONS table missing appropriate
indexes)
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22747) Break up DDLSemanticAnalyzer - extract Table info and lock analyzers
[ https://issues.apache.org/jira/browse/HIVE-22747?focusedWorklogId=374595=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374595 ] ASF GitHub Bot logged work on HIVE-22747: - Author: ASF GitHub Bot Created on: 20/Jan/20 15:01 Start Date: 20/Jan/20 15:01 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #882: HIVE-22747 Break up DDLSemanticAnalyzer - extract Table info and lock analyzers URL: https://github.com/apache/hive/pull/882 DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is to refactor it in order to have everything cut into more handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: - have a separate class for each analyzers - have a package for each operation, containing an analyzer, a description, and an operation, so the amount of classes under a package is more manageable Step #13: extract the table info and lock related analyzers from DDLSemanticAnalyzer, and move them under the new package. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 374595) Remaining Estimate: 0h Time Spent: 10m > Break up DDLSemanticAnalyzer - extract Table info and lock analyzers > > > Key: HIVE-22747 > URL: https://issues.apache.org/jira/browse/HIVE-22747 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22747.01.patch, HIVE-22747.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #13: extract the table info and lock related analyzers from > DDLSemanticAnalyzer, and move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[
https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019534#comment-17019534
]
Laszlo Pinter commented on HIVE-21931:
--
[~csringhofer] It's a session property.
> Slow compaction for tiny tables
> ---
>
> Key: HIVE-21931
> URL: https://issues.apache.org/jira/browse/HIVE-21931
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Csaba Ringhofer
>Priority: Major
> Labels: compaction
>
> I observed the issue in Impala development environment when (major)
> compacting insert_only transactional tables in Hive. The compaction could
> take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The
> actual work was done much earlier, the new base file was correctly written to
> HDFS, and Hive seemed to wait without doing any work.
> The compactions are started manually, hive.compactor.initiator.on=false to
> avoid "surprise compaction" during tests.
> {code}
> hive.compactor.abortedtxn.threshold=1000
> hive.compactor.check.interval=300s
> hive.compactor.cleaner.run.interval=5000ms
> hive.compactor.compact.insert.only=true
> hive.compactor.crud.query.based=false
> hive.compactor.delta.num.threshold=10
> hive.compactor.delta.pct.threshold=0.1
> hive.compactor.history.reaper.interval=2m
> hive.compactor.history.retention.attempted=2
> hive.compactor.history.retention.failed=3
> hive.compactor.history.retention.succeeded=3
> hive.compactor.initiator.failed.compacts.threshold=2
> hive.compactor.initiator.on=false
> hive.compactor.max.num.delta=500
> hive.compactor.worker.threads=4
> hive.compactor.worker.timeout=86400s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[
https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019518#comment-17019518
]
Csaba Ringhofer commented on HIVE-21931:
[~lpinter] yes, that sounds good. I still don't understand why do we need such
a large default though.
Is it a session property, or I have to set it at Sentry startup time?
> Slow compaction for tiny tables
> ---
>
> Key: HIVE-21931
> URL: https://issues.apache.org/jira/browse/HIVE-21931
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Csaba Ringhofer
>Priority: Major
> Labels: compaction
>
> I observed the issue in Impala development environment when (major)
> compacting insert_only transactional tables in Hive. The compaction could
> take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The
> actual work was done much earlier, the new base file was correctly written to
> HDFS, and Hive seemed to wait without doing any work.
> The compactions are started manually, hive.compactor.initiator.on=false to
> avoid "surprise compaction" during tests.
> {code}
> hive.compactor.abortedtxn.threshold=1000
> hive.compactor.check.interval=300s
> hive.compactor.cleaner.run.interval=5000ms
> hive.compactor.compact.insert.only=true
> hive.compactor.crud.query.based=false
> hive.compactor.delta.num.threshold=10
> hive.compactor.delta.pct.threshold=0.1
> hive.compactor.history.reaper.interval=2m
> hive.compactor.history.retention.attempted=2
> hive.compactor.history.retention.failed=3
> hive.compactor.history.retention.succeeded=3
> hive.compactor.initiator.failed.compacts.threshold=2
> hive.compactor.initiator.on=false
> hive.compactor.max.num.delta=500
> hive.compactor.worker.threads=4
> hive.compactor.worker.timeout=86400s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[
https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019501#comment-17019501
]
Laszlo Pinter commented on HIVE-21931:
--
[~csringhofer] With HIVE-22554 you can configure the initial wait time out to a
much lower value, like 2000 millisec. Does that satisfies your needs?
> Slow compaction for tiny tables
> ---
>
> Key: HIVE-21931
> URL: https://issues.apache.org/jira/browse/HIVE-21931
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Csaba Ringhofer
>Priority: Major
> Labels: compaction
>
> I observed the issue in Impala development environment when (major)
> compacting insert_only transactional tables in Hive. The compaction could
> take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The
> actual work was done much earlier, the new base file was correctly written to
> HDFS, and Hive seemed to wait without doing any work.
> The compactions are started manually, hive.compactor.initiator.on=false to
> avoid "surprise compaction" during tests.
> {code}
> hive.compactor.abortedtxn.threshold=1000
> hive.compactor.check.interval=300s
> hive.compactor.cleaner.run.interval=5000ms
> hive.compactor.compact.insert.only=true
> hive.compactor.crud.query.based=false
> hive.compactor.delta.num.threshold=10
> hive.compactor.delta.pct.threshold=0.1
> hive.compactor.history.reaper.interval=2m
> hive.compactor.history.retention.attempted=2
> hive.compactor.history.retention.failed=3
> hive.compactor.history.retention.succeeded=3
> hive.compactor.initiator.failed.compacts.threshold=2
> hive.compactor.initiator.on=false
> hive.compactor.max.num.delta=500
> hive.compactor.worker.threads=4
> hive.compactor.worker.timeout=86400s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[
https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019466#comment-17019466
]
Csaba Ringhofer commented on HIVE-21931:
[~Rajkumar Singh] Sorry, I missed your comment for some reason.
[~lpinter] Yes, I do run it with " and wait". This is needed for the tests, as
we specifically test if Impala can read a table after/during compaction.
> Slow compaction for tiny tables
> ---
>
> Key: HIVE-21931
> URL: https://issues.apache.org/jira/browse/HIVE-21931
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Csaba Ringhofer
>Priority: Major
> Labels: compaction
>
> I observed the issue in Impala development environment when (major)
> compacting insert_only transactional tables in Hive. The compaction could
> take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The
> actual work was done much earlier, the new base file was correctly written to
> HDFS, and Hive seemed to wait without doing any work.
> The compactions are started manually, hive.compactor.initiator.on=false to
> avoid "surprise compaction" during tests.
> {code}
> hive.compactor.abortedtxn.threshold=1000
> hive.compactor.check.interval=300s
> hive.compactor.cleaner.run.interval=5000ms
> hive.compactor.compact.insert.only=true
> hive.compactor.crud.query.based=false
> hive.compactor.delta.num.threshold=10
> hive.compactor.delta.pct.threshold=0.1
> hive.compactor.history.reaper.interval=2m
> hive.compactor.history.retention.attempted=2
> hive.compactor.history.retention.failed=3
> hive.compactor.history.retention.succeeded=3
> hive.compactor.initiator.failed.compacts.threshold=2
> hive.compactor.initiator.on=false
> hive.compactor.max.num.delta=500
> hive.compactor.worker.threads=4
> hive.compactor.worker.timeout=86400s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019460#comment-17019460
]
Hive QA commented on HIVE-22729:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991374/HIVE-22729.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20251/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20251/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20251/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-01-20 13:06:14.659
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20251/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-01-20 13:06:14.662
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
f0f46b7..77bec20 master -> origin/master
+ git reset --hard HEAD
HEAD is now at f0f46b7 HIVE-22735: TopNKey operator deduplication (Krisztian
Kasa, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 77bec20 HIVE-22703: Compaction configuration check when starting
HMS/HS2 (Laszlo Pinter reviewed by Denys Kuzmenko and Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-01-20 13:06:19.187
+ rm -rf ../yetus_PreCommit-HIVE-Build-20251
+ mkdir ../yetus_PreCommit-HIVE-Build-20251
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20251
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20251/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error:
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsDesc.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java: does
not exist in index
error:
a/ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java:
does not exist in index
error:
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[
https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019458#comment-17019458
]
Laszlo Pinter commented on HIVE-21931:
--
[~csringhofer] Did you run compaction in blocking mode?
HIVE-22554 provides a way to configure the wait time out.
> Slow compaction for tiny tables
> ---
>
> Key: HIVE-21931
> URL: https://issues.apache.org/jira/browse/HIVE-21931
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Csaba Ringhofer
>Priority: Major
> Labels: compaction
>
> I observed the issue in Impala development environment when (major)
> compacting insert_only transactional tables in Hive. The compaction could
> take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The
> actual work was done much earlier, the new base file was correctly written to
> HDFS, and Hive seemed to wait without doing any work.
> The compactions are started manually, hive.compactor.initiator.on=false to
> avoid "surprise compaction" during tests.
> {code}
> hive.compactor.abortedtxn.threshold=1000
> hive.compactor.check.interval=300s
> hive.compactor.cleaner.run.interval=5000ms
> hive.compactor.compact.insert.only=true
> hive.compactor.crud.query.based=false
> hive.compactor.delta.num.threshold=10
> hive.compactor.delta.pct.threshold=0.1
> hive.compactor.history.reaper.interval=2m
> hive.compactor.history.retention.attempted=2
> hive.compactor.history.retention.failed=3
> hive.compactor.history.retention.succeeded=3
> hive.compactor.initiator.failed.compacts.threshold=2
> hive.compactor.initiator.on=false
> hive.compactor.max.num.delta=500
> hive.compactor.worker.threads=4
> hive.compactor.worker.timeout=86400s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Status: Patch Available (was: Open)
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22750) Consolidate LockType naming
[
https://issues.apache.org/jira/browse/HIVE-22750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan reassigned HIVE-22750:
> Consolidate LockType naming
> ---
>
> Key: HIVE-22750
> URL: https://issues.apache.org/jira/browse/HIVE-22750
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
>
> Extend enum with string literal to remove unnecessary `id` to `char` casting
> for the LockType:
> {code:java}
> switch (lockType) {
> case EXCLUSIVE:
> lockChar = LOCK_EXCLUSIVE;
> break;
> case SHARED_READ:
> lockChar = LOCK_SHARED;
> break;
> case SHARED_WRITE:
> lockChar = LOCK_SEMI_SHARED;
> break;
> }
> {code}
> Consolidate LockType naming in code and schema upgrade scripts:
> {code:java}
> CASE WHEN HL.`HL_LOCK_TYPE` = 'e' THEN 'exclusive' WHEN HL.`HL_LOCK_TYPE` =
> 'r' THEN 'shared' WHEN HL.`HL_LOCK_TYPE` = 'w' THEN *'semi-shared'* END AS
> LOCK_TYPE,
> {code}
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.01.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-20890) ACID: Allow whole table ReadLocks to skip all partition locks
[
https://issues.apache.org/jira/browse/HIVE-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan reassigned HIVE-20890:
Assignee: Zoltan Chovan
> ACID: Allow whole table ReadLocks to skip all partition locks
> -
>
> Key: HIVE-20890
> URL: https://issues.apache.org/jira/browse/HIVE-20890
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Zoltan Chovan
>Priority: Major
>
> HIVE-19369 proposes adding a EXCL_WRITE lock which does not wait for any
> SHARED_READ locks for insert operations - in the presence of that lock, the
> insert overwrite no longer takes an exclusive lock.
> The only exclusive operation will be a schema change or drop table, which
> should take an exclusive lock on the entire table directly.
> {code}
> explain locks select * from tpcds_bin_partitioned_orc_1000.store_sales where
> ss_sold_date_sk=2452626
> ++
> | Explain |
> ++
> | LOCK INFORMATION: |
> | tpcds_bin_partitioned_orc_1000.store_sales -> SHARED_READ |
> | tpcds_bin_partitioned_orc_1000.store_sales.ss_sold_date_sk=2452626 ->
> SHARED_READ |
> ++
> {code}
> So the per-partition SHARED_READ locks are no longer necessary, if the lock
> builder already includes the table-wide SHARED_READ locks.
> The removal of entire partitions is the only part which needs to be taken
> care of within this semantics as row-removal instead of directory removal
> (i.e "drop partition" -> "truncate partition" and have the truncation trigger
> a whole directory cleaner, so that the partition disappears when there are 0
> rows left).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22663) Quote all table and column names or do not quote any
[
https://issues.apache.org/jira/browse/HIVE-22663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019388#comment-17019388
]
Peter Vary commented on HIVE-22663:
---
+1
> Quote all table and column names or do not quote any
>
>
> Key: HIVE-22663
> URL: https://issues.apache.org/jira/browse/HIVE-22663
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Zoltan Chovan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22663.2.patch, HIVE-22663.3.patch,
> HIVE-22663.4.patch, HIVE-22663.5.patch, HIVE-22663.6.patch, HIVE-22663.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The change in HIVE-22546 is causing following stack trace when I run Hive
> with PostgreSQL as backend db for the metastore.
> 0: jdbc:hive2://localhost:1> create database dumpdb with
> ('repl.source.for'='1,2,3');0: jdbc:hive2://localhost:1> create database
> dumpdb with ('repl.source.for'='1,2,3');Error: Error while compiling
> statement: FAILED: ParseException line 1:28 missing KW_DBPROPERTIES at '('
> near '' (state=42000,code=4)0: jdbc:hive2://localhost:1> create
> database dumpdb with dbproperties ('repl.source.for'='1,2,3');ERROR : FAILED:
> Hive Internal Error: org.apache.hadoop.hive.ql.lockmgr.LockException(Error
> communicating with the
> metastore)org.apache.hadoop.hive.ql.lockmgr.LockException: Error
> communicating with the metastore at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.commitTxn(DbTxnManager.java:541)
> at
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:687)
> at
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:653)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:969)
> ... stack trace clipped
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)Caused by:
> MetaException(message:Unable to update transaction database
> org.postgresql.util.PSQLException: ERROR: relation
> "materialization_rebuild_locks" does not exist Position: 13 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at
> This happens because the table names in all the queries in TxnHandler.java
> (including the one at 1312, which causes this stack trace) are not quoting
> the table names. All the tablenames and column names should be quoted there.
> Just the change in HIVE-22546 won't suffice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21050) Use Parquet LogicalTypes
[ https://issues.apache.org/jira/browse/HIVE-21050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21050: - Attachment: (was: HIVE-21050.12.patch) > Use Parquet LogicalTypes > > > Key: HIVE-21050 > URL: https://issues.apache.org/jira/browse/HIVE-21050 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: Parquet, parquet > Attachments: HIVE-21050.1.patch, HIVE-21050.1.patch, > HIVE-21050.1.patch, HIVE-21050.10.patch, HIVE-21050.11.patch, > HIVE-21050.11.patch, HIVE-21050.2.patch, HIVE-21050.3.patch, > HIVE-21050.4.patch, HIVE-21050.4.patch, HIVE-21050.4.patch, > HIVE-21050.5.patch, HIVE-21050.5.patch, HIVE-21050.5.patch, > HIVE-21050.6.patch, HIVE-21050.6.patch, HIVE-21050.6.patch, > HIVE-21050.6.patch.txt, HIVE-21050.7.patch, HIVE-21050.7.patch, > HIVE-21050.8.patch, HIVE-21050.9.patch > > > [WIP until Parquet community releases version 1.11.0] > The new Parquet version (1.11.0) uses > [LogicalTypes|https://github.com/apache/parquet-format/blob/master/LogicalTypes.md] > instead of OriginalTypes. > These are backwards-compatible with OriginalTypes. > Thanks to [~kuczoram] for her work on this patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21050) Use Parquet LogicalTypes
[ https://issues.apache.org/jira/browse/HIVE-21050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21050: - Attachment: (was: HIVE-21050.12.patch) > Use Parquet LogicalTypes > > > Key: HIVE-21050 > URL: https://issues.apache.org/jira/browse/HIVE-21050 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: Parquet, parquet > Attachments: HIVE-21050.1.patch, HIVE-21050.1.patch, > HIVE-21050.1.patch, HIVE-21050.10.patch, HIVE-21050.11.patch, > HIVE-21050.11.patch, HIVE-21050.2.patch, HIVE-21050.3.patch, > HIVE-21050.4.patch, HIVE-21050.4.patch, HIVE-21050.4.patch, > HIVE-21050.5.patch, HIVE-21050.5.patch, HIVE-21050.5.patch, > HIVE-21050.6.patch, HIVE-21050.6.patch, HIVE-21050.6.patch, > HIVE-21050.6.patch.txt, HIVE-21050.7.patch, HIVE-21050.7.patch, > HIVE-21050.8.patch, HIVE-21050.9.patch > > > [WIP until Parquet community releases version 1.11.0] > The new Parquet version (1.11.0) uses > [LogicalTypes|https://github.com/apache/parquet-format/blob/master/LogicalTypes.md] > instead of OriginalTypes. > These are backwards-compatible with OriginalTypes. > Thanks to [~kuczoram] for her work on this patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22730) Do not acquire read lock for dummy input
[
https://issues.apache.org/jira/browse/HIVE-22730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019362#comment-17019362
]
Denys Kuzmenko commented on HIVE-22730:
---
Thank you, [~jcamachorodriguez]!
> Do not acquire read lock for dummy input
>
>
> Key: HIVE-22730
> URL: https://issues.apache.org/jira/browse/HIVE-22730
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22730.1.patch, HIVE-22730.2.patch,
> HIVE-22730.3.patch, HIVE-22730.4.patch, Screenshot 2020-01-15 at 16.49.16.png
>
>
> {code}
> insert into b values(1);
> {code}
> List lockComponents =
> AcidUtils.makeLockComponents(plan.getOutputs(), plan.getInputs(), conf);
> plan.getInputs() contains single entry <_dummy_database@_dummy_table>
> !Screenshot 2020-01-15 at 16.49.16.png|width=800,height=60!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21050) Use Parquet LogicalTypes
[ https://issues.apache.org/jira/browse/HIVE-21050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019356#comment-17019356 ] Marta Kuczora commented on HIVE-21050: -- +1 Thanks a lot [~klcopp] for the patch. > Use Parquet LogicalTypes > > > Key: HIVE-21050 > URL: https://issues.apache.org/jira/browse/HIVE-21050 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: Parquet, parquet > Attachments: HIVE-21050.1.patch, HIVE-21050.1.patch, > HIVE-21050.1.patch, HIVE-21050.10.patch, HIVE-21050.11.patch, > HIVE-21050.11.patch, HIVE-21050.12.patch, HIVE-21050.12.patch, > HIVE-21050.2.patch, HIVE-21050.3.patch, HIVE-21050.4.patch, > HIVE-21050.4.patch, HIVE-21050.4.patch, HIVE-21050.5.patch, > HIVE-21050.5.patch, HIVE-21050.5.patch, HIVE-21050.6.patch, > HIVE-21050.6.patch, HIVE-21050.6.patch, HIVE-21050.6.patch.txt, > HIVE-21050.7.patch, HIVE-21050.7.patch, HIVE-21050.8.patch, HIVE-21050.9.patch > > > [WIP until Parquet community releases version 1.11.0] > The new Parquet version (1.11.0) uses > [LogicalTypes|https://github.com/apache/parquet-format/blob/master/LogicalTypes.md] > instead of OriginalTypes. > These are backwards-compatible with OriginalTypes. > Thanks to [~kuczoram] for her work on this patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22703: -- Comment: was deleted (was: Thank you [~lpinter], [~pvary] for review! ) > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22703.01.patch, HIVE-22703.02.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019342#comment-17019342 ] Denys Kuzmenko commented on HIVE-22703: --- Thank you [~lpinter], [~pvary] for review! > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22703.01.patch, HIVE-22703.02.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019340#comment-17019340 ] Laszlo Pinter commented on HIVE-22703: -- Thanks [~pvary] and [~dkuzmenko] > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22703.01.patch, HIVE-22703.02.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-22703: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~lpinter], and [~dkuzmenko] for the review! > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22703.01.patch, HIVE-22703.02.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22727) Add hive db schema changes introduced in HIVE-21884 to the schema upgrade scripts
[ https://issues.apache.org/jira/browse/HIVE-22727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-22727: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~zchovan], and [~kgyrtkirk] for the review! > Add hive db schema changes introduced in HIVE-21884 to the schema upgrade > scripts > - > > Key: HIVE-22727 > URL: https://issues.apache.org/jira/browse/HIVE-22727 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22727.2.patch, HIVE-22727.3.patch, HIVE-22727.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22627) Add schema changes introduced in HIVE-21443 to the schema upgrade scripts
[ https://issues.apache.org/jira/browse/HIVE-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019320#comment-17019320 ] Peter Vary commented on HIVE-22627: --- +1 > Add schema changes introduced in HIVE-21443 to the schema upgrade scripts > - > > Key: HIVE-22627 > URL: https://issues.apache.org/jira/browse/HIVE-22627 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22627.2.patch, HIVE-22627.3.patch, HIVE-22627.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22727) Add hive db schema changes introduced in HIVE-21884 to the schema upgrade scripts
[ https://issues.apache.org/jira/browse/HIVE-22727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019319#comment-17019319 ] Zoltan Chovan commented on HIVE-22727: -- [~pvary] [~kgyrtkirk] could you take a look? > Add hive db schema changes introduced in HIVE-21884 to the schema upgrade > scripts > - > > Key: HIVE-22727 > URL: https://issues.apache.org/jira/browse/HIVE-22727 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-22727.2.patch, HIVE-22727.3.patch, HIVE-22727.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22627) Add schema changes introduced in HIVE-21443 to the schema upgrade scripts
[ https://issues.apache.org/jira/browse/HIVE-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019317#comment-17019317 ] Zoltan Chovan commented on HIVE-22627: -- [~pvary] [~kgyrtkirk] could you take a look? > Add schema changes introduced in HIVE-21443 to the schema upgrade scripts > - > > Key: HIVE-22627 > URL: https://issues.apache.org/jira/browse/HIVE-22627 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22627.2.patch, HIVE-22627.3.patch, HIVE-22627.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22663) Quote all table and column names or do not quote any
[
https://issues.apache.org/jira/browse/HIVE-22663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019318#comment-17019318
]
Zoltan Chovan commented on HIVE-22663:
--
[~pvary] [~kgyrtkirk] could you take a look?
> Quote all table and column names or do not quote any
>
>
> Key: HIVE-22663
> URL: https://issues.apache.org/jira/browse/HIVE-22663
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Zoltan Chovan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22663.2.patch, HIVE-22663.3.patch,
> HIVE-22663.4.patch, HIVE-22663.5.patch, HIVE-22663.6.patch, HIVE-22663.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The change in HIVE-22546 is causing following stack trace when I run Hive
> with PostgreSQL as backend db for the metastore.
> 0: jdbc:hive2://localhost:1> create database dumpdb with
> ('repl.source.for'='1,2,3');0: jdbc:hive2://localhost:1> create database
> dumpdb with ('repl.source.for'='1,2,3');Error: Error while compiling
> statement: FAILED: ParseException line 1:28 missing KW_DBPROPERTIES at '('
> near '' (state=42000,code=4)0: jdbc:hive2://localhost:1> create
> database dumpdb with dbproperties ('repl.source.for'='1,2,3');ERROR : FAILED:
> Hive Internal Error: org.apache.hadoop.hive.ql.lockmgr.LockException(Error
> communicating with the
> metastore)org.apache.hadoop.hive.ql.lockmgr.LockException: Error
> communicating with the metastore at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.commitTxn(DbTxnManager.java:541)
> at
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:687)
> at
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:653)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:969)
> ... stack trace clipped
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)Caused by:
> MetaException(message:Unable to update transaction database
> org.postgresql.util.PSQLException: ERROR: relation
> "materialization_rebuild_locks" does not exist Position: 13 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at
> This happens because the table names in all the queries in TxnHandler.java
> (including the one at 1312, which causes this stack trace) are not quoting
> the table names. All the tablenames and column names should be quoted there.
> Just the change in HIVE-22546 won't suffice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
