[jira] [Updated] (HIVE-23296) Setting Tez caller ID with the actual Hive user
[ https://issues.apache.org/jira/browse/HIVE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-23296: Resolution: Won't Fix Status: Resolved (was: Patch Available) > Setting Tez caller ID with the actual Hive user > --- > > Key: HIVE-23296 > URL: https://issues.apache.org/jira/browse/HIVE-23296 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Attachments: HIVE-23296.01.patch, Screen Shot 2020-04-24 at > 17.20.34.png > > > On the kerberized Hadoop environment, a submitter of an YARN job is the name > part of the Hive server principal. A caller ID of the job is made of the OS > user of the Hive server process. > The view and modify ACLs of the Hive server for all Tez tasks are set by > org.apache.hadoop.hive.ql.exec.tez.TezTask#setAccessControlsForCurrentUser() > so that the admin who has the Hive server principal can see all tasks from > tez-ui. But the admin hardly knows who executed each query. > I suggest to change the caller ID to include the actual Hive user. If the > user is not known, the OS user of the Hive server process is included as is. > The attached picture shows that 'Caller ID' includes 'user1' which is the > Kerberos user name of the actual Hive user. > !Screen Shot 2020-04-24 at 17.20.34.png|width=683,height=29! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23296) Setting Tez caller ID with the actual Hive user
[ https://issues.apache.org/jira/browse/HIVE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099569#comment-17099569 ] Eugene Chung commented on HIVE-23296: - This is the result of my misunderstanding of Hadoop impersonation feature. > Setting Tez caller ID with the actual Hive user > --- > > Key: HIVE-23296 > URL: https://issues.apache.org/jira/browse/HIVE-23296 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Attachments: HIVE-23296.01.patch, Screen Shot 2020-04-24 at > 17.20.34.png > > > On the kerberized Hadoop environment, a submitter of an YARN job is the name > part of the Hive server principal. A caller ID of the job is made of the OS > user of the Hive server process. > The view and modify ACLs of the Hive server for all Tez tasks are set by > org.apache.hadoop.hive.ql.exec.tez.TezTask#setAccessControlsForCurrentUser() > so that the admin who has the Hive server principal can see all tasks from > tez-ui. But the admin hardly knows who executed each query. > I suggest to change the caller ID to include the actual Hive user. If the > user is not known, the OS user of the Hive server process is included as is. > The attached picture shows that 'Caller ID' includes 'user1' which is the > Kerberos user name of the actual Hive user. > !Screen Shot 2020-04-24 at 17.20.34.png|width=683,height=29! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099556#comment-17099556
]
Hive QA commented on HIVE-23349:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002025/HIVE-23349.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17209 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sharedwork]
(batchId=106)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22133/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22133/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22133/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002025 - PreCommit-HIVE-Build
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23349.1.patch, HIVE-23349.2.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099538#comment-17099538
]
Hive QA commented on HIVE-23349:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22133/dev-support/hive-personality.sh
|
| git revision | master / 5a119b9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22133/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23349.1.patch, HIVE-23349.2.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099533#comment-17099533 ] Ashutosh Chauhan commented on HIVE-23282: - Left some comments on RB. In patch, there is an additional RDBMS call which can be avoided. > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch, > HIVE-23282.5.patch, HIVE-23282.6.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23325) Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099532#comment-17099532
]
Hive QA commented on HIVE-23325:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002019/HIVE-23325.5.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17207 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_insert_partitioned]
(batchId=94)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sharedwork]
(batchId=106)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22132/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22132/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22132/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002019 - PreCommit-HIVE-Build
> Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
> --
>
> Key: HIVE-23325
> URL: https://issues.apache.org/jira/browse/HIVE-23325
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch,
> HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch,
> HIVE-23325.4.patch, HIVE-23325.5.patch
>
>
> Currently there are multiple ways to clean up ACID related stuff:
> * AcidHouseKeeperService
> * AcidWriteSetService
> * AcidCompactionHistoryService
> * Initiator
> * Cleaner
> * etc.
> We should consolidate them where possible and improve logging.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23325) Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099526#comment-17099526
]
Hive QA commented on HIVE-23325:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
21s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
36s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 897 unchanged - 3 fixed = 897 total (was 900) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} ql: The patch generated 0 new + 716 unchanged - 4
fixed = 716 total (was 720) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22132/dev-support/hive-personality.sh
|
| git revision | master / 5a119b9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22132/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
> --
>
> Key: HIVE-23325
>
[jira] [Commented] (HIVE-22681) Replace Base64 in hcatalog-webhcat Package
[
https://issues.apache.org/jira/browse/HIVE-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099509#comment-17099509
]
Hive QA commented on HIVE-22681:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002003/HIVE-22681.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17203 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sharedwork]
(batchId=106)
org.apache.hive.hcatalog.api.TestHCatClient.testReplicationTaskIter
(batchId=136)
org.apache.hive.hcatalog.api.repl.commands.TestCommands.testDropDatabaseCommand
(batchId=136)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22131/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22131/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22131/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002003 - PreCommit-HIVE-Build
> Replace Base64 in hcatalog-webhcat Package
> --
>
> Key: HIVE-22681
> URL: https://issues.apache.org/jira/browse/HIVE-22681
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22681.1.patch, HIVE-22681.1.patch,
> HIVE-22681.1.patch, HIVE-22681.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22681) Replace Base64 in hcatalog-webhcat Package
[
https://issues.apache.org/jira/browse/HIVE-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099500#comment-17099500
]
Hive QA commented on HIVE-22681:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} hcatalog/webhcat/java-client in master has 3 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22131/dev-support/hive-personality.sh
|
| git revision | master / 5a119b9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: hcatalog/webhcat/java-client U: hcatalog/webhcat/java-client |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22131/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Replace Base64 in hcatalog-webhcat Package
> --
>
> Key: HIVE-22681
> URL: https://issues.apache.org/jira/browse/HIVE-22681
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22681.1.patch, HIVE-22681.1.patch,
> HIVE-22681.1.patch, HIVE-22681.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23318) TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
[
https://issues.apache.org/jira/browse/HIVE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099492#comment-17099492
]
Hive QA commented on HIVE-23318:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002005/HIVE-23318.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17209 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sharedwork]
(batchId=106)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22130/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22130/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22130/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002005 - PreCommit-HIVE-Build
> TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
> ---
>
> Key: HIVE-23318
> URL: https://issues.apache.org/jira/browse/HIVE-23318
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Marton Bod
>Priority: Minor
> Attachments: HIVE-23318.1.patch
>
>
> Observed the following queries when materialized view or any of its feature
> was not used.
> TxnHandler need not clear this part of txn commit. It would help in reducing
> the sql parsing time in server side as well.
> {noformat}
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398082
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398084
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398112
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23318) TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
[
https://issues.apache.org/jira/browse/HIVE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099487#comment-17099487
]
Hive QA commented on HIVE-23318:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
22s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 2 new + 150 unchanged - 0
fixed = 152 total (was 150) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22130/dev-support/hive-personality.sh
|
| git revision | master / 5a119b9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22130/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22130/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
> ---
>
> Key: HIVE-23318
> URL: https://issues.apache.org/jira/browse/HIVE-23318
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Marton Bod
>Priority: Minor
> Attachments: HIVE-23318.1.patch
>
>
> Observed the following queries when materialized view or any of its feature
> was not used.
> TxnHandler need not clear this part of txn commit. It would help in reducing
> the sql parsing time in server side as well.
> {noformat}
> Gdelete from MATERIALIZATION_R
[jira] [Commented] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[
https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099442#comment-17099442
]
Hive QA commented on HIVE-23363:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22129/dev-support/hive-personality.sh
|
| git revision | master / 5a119b9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore standalone-metastore/metastore-server U:
standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22129/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade DataNucleus dependency to 5.2
> -
>
> Key: HIVE-23363
> URL: https://issues.apache.org/jira/browse/HIVE-23363
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-23363.patch
>
>
> Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been
> retired:
> [http://www.datanucleus.org/documentation/products.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22674) Replace Base64 in serde Package
[
https://issues.apache.org/jira/browse/HIVE-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099432#comment-17099432
]
Hive QA commented on HIVE-22674:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002002/HIVE-22674.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17203 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[compute_stats_binary]
(batchId=85)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sharedwork]
(batchId=106)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testReplTimeouts
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22128/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22128/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22128/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002002 - PreCommit-HIVE-Build
> Replace Base64 in serde Package
> ---
>
> Key: HIVE-22674
> URL: https://issues.apache.org/jira/browse/HIVE-22674
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22674.1.patch, HIVE-22674.2.patch,
> HIVE-22674.2.patch, HIVE-22674.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22674) Replace Base64 in serde Package
[
https://issues.apache.org/jira/browse/HIVE-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099416#comment-17099416
]
Hive QA commented on HIVE-22674:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
52s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} serde: The patch generated 1 new + 33 unchanged - 1
fixed = 34 total (was 34) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22128/dev-support/hive-personality.sh
|
| git revision | master / 5a119b9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22128/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22128/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Replace Base64 in serde Package
> ---
>
> Key: HIVE-22674
> URL: https://issues.apache.org/jira/browse/HIVE-22674
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22674.1.patch, HIVE-22674.2.patch,
> HIVE-22674.2.patch, HIVE-22674.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23350) Upgrade DBCP To DBCP 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-23350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099415#comment-17099415
]
Hive QA commented on HIVE-23350:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
24s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
11s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} jdbc-handler in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} standalone-metastore: The patch generated 0 new + 79
unchanged - 1 fixed = 79 total (was 80) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch metastore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 79 unchanged - 1 fixed = 79 total (was 80) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch jdbc-handler passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
54s{color} | {color:green} root: The patch generated 0 new + 79 unchanged - 1
fixed = 79 total (was 80) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
14s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
12s{color} | {color:red} patch/standalone-metastore/metastore-common cannot run
setBugDatabaseInfo from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
38s{color} | {color:red} patch/metastore cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
15s{color} | {color:red} patch/standalone-metastore/metastore-server cannot run
setBugDatabaseInfo from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
43s{color} | {color:red} patch/jdbc-handler cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
41s{color} | {color:red}
[jira] [Commented] (HIVE-23350) Upgrade DBCP To DBCP 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-23350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099403#comment-17099403
]
Hive QA commented on HIVE-23350:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001840/HIVE-23350.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17203 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[merge_empty] (batchId=15)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sharedwork]
(batchId=106)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22127/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22127/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22127/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001840 - PreCommit-HIVE-Build
> Upgrade DBCP To DBCP 2.7.0
> --
>
> Key: HIVE-23350
> URL: https://issues.apache.org/jira/browse/HIVE-23350
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23350.1.patch
>
>
> In {{hive-standalone-metastore-common}} the current version is 1.4 which is
> very antiquated and apparently (emphasis mine):
> bq. DBCP 1.4 compiles and runs under Java 6 *only* (JDBC 4)
> Hive is on Java 8
> {code:xml}
> 2.6.0
> 1.4
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23365) Put RS deduplication optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23365: --- Description: Currently, RS deduplication is always executed whenever it is semantically correct. However, it could be beneficial to leave both RS operators in the plan, e.g., if the NDV of the second RS is very low. Thus, we would like this decision to be cost-based. We could use a simple heuristic that would work fine for most of the cases without introducing regressions for existing cases, e.g., if NDV for partition column is less than estimated parallelism in the second RS, do not execute deduplication. (was: Currently, RS deduplication is always executed whenever it is semantically correct. However, it could be beneficial if t to leave both RS operators in the plan, e.g., if the NDV of the second RS is very low. Thus, we would like this decision to be cost-based. We could use a simple heuristic that would work fine for most of the cases without introducing regressions for existing cases, e.g., if NDV for partition column is less than estimated parallelism in the second RS, do not execute deduplication.) > Put RS deduplication optimization under cost based decision > --- > > Key: HIVE-23365 > URL: https://issues.apache.org/jira/browse/HIVE-23365 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Priority: Major > > Currently, RS deduplication is always executed whenever it is semantically > correct. However, it could be beneficial to leave both RS operators in the > plan, e.g., if the NDV of the second RS is very low. Thus, we would like this > decision to be cost-based. We could use a simple heuristic that would work > fine for most of the cases without introducing regressions for existing > cases, e.g., if NDV for partition column is less than estimated parallelism > in the second RS, do not execute deduplication. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23359) "show tables like" support for SQL wildcard characters (% and _)
[ https://issues.apache.org/jira/browse/HIVE-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23359: -- Status: Patch Available (was: Open) > "show tables like" support for SQL wildcard characters (% and _) > > > Key: HIVE-23359 > URL: https://issues.apache.org/jira/browse/HIVE-23359 > Project: Hive > Issue Type: Improvement > Components: SQL >Affects Versions: 2.3.7 >Reporter: Yuming Wang >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23359.01.patch > > > https://docs.snowflake.com/en/sql-reference/sql/show-tables.html > https://clickhouse.tech/docs/en/sql-reference/statements/show/ > https://www.mysqltutorial.org/mysql-show-tables/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23266) Remove QueryWrapper from ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099368#comment-17099368
]
Hive QA commented on HIVE-23266:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002000/HIVE-23266.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17203 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sharedwork]
(batchId=106)
org.apache.hadoop.hive.metastore.tools.metatool.TestHiveMetaTool.testExecuteJDOQL
(batchId=173)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22125/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22125/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22125/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002000 - PreCommit-HIVE-Build
> Remove QueryWrapper from ObjectStore
>
>
> Key: HIVE-23266
> URL: https://issues.apache.org/jira/browse/HIVE-23266
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23266.1.patch, HIVE-23266.2.patch,
> HIVE-23266.2.patch, HIVE-23266.3.patch, HIVE-23266.4.patch,
> HIVE-23266.5.patch, HIVE-23266.6.patch, HIVE-23266.7.patch,
> HIVE-23266.8.patch, HIVE-23266.9.patch
>
>
> There is currently a utility called {{QueryWrapper}} that makes a normal
> {{Query}} auto-closable. However, {{Query}} is now in fact already
> auto-closing, so there is no need for this class. In trying to remove it, I
> realized that this wrapper was being passed around in pretty convoluted ways
> and also it was sometimes being created in a {{try-with-resources}} block but
> then never actually used in any way.
> Remove the {{QueryWrapper}} from the class and simplify some of the DB
> interactions.
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L178
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[ https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23301: -- Attachment: hive-23301.5.patch Status: Patch Available (was: In Progress) > Optimising privilege synchroniser: UDF for updating privileges > -- > > Key: HIVE-23301 > URL: https://issues.apache.org/jira/browse/HIVE-23301 > Project: Hive > Issue Type: Improvement > Components: Metastore, UDF >Affects Versions: 3.1.1 >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch, > hive-23301-2.patch, hive-23301.3.patch, hive-23301.4.patch, hive-23301.5.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > The current UDF Maps a bit string to a privilege based on if the privilege > is granted or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[ https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23301: -- Status: In Progress (was: Patch Available) > Optimising privilege synchroniser: UDF for updating privileges > -- > > Key: HIVE-23301 > URL: https://issues.apache.org/jira/browse/HIVE-23301 > Project: Hive > Issue Type: Improvement > Components: Metastore, UDF >Affects Versions: 3.1.1 >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch, > hive-23301-2.patch, hive-23301.3.patch, hive-23301.4.patch, hive-23301.5.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > The current UDF Maps a bit string to a privilege based on if the privilege > is granted or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23266) Remove QueryWrapper from ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099348#comment-17099348
]
Hive QA commented on HIVE-23266:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 29 new + 277 unchanged - 28 fixed = 306 total (was 305) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
18s{color} | {color:red} standalone-metastore/metastore-server generated 2 new
+ 186 unchanged - 1 fixed = 188 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Impossible cast from String to
org.apache.hadoop.hive.metastore.model.MPartition in
org.apache.hadoop.hive.metastore.ObjectStore.listPartitionsPsWithAuth(String,
String, String, List, short, String, List) At
ObjectStore.java:org.apache.hadoop.hive.metastore.model.MPartition in
org.apache.hadoop.hive.metastore.ObjectStore.listPartitionsPsWithAuth(String,
String, String, List, short, String, List) At ObjectStore.java:[line 3175] |
| |
org.apache.hadoop.hive.metastore.ObjectStore.getMPartitionColumnStatistics(Table,
List, List, String) concatenates strings using + in a loop At
ObjectStore.java:strings using + in a loop At ObjectStore.java:[line 9128] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22125/dev-support/hive-personality.sh
|
| git revision | master / 5a119b9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22125/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22125/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22125/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove QueryWrapper from ObjectStore
>
>
> Key: HIVE-23266
> URL: https://issues.apache.org/jira/browse/HIVE-23266
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23266.1.patch
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[
https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099340#comment-17099340
]
Hive QA commented on HIVE-23340:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001997/HIVE-23340.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17202 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_insert_partitioned]
(batchId=94)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sharedwork]
(batchId=106)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22124/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22124/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22124/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001997 - PreCommit-HIVE-Build
> TxnHandler cleanup
> --
>
> Key: HIVE-23340
> URL: https://issues.apache.org/jira/browse/HIVE-23340
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch,
> HIVE-23340.3.patch
>
>
> * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication
> * Remove TxnStatus character constants and use the enum values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-22259) Rewriting fails for `BETWEEN` clauses with different ranges in MV and query
[ https://issues.apache.org/jira/browse/HIVE-22259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-22259. Fix Version/s: 4.0.0 Resolution: Fixed Fixed in HIVE-22256. > Rewriting fails for `BETWEEN` clauses with different ranges in MV and query > --- > > Key: HIVE-22259 > URL: https://issues.apache.org/jira/browse/HIVE-22259 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Reporter: Steve Carlin >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0 > > Attachments: expr5.sql > > > Script attached. > The following query does not rewrite: > create materialized view view9 stored as orc as (select prod_id, cust_id, > store_id, sale_date, qty, amt, descr from sales where cust_id >= 1 and > prod_id < 31); > > -- this is not ok > explain extended select * from sales where cust_id between 1 and 20 and > prod_id < 31; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22256) Rewriting fails when `IN` clause has items in different order in MV and query
[ https://issues.apache.org/jira/browse/HIVE-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22256: --- Summary: Rewriting fails when `IN` clause has items in different order in MV and query (was: Rewriting fails when `IN` clause has items in different order in MV and query.) > Rewriting fails when `IN` clause has items in different order in MV and query > - > > Key: HIVE-22256 > URL: https://issues.apache.org/jira/browse/HIVE-22256 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22256.01.patch, HIVE-22256.patch, expr2.sql > > Time Spent: 50m > Remaining Estimate: 0h > > Rewriting fails on following materialized view and query (script is also > attached): > create materialized view view2 stored as orc as (select prod_id, cust_id, > store_id, sale_date, qty, amt, descr from sales where cust_id in (1,2,3,4,5)); > explain extended select prod_id, cust_id from sales where cust_id in > (5,1,2,3,4); -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22256) Rewriting fails when `IN` clause has items in different order in MV and query.
[ https://issues.apache.org/jira/browse/HIVE-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22256: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~vgarg]. > Rewriting fails when `IN` clause has items in different order in MV and query. > -- > > Key: HIVE-22256 > URL: https://issues.apache.org/jira/browse/HIVE-22256 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22256.01.patch, HIVE-22256.patch, expr2.sql > > Time Spent: 50m > Remaining Estimate: 0h > > Rewriting fails on following materialized view and query (script is also > attached): > create materialized view view2 stored as orc as (select prod_id, cust_id, > store_id, sale_date, qty, amt, descr from sales where cust_id in (1,2,3,4,5)); > explain extended select prod_id, cust_id from sales where cust_id in > (5,1,2,3,4); -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-22258) Rewriting fails for `IN` clauses in MV and query when we use equals or subset in the query
[ https://issues.apache.org/jira/browse/HIVE-22258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-22258. Fix Version/s: 4.0.0 Resolution: Fixed Fixed in HIVE-22256. > Rewriting fails for `IN` clauses in MV and query when we use equals or subset > in the query > -- > > Key: HIVE-22258 > URL: https://issues.apache.org/jira/browse/HIVE-22258 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0 > > Attachments: in-pred.sql > > > ...as stated in title. Script is attached. The issue can be seen with these > queries: > > create materialized view av1 stored as orc as select state, year, > sum(population) from census_pop where year IN (2010, 2018) group by state, > year; > -- this is ok > explain extended select state, year, sum(population) from census_pop where > year IN (2010, 2018) group by state, year; > -- this is not ok > explain extended select state, year, sum(population) from census_pop where > year = 2010 group by state, year; > -- this is not ok > explain extended select state, year, sum(population) from census_pop where > year in (2010) group by state, year; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[
https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099297#comment-17099297
]
Hive QA commented on HIVE-23340:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 543 unchanged - 9 fixed = 543 total (was 552) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 186 unchanged - 1 fixed = 187 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxnsList(boolean)
passes a nonconstant String to an execute or addBatch method on an SQL
statement At TxnHandler.java:to an execute or addBatch method on an SQL
statement At TxnHandler.java:[line 556] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22124/dev-support/hive-personality.sh
|
| git revision | master / c75412f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22124/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22124/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TxnHandler cleanup
> --
>
> Key: HIVE-23340
> URL: https://issues.apache.org/jira/browse/HIVE-23340
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch,
> HIVE-23340.3.patch
>
>
> * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication
> * Remove TxnStatus character constants and use the enum values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-14594) CBO: Calcite Operator To Hive Operator(Calcite Return Path): Fix wrong result in join_filters_overlap
[
https://issues.apache.org/jira/browse/HIVE-14594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099296#comment-17099296
]
Vineet Garg commented on HIVE-14594:
Actually CALCITE issue is added as related so I guess we can keep it around.
> CBO: Calcite Operator To Hive Operator(Calcite Return Path): Fix wrong result
> in join_filters_overlap
> -
>
> Key: HIVE-14594
> URL: https://issues.apache.org/jira/browse/HIVE-14594
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> Setup
> {code} create table a as SELECT 100 as key, a.value as value FROM src LATERAL
> VIEW explode(array(40, 50, 60)) a as value limit 3; {code}
> Query:
> {code} select * from a left outer join a b on (a.key=b.key AND a.value=50 AND
> b.value=50) left outer join a c on (a.key=c.key AND a.value=60 AND
> c.value=60); {code}
> Expected Result:
> {code}
> 100 40 NULLNULLNULLNULL
> 100 50 100 50 NULLNULL
> 100 60 NULLNULL100 60
> {code}
> Actual Result:
> {code}
> 100 40 NULL50 NULL60
> 100 50 100 50 NULL60
> 100 60 NULL50 100 60
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-14594) CBO: Calcite Operator To Hive Operator(Calcite Return Path): Fix wrong result in join_filters_overlap
[
https://issues.apache.org/jira/browse/HIVE-14594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099295#comment-17099295
]
Vineet Garg commented on HIVE-14594:
[~zabetak] I just re-ran this test and I see the the query is not failing at
runtime. It looks like it is a separate issue. Since Calcite issue has been
fixed and released i will remove the dependency but i will keep the jira open.
> CBO: Calcite Operator To Hive Operator(Calcite Return Path): Fix wrong result
> in join_filters_overlap
> -
>
> Key: HIVE-14594
> URL: https://issues.apache.org/jira/browse/HIVE-14594
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> Setup
> {code} create table a as SELECT 100 as key, a.value as value FROM src LATERAL
> VIEW explode(array(40, 50, 60)) a as value limit 3; {code}
> Query:
> {code} select * from a left outer join a b on (a.key=b.key AND a.value=50 AND
> b.value=50) left outer join a c on (a.key=c.key AND a.value=60 AND
> c.value=60); {code}
> Expected Result:
> {code}
> 100 40 NULLNULLNULLNULL
> 100 50 100 50 NULLNULL
> 100 60 NULLNULL100 60
> {code}
> Actual Result:
> {code}
> 100 40 NULL50 NULL60
> 100 50 100 50 NULL60
> 100 60 NULL50 100 60
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22256) Rewriting fails when `IN` clause has items in different order in MV and query.
[ https://issues.apache.org/jira/browse/HIVE-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099291#comment-17099291 ] Vineet Garg commented on HIVE-22256: +1 > Rewriting fails when `IN` clause has items in different order in MV and query. > -- > > Key: HIVE-22256 > URL: https://issues.apache.org/jira/browse/HIVE-22256 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22256.01.patch, HIVE-22256.patch, expr2.sql > > Time Spent: 50m > Remaining Estimate: 0h > > Rewriting fails on following materialized view and query (script is also > attached): > create materialized view view2 stored as orc as (select prod_id, cust_id, > store_id, sale_date, qty, amt, descr from sales where cust_id in (1,2,3,4,5)); > explain extended select prod_id, cust_id from sales where cust_id in > (5,1,2,3,4); -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23323) Add qsplits profile
[
https://issues.apache.org/jira/browse/HIVE-23323?focusedWorklogId=430375&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430375
]
ASF GitHub Bot logged work on HIVE-23323:
-
Author: ASF GitHub Bot
Created on: 04/May/20 19:39
Start Date: 04/May/20 19:39
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1000:
URL: https://github.com/apache/hive/pull/1000#discussion_r419670022
##
File path:
itests/util/src/test/java/org/apache/hadoop/hive/cli/control/TestSplitSupport.java
##
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.cli.control;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import org.junit.Test;
+
+import com.google.common.collect.Sets;
+
+public class TestSplitSupport {
+
+ @Test
+ public void testSplitParams() {
Review comment:
removed it
##
File path: itests/qtest-spark/pom.xml
##
@@ -417,5 +417,53 @@
-
+
Review comment:
I don't know why my editor switched to 4 spaces...
##
File path: itests/bin/generate-cli-splits.sh
##
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+usage() {
+ echo "$0 "
+ exit 1
+}
+
+[ "$1" == "" ] && usage
+[ "$2" == "" ] && usage
+
+
+inDir="$1"
+outDir="$2"
+
+git grep SplitSupport.process | grep "$1" | cut -d ':' -f1|while read f;do
Review comment:
yes the number of batches will be fixed; and it will have nothing to do
with the number of batches in a test suite.
I've considered a highly sophisticated one prior to ending up with this
solution (which was also inherently complex as well)...after some time I
concluded that actually there is no big gain from making it "very good"...if
all testclasses will be below say 30 minutes it will be good enough.
So the goal of this thing is not to make the perfect sized split; but to
make them run under say N minutes; when I've calibrated the actual split counts
I tried to go for ~15 minutes so it will take some time when it will needs
adjusting from time-to-time.
About isolated tests: this patch doesn't tries to address that - right now I
think the option to put tests into "isolation" will not be supported anymore.
The best would be to fix them but putting them on ignore could be also an
option...
Right now I don't see any easy way to define "isolated"
suites...howeverit could be probably forced into this splits logic in some
way...for now I think let's fix or ignore them - later we can get back to this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 430375)
Time Spent: 50m (was: 40m)
> Add qsplits profile
> ---
>
> Key: HIVE-23323
> URL: https://issues.apache.org/jira/browse/HIVE-23323
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23323.01.patch, HIVE-23323.02.patch,
> HIVE-23323.02.patch, HIVE-23323.02.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> provide an in-maven way to split-up big testclasses into smaller ones
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23349:
--
Attachment: HIVE-23349.2.patch
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23349.1.patch, HIVE-23349.2.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099279#comment-17099279
]
Hive QA commented on HIVE-23349:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002014/HIVE-23349.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 17207 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestNewGetSplitsFormat.testComplexQuery (batchId=214)
org.apache.hive.jdbc.TestNewGetSplitsFormat.testDataTypes (batchId=214)
org.apache.hive.jdbc.TestNewGetSplitsFormat.testEscapedStrings (batchId=214)
org.apache.hive.jdbc.TestNewGetSplitsFormat.testInvalidReferenceCountScenario
(batchId=214)
org.apache.hive.jdbc.TestNewGetSplitsFormat.testKillQuery (batchId=214)
org.apache.hive.jdbc.TestNewGetSplitsFormat.testLlapInputFormatEndToEnd
(batchId=214)
org.apache.hive.jdbc.TestNewGetSplitsFormat.testLlapInputFormatEndToEndWithMultipleBatches
(batchId=214)
org.apache.hive.jdbc.TestNewGetSplitsFormat.testMultipleBatchesOfComplexTypes
(batchId=214)
org.apache.hive.jdbc.TestNewGetSplitsFormat.testNonAsciiStrings (batchId=214)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22123/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22123/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22123/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002014 - PreCommit-HIVE-Build
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23349.1.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23206) Project not defined correctly after reordering a join
[ https://issues.apache.org/jira/browse/HIVE-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-23206: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~kkasa] > Project not defined correctly after reordering a join > - > > Key: HIVE-23206 > URL: https://issues.apache.org/jira/browse/HIVE-23206 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Steve Carlin >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23206.1.patch, HIVE-23206.2.patch, > HIVE-23206.3.patch, HIVE-23206.4.patch, HIVE-23206.5.patch, HIVE-23206.6.patch > > > The following highlighted line seems to be incorrect in the test suite: > [https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/perf/tez/cbo_query95.q.out#L89] > Note that the project takes all the columns from the table scan, yet it only > needs a couple of them. > I did some very small debugging on this. When I removed the > applyJoinOrderingTransform here: > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1897] > ... the problem goes away. So presumably one of the rules in there is > causing the problem. > Here is a slightly simplified version of the query which has the same problem > (using the same tpc-ds database): > explain cbo with ws_wh as > (select ws1.ws_order_number > from web_sales ws1,web_returns wr2 > where ws1.ws_order_number = wr2.wr_order_number) > select > ws_order_number > from > web_sales ws1 > where > ws1.ws_order_number in (select wr_order_number > from web_returns,ws_wh > where wr_order_number = ws_wh.ws_order_number) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23206) Project not defined correctly after reordering a join
[ https://issues.apache.org/jira/browse/HIVE-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099276#comment-17099276 ] Vineet Garg commented on HIVE-23206: +1 > Project not defined correctly after reordering a join > - > > Key: HIVE-23206 > URL: https://issues.apache.org/jira/browse/HIVE-23206 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Steve Carlin >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-23206.1.patch, HIVE-23206.2.patch, > HIVE-23206.3.patch, HIVE-23206.4.patch, HIVE-23206.5.patch, HIVE-23206.6.patch > > > The following highlighted line seems to be incorrect in the test suite: > [https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/perf/tez/cbo_query95.q.out#L89] > Note that the project takes all the columns from the table scan, yet it only > needs a couple of them. > I did some very small debugging on this. When I removed the > applyJoinOrderingTransform here: > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1897] > ... the problem goes away. So presumably one of the rules in there is > causing the problem. > Here is a slightly simplified version of the query which has the same problem > (using the same tpc-ds database): > explain cbo with ws_wh as > (select ws1.ws_order_number > from web_sales ws1,web_returns wr2 > where ws1.ws_order_number = wr2.wr_order_number) > select > ws_order_number > from > web_sales ws1 > where > ws1.ws_order_number in (select wr_order_number > from web_returns,ws_wh > where wr_order_number = ws_wh.ws_order_number) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099253#comment-17099253
]
Hive QA commented on HIVE-23349:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22123/dev-support/hive-personality.sh
|
| git revision | master / c34ee9d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22123/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23349.1.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099221#comment-17099221
]
Hive QA commented on HIVE-23282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001991/HIVE-23282.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17201 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22122/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22122/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22122/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001991 - PreCommit-HIVE-Build
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch,
> HIVE-23282.5.patch, HIVE-23282.6.patch, image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information
> for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew
> column, parameters, sort, bucket cols etc). This makes it a lot more
> expensive call. It would be good to check if MTable itself can be used
> instead of Table.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669]
>
> !image-2020-04-23-14-07-06-077.png|width=665,height=592!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23364) Push down null-safe cast-s
[
https://issues.apache.org/jira/browse/HIVE-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099189#comment-17099189
]
Stamatis Zampetakis commented on HIVE-23364:
Do you mean extracting the computation {{CAST(_col AS STRING)}} into a
separate projection and pushing the projection further down the plan?
> Push down null-safe cast-s
> ---
>
> Key: HIVE-23364
> URL: https://issues.apache.org/jira/browse/HIVE-23364
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Priority: Major
>
> right now we may endup with conditionals like this:
> {code}
> predicate: CASE WHEN (_col1 is not null) THEN (CASE WHEN
> (_col4 is not null) THEN ((CAST( _col4 AS STRING) = CAST( _col1 AS STRING)))
> ELSE (('EMPTY' = CAST( _col1 AS STRING))) END) ELSE (CASE WHEN (_col4 is not
> null) THEN ((CAST( _col4 AS STRING) = 'EMPTY')) ELSE (('EMPTY' = 'EMPTY'))
> END) END (type: boolean)
> {code}
> if this is after a join operator; then the same CAST might be evaluated for
> the same value...
> in case the CAST is to a type which doesn't widen nullabiltity in Hive - we
> may push down the CAST
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099177#comment-17099177
]
Hive QA commented on HIVE-23282:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
11s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 21 new + 598 unchanged - 6 fixed = 619 total (was 604) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 187 unchanged - 0 fixed = 188 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Possible null pointer dereference of null in
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilterInternal(String,
String, String, String, short, boolean, boolean) Dereferenced at
ObjectStore.java:null in
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilterInternal(String,
String, String, String, short, boolean, boolean) Dereferenced at
ObjectStore.java:[line 4077] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22122/dev-support/hive-personality.sh
|
| git revision | master / c34ee9d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22122/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22122/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22122/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-2
[jira] [Work logged] (HIVE-23323) Add qsplits profile
[
https://issues.apache.org/jira/browse/HIVE-23323?focusedWorklogId=430329&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430329
]
ASF GitHub Bot logged work on HIVE-23323:
-
Author: ASF GitHub Bot
Created on: 04/May/20 17:46
Start Date: 04/May/20 17:46
Worklog Time Spent: 10m
Work Description: miklosgergely commented on a change in pull request
#1000:
URL: https://github.com/apache/hive/pull/1000#discussion_r419613747
##
File path: itests/bin/generate-cli-splits.sh
##
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+usage() {
+ echo "$0 "
+ exit 1
+}
+
+[ "$1" == "" ] && usage
+[ "$2" == "" ] && usage
+
+
+inDir="$1"
+outDir="$2"
+
+git grep SplitSupport.process | grep "$1" | cut -d ':' -f1|while read f;do
Review comment:
As I see in this model the number of batches would be fixed, not the
number of tests in a batch. I see why is this useful for generating the files,
but it would be better to fix the batch size.
How the isolated tests are going to be handled?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 430329)
Time Spent: 40m (was: 0.5h)
> Add qsplits profile
> ---
>
> Key: HIVE-23323
> URL: https://issues.apache.org/jira/browse/HIVE-23323
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23323.01.patch, HIVE-23323.02.patch,
> HIVE-23323.02.patch, HIVE-23323.02.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> provide an in-maven way to split-up big testclasses into smaller ones
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23323) Add qsplits profile
[
https://issues.apache.org/jira/browse/HIVE-23323?focusedWorklogId=430327&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430327
]
ASF GitHub Bot logged work on HIVE-23323:
-
Author: ASF GitHub Bot
Created on: 04/May/20 17:44
Start Date: 04/May/20 17:44
Worklog Time Spent: 10m
Work Description: miklosgergely commented on a change in pull request
#1000:
URL: https://github.com/apache/hive/pull/1000#discussion_r419610881
##
File path: itests/qtest-spark/pom.xml
##
@@ -417,5 +417,53 @@
-
+
Review comment:
Please use 2 spaces as indentation, to keep consistency
##
File path: itests/qtest/pom.xml
##
@@ -426,15 +426,62 @@
test
-
-
-
- minimr
-
-
-
-
-
+
Review comment:
Please use 2 spaces as indentation, to keep consistency
##
File path:
itests/util/src/test/java/org/apache/hadoop/hive/cli/control/TestSplitSupport.java
##
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.cli.control;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import org.junit.Test;
+
+import com.google.common.collect.Sets;
+
+public class TestSplitSupport {
+
+ @Test
+ public void testSplitParams() {
Review comment:
What is this test for?
##
File path: itests/bin/generate-cli-splits.sh
##
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+usage() {
+ echo "$0 "
+ exit 1
+}
+
+[ "$1" == "" ] && usage
+[ "$2" == "" ] && usage
+
+
+inDir="$1"
+outDir="$2"
+
+git grep SplitSupport.process | grep "$1" | cut -d ':' -f1|while read f;do
Review comment:
Nit: Please be consistent about putting two spaces next to the |
characters when they are used as pipeline separators.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 430327)
Time Spent: 0.5h (was: 20m)
> Add qsplits profile
> ---
>
> Key: HIVE-23323
> URL: https://issues.apache.org/jira/browse/HIVE-23323
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23323.01.patch, HIVE-23323.02.patch,
> HIVE-23323.02.patch, HIVE-23323.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> provide an in-maven way to split-up big testclasses into smaller ones
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23325) Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23325: -- Attachment: HIVE-23325.5.patch > Clean up cleanup tasks for TxnHandler/CompactionTxnHandler > -- > > Key: HIVE-23325 > URL: https://issues.apache.org/jira/browse/HIVE-23325 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch, > HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch, > HIVE-23325.4.patch, HIVE-23325.5.patch > > > Currently there are multiple ways to clean up ACID related stuff: > * AcidHouseKeeperService > * AcidWriteSetService > * AcidCompactionHistoryService > * Initiator > * Cleaner > * etc. > We should consolidate them where possible and improve logging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23280) Trigger compaction with old aborted txns
[
https://issues.apache.org/jira/browse/HIVE-23280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099144#comment-17099144
]
Hive QA commented on HIVE-23280:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001994/HIVE-23280.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17175 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances
(batchId=194)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22121/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22121/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22121/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001994 - PreCommit-HIVE-Build
> Trigger compaction with old aborted txns
>
>
> Key: HIVE-23280
> URL: https://issues.apache.org/jira/browse/HIVE-23280
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23280.01.patch, HIVE-23280.01.patch,
> HIVE-23280.01.patch, HIVE-23280.01.patch, HIVE-23280.02.patch,
> HIVE-23280.03.patch, HIVE-23280.04.patch, HIVE-23280.04.patch
>
>
> When a txn is aborted and the compaction threshold for number of aborted txns
> is not reached then the aborted transaction can remain forever in the RDBMS
> database. This could result in several serious performance degradations:
> - getOpenTxns has to list this aborted txn forever
> - TXN_TO_WRITE_ID table is not cleaned
> We should add a threshold, so after a given time the compaction is started
> anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23280) Trigger compaction with old aborted txns
[
https://issues.apache.org/jira/browse/HIVE-23280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099123#comment-17099123
]
Hive QA commented on HIVE-23280:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
26s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
9s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 2 new + 105 unchanged - 3 fixed = 107 total (was 108) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
22s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 186 unchanged - 1 fixed = 187 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findPotentialCompactions(int,
long, long) passes a nonconstant String to an execute or addBatch method on an
SQL statement At CompactionTxnHandler.java:nonconstant String to an execute or
addBatch method on an SQL statement At CompactionTxnHandler.java:[line 97] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22121/dev-support/hive-personality.sh
|
| git revision | master / c34ee9d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22121/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22121/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.2
[jira] [Updated] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23349:
--
Attachment: HIVE-23349.1.patch
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23349.1.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23349:
--
Attachment: (was: HIVE-23349.1.patch)
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23364) Push down null-safe cast-s
[
https://issues.apache.org/jira/browse/HIVE-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099085#comment-17099085
]
Zoltan Haindrich commented on HIVE-23364:
-
nullsafe in the above sense means:
{code}
(c IS NULL ) iff c IS NULL
{code}
> Push down null-safe cast-s
> ---
>
> Key: HIVE-23364
> URL: https://issues.apache.org/jira/browse/HIVE-23364
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Priority: Major
>
> right now we may endup with conditionals like this:
> {code}
> predicate: CASE WHEN (_col1 is not null) THEN (CASE WHEN
> (_col4 is not null) THEN ((CAST( _col4 AS STRING) = CAST( _col1 AS STRING)))
> ELSE (('EMPTY' = CAST( _col1 AS STRING))) END) ELSE (CASE WHEN (_col4 is not
> null) THEN ((CAST( _col4 AS STRING) = 'EMPTY')) ELSE (('EMPTY' = 'EMPTY'))
> END) END (type: boolean)
> {code}
> if this is after a join operator; then the same CAST might be evaluated for
> the same value...
> in case the CAST is to a type which doesn't widen nullabiltity in Hive - we
> may push down the CAST
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23364) Push down null-safe cast-s
[
https://issues.apache.org/jira/browse/HIVE-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099084#comment-17099084
]
Zoltan Haindrich commented on HIVE-23364:
-
I think this could be done for:
* string casts
* integer type widening
> Push down null-safe cast-s
> ---
>
> Key: HIVE-23364
> URL: https://issues.apache.org/jira/browse/HIVE-23364
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Priority: Major
>
> right now we may endup with conditionals like this:
> {code}
> predicate: CASE WHEN (_col1 is not null) THEN (CASE WHEN
> (_col4 is not null) THEN ((CAST( _col4 AS STRING) = CAST( _col1 AS STRING)))
> ELSE (('EMPTY' = CAST( _col1 AS STRING))) END) ELSE (CASE WHEN (_col4 is not
> null) THEN ((CAST( _col4 AS STRING) = 'EMPTY')) ELSE (('EMPTY' = 'EMPTY'))
> END) END (type: boolean)
> {code}
> if this is after a join operator; then the same CAST might be evaluated for
> the same value...
> in case the CAST is to a type which doesn't widen nullabiltity in Hive - we
> may push down the CAST
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23361) Optimising privilege synchroniser
[
https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099076#comment-17099076
]
Hive QA commented on HIVE-23361:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001993/hive-23361.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 17201 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=89)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_analyze]
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_ingest]
(batchId=99)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_materialized]
(batchId=95)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables_sysdb]
(batchId=91)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb_schq]
(batchId=92)
org.apache.hive.service.server.TestInformationSchemaWithPrivilegeZookeeperPlain.test
(batchId=211)
org.apache.hive.service.server.TestInformationSchemaWithPrivilegeZookeeperSSL.test
(batchId=210)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22120/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22120/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22120/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001993 - PreCommit-HIVE-Build
> Optimising privilege synchroniser
> -
>
> Key: HIVE-23361
> URL: https://issues.apache.org/jira/browse/HIVE-23361
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Simhadri G
>Priority: Minor
> Attachments: hive-23361.patch, hive-23361.patch
>
>
> Privilege synchronizer pulls the list of databases, tables and columns from
> the Hive Metastore. For each of these objects it fetches the privilege
> information and invokes HMS API to refresh the privilege information in HMS.
> This patch store the privilege information as bit string. This is done to
> reduce the size of the tbl_col_privs tables in metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[
https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099072#comment-17099072
]
Adesh Kumar Rao commented on HIVE-23347:
[~srahman]
In your example:
{noformat}
partition from metastore: /year=2020/month=3/day=2;
partition from fileSystem: /Year=2020/Month=3/Day=2;
{noformat}
The partition from metastore is wrong. It will contain:
"tablepath/Year=2020/Month=3/Day=2" (the actual hdfs path)
because partition path fetched is being fetched from metastore (the partition
name is "year=2020/month=3/day=2", but the path will be
"tablepath/Year=2020/Month=3/Day=2")
Links for how the partPaths variable is populated:
[Path partPath = getDataLocation(table,
partition);|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java#L316]
and
[getDataLocation.|https://github.com/apache/hive/blob/c34ee9d79bf6931a92b61af98a6c8f09c6b9ad73/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java#L1374]
> MSCK REPAIR cannot discover partitions with upper case directory names.
> ---
>
> Key: HIVE-23347
> URL: https://issues.apache.org/jira/browse/HIVE-23347
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch,
> HIVE-23347.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For the following scenario, we expect MSCK REPAIR to discover partitions but
> it couldn't.
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1; --> Returns zero partitions
> 5. select * from t1; --> Returns empty data.
> When the partition directory names are changed to lower case, this works fine.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23314) Upgrade to Kudu 1.12
[ https://issues.apache.org/jira/browse/HIVE-23314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099062#comment-17099062 ] Zoltan Haindrich commented on HIVE-23314: - https://mail-archives.apache.org/mod_mbox/kudu-dev/202005.mbox/browser > Upgrade to Kudu 1.12 > > > Key: HIVE-23314 > URL: https://issues.apache.org/jira/browse/HIVE-23314 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Priority: Major > > we need KUDU-3044 because it could cause random failures... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21304) Make bucketing version usage more robust
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you Jesus for reviewing the changes! > Make bucketing version usage more robust > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch, > HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch, > HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch, > HIVE-21304.21.patch, HIVE-21304.22.patch, HIVE-21304.23.patch, > HIVE-21304.24.patch, HIVE-21304.25.patch, HIVE-21304.26.patch, > HIVE-21304.27.patch, HIVE-21304.28.patch, HIVE-21304.29.patch, > HIVE-21304.30.patch, HIVE-21304.31.patch, HIVE-21304.32.patch, > HIVE-21304.33.patch, HIVE-21304.33.patch, HIVE-21304.33.patch, > HIVE-21304.34.patch, HIVE-21304.34.patch, HIVE-21304.35.patch, > HIVE-21304.35.patch, HIVE-21304.36.patch, HIVE-21304.37.patch, > HIVE-21304.38.patch, HIVE-21304.38.patch, HIVE-21304.38.patch, > HIVE-21304.39.patch, HIVE-21304.40.patch > > Time Spent: 50m > Remaining Estimate: 0h > > * Show Bucketing version for ReduceSinkOp in explain extended plan - this > helps identify what hashing algorithm is being used by by ReduceSinkOp. > * move the actually selected version to the "conf" so that it doesn't get lost > * replace trait related logic with a separate optimizer rule > * do version selection based on a group of operator - this is more reliable > * skip bucketingversion selection for tables with 1 buckets > * prefer to use version 2 if possible > * fix operator creations which didn't set a new conf -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23361) Optimising privilege synchroniser
[
https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099043#comment-17099043
]
Hive QA commented on HIVE-23361:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
35s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 6 new + 0 unchanged - 0 fixed
= 6 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22120/dev-support/hive-personality.sh
|
| git revision | master / c34ee9d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22120/yetus/diff-checkstyle-ql.txt
|
| modules | C: metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22120/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimising privilege synchroniser
> -
>
> Key: HIVE-23361
> URL: https://issues.apache.org/jira/browse/HIVE-23361
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Simhadri G
>Priority: Minor
> Attachments: hive-23361.patch, hive-23361.patch
>
>
> Privilege synchronizer pulls the list of databases, tables and columns from
> the Hive Metastore. For each of these objects it fetches the privilege
> information and invokes HMS API to refresh the privilege information in HMS.
> This patch store the privilege information as bit string. This is done to
> reduce the size of the tbl_col_privs tables in metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[
https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099010#comment-17099010
]
Hive QA commented on HIVE-23347:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001986/HIVE-23347.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17201 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22119/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22119/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22119/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001986 - PreCommit-HIVE-Build
> MSCK REPAIR cannot discover partitions with upper case directory names.
> ---
>
> Key: HIVE-23347
> URL: https://issues.apache.org/jira/browse/HIVE-23347
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch,
> HIVE-23347.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For the following scenario, we expect MSCK REPAIR to discover partitions but
> it couldn't.
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1; --> Returns zero partitions
> 5. select * from t1; --> Returns empty data.
> When the partition directory names are changed to lower case, this works fine.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23359) "show tables like" support for SQL wildcard characters (% and _)
[ https://issues.apache.org/jira/browse/HIVE-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23359: -- Attachment: HIVE-23359.01.patch > "show tables like" support for SQL wildcard characters (% and _) > > > Key: HIVE-23359 > URL: https://issues.apache.org/jira/browse/HIVE-23359 > Project: Hive > Issue Type: Improvement > Components: SQL >Affects Versions: 2.3.7 >Reporter: Yuming Wang >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23359.01.patch > > > https://docs.snowflake.com/en/sql-reference/sql/show-tables.html > https://clickhouse.tech/docs/en/sql-reference/statements/show/ > https://www.mysqltutorial.org/mysql-show-tables/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22942) Replace PTest with an alternative
[ https://issues.apache.org/jira/browse/HIVE-22942?focusedWorklogId=430231&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430231 ] ASF GitHub Bot logged work on HIVE-22942: - Author: ASF GitHub Bot Created on: 04/May/20 14:36 Start Date: 04/May/20 14:36 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #948: URL: https://github.com/apache/hive/pull/948#issuecomment-623502273 another comment This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 430231) Time Spent: 0.5h (was: 20m) > Replace PTest with an alternative > - > > Key: HIVE-22942 > URL: https://issues.apache.org/jira/browse/HIVE-22942 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > I never opened a jira about this...but it might actually help collect ideas > and actually start going somewhere sooner than later :D > Right now we maintain the ptest2 project inside Hive to be able to run Hive > tests in a distributed fashion...the backstab of this solution is that we are > putting much effort into maintaining a distributed test execution framework... > I think it would be better if we could find an off the shelf solution for the > task and migrate to that instead of putting more efforts into the ptest > framework -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098985#comment-17098985 ] John Sherman commented on HIVE-23354: - [~kuczoram] Thanks for the feedback. I'll see if I can come up with a better solution. This is exactly why I didn't rush this through the commit process since there are so many cases to consider. I definitely don't want to introduce any regressions here since it is such a relied on part. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23325) Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23325: -- Attachment: HIVE-23325.4.patch > Clean up cleanup tasks for TxnHandler/CompactionTxnHandler > -- > > Key: HIVE-23325 > URL: https://issues.apache.org/jira/browse/HIVE-23325 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch, > HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch, HIVE-23325.4.patch > > > Currently there are multiple ways to clean up ACID related stuff: > * AcidHouseKeeperService > * AcidWriteSetService > * AcidCompactionHistoryService > * Initiator > * Cleaner > * etc. > We should consolidate them where possible and improve logging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[
https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098984#comment-17098984
]
Hive QA commented on HIVE-23347:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
24s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22119/dev-support/hive-personality.sh
|
| git revision | master / 342f8fb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22119/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MSCK REPAIR cannot discover partitions with upper case directory names.
> ---
>
> Key: HIVE-23347
> URL: https://issues.apache.org/jira/browse/HIVE-23347
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch,
> HIVE-23347.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For the following scenario, we expect MSCK REPAIR to discover partitions but
> it couldn't.
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11
> 2. create external
[jira] [Updated] (HIVE-23318) TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
[
https://issues.apache.org/jira/browse/HIVE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Bod updated HIVE-23318:
--
Attachment: HIVE-23318.1.patch
Status: Patch Available (was: Open)
> TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
> ---
>
> Key: HIVE-23318
> URL: https://issues.apache.org/jira/browse/HIVE-23318
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Marton Bod
>Priority: Minor
> Attachments: HIVE-23318.1.patch
>
>
> Observed the following queries when materialized view or any of its feature
> was not used.
> TxnHandler need not clear this part of txn commit. It would help in reducing
> the sql parsing time in server side as well.
> {noformat}
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398082
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398084
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398112
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22681) Replace Base64 in hcatalog-webhcat Package
[ https://issues.apache.org/jira/browse/HIVE-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22681: -- Attachment: HIVE-22681.1.patch > Replace Base64 in hcatalog-webhcat Package > -- > > Key: HIVE-22681 > URL: https://issues.apache.org/jira/browse/HIVE-22681 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22681.1.patch, HIVE-22681.1.patch, > HIVE-22681.1.patch, HIVE-22681.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23363: - Description: Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been retired: [http://www.datanucleus.org/documentation/products.html] > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.patch > > > Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been > retired: > [http://www.datanucleus.org/documentation/products.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22674) Replace Base64 in serde Package
[ https://issues.apache.org/jira/browse/HIVE-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22674: -- Attachment: HIVE-22674.2.patch > Replace Base64 in serde Package > --- > > Key: HIVE-22674 > URL: https://issues.apache.org/jira/browse/HIVE-22674 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22674.1.patch, HIVE-22674.2.patch, > HIVE-22674.2.patch, HIVE-22674.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23363 started by Zoltan Chovan. > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23363: - Attachment: HIVE-23363.patch > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-23363: Assignee: Zoltan Chovan > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23363: - Affects Version/s: 4.0.0 Status: Patch Available (was: In Progress) > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22674) Replace Base64 in serde Package
[ https://issues.apache.org/jira/browse/HIVE-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098959#comment-17098959 ] David Mollitor commented on HIVE-22674: --- [~kgyrtkirk] Thanks for the review! I don't know, but I have some plans of searching for all TODOs in the code and making JIRAs. I would like to look at making it configurable in a separate ticket. > Replace Base64 in serde Package > --- > > Key: HIVE-22674 > URL: https://issues.apache.org/jira/browse/HIVE-22674 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22674.1.patch, HIVE-22674.2.patch, > HIVE-22674.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23355) Fix quoting behavior when masking is enabled
[
https://issues.apache.org/jira/browse/HIVE-23355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098954#comment-17098954
]
David Mollitor commented on HIVE-23355:
---
[~jcamachorodriguez] and [~jfs] I think the problem extends beyond just this
one use case. Check out. [HIVE-23182]
> Fix quoting behavior when masking is enabled
>
>
> Key: HIVE-23355
> URL: https://issues.apache.org/jira/browse/HIVE-23355
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23355.1.patch
>
>
> Test framework repro:
> {code:java}
> --- a/ql/src/test/queries/clientpositive/masking_reserved.q
> +++ b/ql/src/test/queries/clientpositive/masking_reserved.q
> @@ -5,6 +5,7 @@ set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.autho
> create table keyword_test_off (id int, `etad` string, key int);
> create table keyword_test_on (id int, `date` string, key int);
> create table masking_test_n_masking_reserved (id int, value string, key int);
> +create temporary table masking_test_n_masking_temp stored as orc as select *
> from masking_test_n_masking_reserved; explain select a.`etad`, b.value from
> keyword_test_off a join masking_test_n_masking_reserved b on b.id = a.id;
> select a.`etad`, b.value from keyword_test_off a join
> masking_test_n_masking_reserved b on b.id = a.id; {code}
> fails with:
> {code:java}
> 2020-04-30T20:19:06,833 INFO [bba048b2-380a-46e2-b283-52c1090a0d5c main]
> parse.CalcitePlanner: Creating table default.masking_test_n_masking_temp
> position=23
> 2020-04-30T20:19:06,851 ERROR [bba048b2-380a-46e2-b283-52c1090a0d5c main]
> ql.Driver: FAILED: SemanticException Unrecognized file format in STORED AS
> clause: '`ORC`'
> org.apache.hadoop.hive.ql.parse.SemanticException: Unrecognized file format
> in STORED AS clause: '`ORC`'
> at
> org.apache.hadoop.hive.ql.parse.StorageFormat.processStorageFormat(StorageFormat.java:89)
> at
> org.apache.hadoop.hive.ql.parse.StorageFormat.fillStorageFormat(StorageFormat.java:74)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:13351)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12256)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12471)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:432)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:184) {code}
> We should either not quote the STORED BY argument in this path OR strip
> quotes in processStorageFormat.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23350) Upgrade DBCP To DBCP 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-23350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23350:
--
Status: Patch Available (was: Open)
> Upgrade DBCP To DBCP 2.7.0
> --
>
> Key: HIVE-23350
> URL: https://issues.apache.org/jira/browse/HIVE-23350
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23350.1.patch
>
>
> In {{hive-standalone-metastore-common}} the current version is 1.4 which is
> very antiquated and apparently (emphasis mine):
> bq. DBCP 1.4 compiles and runs under Java 6 *only* (JDBC 4)
> Hive is on Java 8
> {code:xml}
> 2.6.0
> 1.4
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23266) Remove QueryWrapper from ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23266:
--
Attachment: HIVE-23266.9.patch
> Remove QueryWrapper from ObjectStore
>
>
> Key: HIVE-23266
> URL: https://issues.apache.org/jira/browse/HIVE-23266
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23266.1.patch, HIVE-23266.2.patch,
> HIVE-23266.2.patch, HIVE-23266.3.patch, HIVE-23266.4.patch,
> HIVE-23266.5.patch, HIVE-23266.6.patch, HIVE-23266.7.patch,
> HIVE-23266.8.patch, HIVE-23266.9.patch
>
>
> There is currently a utility called {{QueryWrapper}} that makes a normal
> {{Query}} auto-closable. However, {{Query}} is now in fact already
> auto-closing, so there is no need for this class. In trying to remove it, I
> realized that this wrapper was being passed around in pretty convoluted ways
> and also it was sometimes being created in a {{try-with-resources}} block but
> then never actually used in any way.
> Remove the {{QueryWrapper}} from the class and simplify some of the DB
> interactions.
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L178
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23325) Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098945#comment-17098945
]
Hive QA commented on HIVE-23325:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001984/HIVE-23325.3.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17204 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22117/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22117/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22117/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001984 - PreCommit-HIVE-Build
> Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
> --
>
> Key: HIVE-23325
> URL: https://issues.apache.org/jira/browse/HIVE-23325
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch,
> HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch
>
>
> Currently there are multiple ways to clean up ACID related stuff:
> * AcidHouseKeeperService
> * AcidWriteSetService
> * AcidCompactionHistoryService
> * Initiator
> * Cleaner
> * etc.
> We should consolidate them where possible and improve logging.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23276) remove HttpException NPE possibility
[
https://issues.apache.org/jira/browse/HIVE-23276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098932#comment-17098932
]
Eugene Chung commented on HIVE-23276:
-
The test is failed with query result error but it's nothing to do with my patch.
{code:java}
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing multi_insert_partitioned.q
176,187c176,187
< 0 456 2
< 10456 2
< 100 457 2
< 103 457 2
< 455 97 1
< 455 98 1
< 456 0 1
< 456 10 1
< 457 100 1
< 457 103 1
< 97455 2
< 98455 2
---
> 0 456 1
> 10456 1
> 100 457 1
> 103 457 1
> 455 97 2
> 455 98 2
> 456 0 2
> 456 10 2
> 457 100 2
> 457 103 2
> 97455 1
> 98455 1
265d264
< 0 456 2
267d265
< 10456 2
269d266
< 100 457 2
271d267
< 103 457 2
272a269,270
> 455 97 2
> 455 98 2
274a273
> 456 0 2
275a275,276
> 456 10 2
> 457 100 2
277a279
> 457 103 2
279d280
< 97455 2
281d281
< 98455 2
372d371
< 0 456 2
375d373
< 10456 2
378d375
< 100 457 2
381c378
< 103 457 2
---
> 455 97 2
384a382
> 455 98 2
386a385
> 456 0 2
388a388
> 456 10 2
390a391
> 457 100 2
392a394
> 457 103 2
396d397
< 97455 2
399d399
< 98455 2
484d483
< 0 456 2
488d486
< 10456 2
492d489
< 100 457 2
496d492
< 103 457 2
498a495
> 455 97 2
501a499
> 455 98 2
504a503
> 456 0 2
507a507
> 456 10 2
510a511
> 457 100 2
513a515
> 457 103 2
518d519
< 97455 2
522d522
< 98455 2
{code}
> remove HttpException NPE possibility
> -
>
> Key: HIVE-23276
> URL: https://issues.apache.org/jira/browse/HIVE-23276
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Eugene Chung
>Assignee: Eugene Chung
>Priority: Minor
> Attachments: HIVE-23276.01.patch
>
>
> {code:java}
> java.lang.NullPointerException: null
> at org.apache.http.HttpException.clean(HttpException.java:48)
> ~[httpcore-4.4.10.jar:4.4.10]
> at org.apache.http.HttpException.(HttpException.java:105)
> ~[httpcore-4.4.10.jar:4.4.10]
> at
> org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:67)
> ~[hive-jdbc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74)
> ~[hive-jdbc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133)
> ~[httpcore-4.4.10.jar:4.4.10]
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:183)
> ~[httpclient-4.5.6.jar:4.5.6]
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> ~[httpclient-4.5.6.jar:4.5.6]
> at
> org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
> ~[httpclient-4.5.6.jar:4.5.6]
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> ~[httpclient-4.5.6.jar:4.5.6]
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> ~[httpclient-4.5.6.jar:4.5.6]
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
> ~[httpclient-4.5.6.jar:4.5.6]
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> ~[httpclient-4.5.6.jar:4.5.6]
> at
> org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:170)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:162)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:862)
> ~[hive-jdbc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:320)
> [hive-jdbc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:94)
[jira] [Commented] (HIVE-23325) Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098929#comment-17098929
]
Hive QA commented on HIVE-23325:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
29s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
28s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 118 new + 897 unchanged - 3 fixed = 1015 total (was 900) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 7 new + 744 unchanged - 4
fixed = 751 total (was 748) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22117/dev-support/hive-personality.sh
|
| git revision | master / 342f8fb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22117/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22117/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-common common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22117/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
> --
>
> Key: HIVE-23325
> URL: https://issues.apache.org/jira/browse/HIVE-23325
> Project: Hive
>
[jira] [Updated] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23349:
--
Status: Patch Available (was: Open)
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23349.1.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23349:
--
Description:
2 concurrent MERGE INSERT operations generate duplicates due to lack of
locking.
MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or EXCL_WRITE
if hive.txn.write.xlock=false;
{code}
create table target (a int, b int) stored as orc TBLPROPERTIES
('transactional'='true')");
insert into target values (1,2), (3,4)
create table source (a int, b int)
{code}
execute in parallel:
{code}
insert into source values (5,6), (7,8)
{code}
was:
2 concurrent MERGE INSERT operations generate duplicates due to lack of
locking.
MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
doesn't prevent other SHARED_READs. We should use EXCL_WRITE lock here or
SHARED_WRITE if hive.txn.write.xlock=false;
{code}
create table target (a int, b int) stored as orc TBLPROPERTIES
('transactional'='true')");
insert into target values (1,2), (3,4)
create table source (a int, b int)
{code}
execute in parallel:
{code}
insert into source values (5,6), (7,8)
{code}
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23349.1.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098922#comment-17098922 ] Peter Varga commented on HIVE-23340: [~pvary] I did see a little performance degradation, so I changed to not read any unnecessary columns for getOpenTxns, this way the change is not measurable anymore. > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Status: Patch Available (was: In Progress) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Status: In Progress (was: Patch Available) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Attachment: HIVE-23340.3.patch > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-23349:
--
Attachment: HIVE-23349.1.patch
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23349.1.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other SHARED_READs. We should use EXCL_WRITE lock here or
> SHARED_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23280) Trigger compaction with old aborted txns
[ https://issues.apache.org/jira/browse/HIVE-23280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23280: - Attachment: HIVE-23280.04.patch Status: Patch Available (was: Open) > Trigger compaction with old aborted txns > > > Key: HIVE-23280 > URL: https://issues.apache.org/jira/browse/HIVE-23280 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23280.01.patch, HIVE-23280.01.patch, > HIVE-23280.01.patch, HIVE-23280.01.patch, HIVE-23280.02.patch, > HIVE-23280.03.patch, HIVE-23280.04.patch, HIVE-23280.04.patch > > > When a txn is aborted and the compaction threshold for number of aborted txns > is not reached then the aborted transaction can remain forever in the RDBMS > database. This could result in several serious performance degradations: > - getOpenTxns has to list this aborted txn forever > - TXN_TO_WRITE_ID table is not cleaned > We should add a threshold, so after a given time the compaction is started > anyway. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23280) Trigger compaction with old aborted txns
[ https://issues.apache.org/jira/browse/HIVE-23280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23280: - Status: Open (was: Patch Available) > Trigger compaction with old aborted txns > > > Key: HIVE-23280 > URL: https://issues.apache.org/jira/browse/HIVE-23280 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23280.01.patch, HIVE-23280.01.patch, > HIVE-23280.01.patch, HIVE-23280.01.patch, HIVE-23280.02.patch, > HIVE-23280.03.patch, HIVE-23280.04.patch, HIVE-23280.04.patch > > > When a txn is aborted and the compaction threshold for number of aborted txns > is not reached then the aborted transaction can remain forever in the RDBMS > database. This could result in several serious performance degradations: > - getOpenTxns has to list this aborted txn forever > - TXN_TO_WRITE_ID table is not cleaned > We should add a threshold, so after a given time the compaction is started > anyway. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23361) Optimising privilege synchroniser
[ https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23361: -- Attachment: hive-23361.patch Status: Patch Available (was: Open) > Optimising privilege synchroniser > - > > Key: HIVE-23361 > URL: https://issues.apache.org/jira/browse/HIVE-23361 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Simhadri G >Priority: Minor > Attachments: hive-23361.patch, hive-23361.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > This patch store the privilege information as bit string. This is done to > reduce the size of the tbl_col_privs tables in metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098894#comment-17098894 ] Syed Shameerur Rahman commented on HIVE-23347: -- [~adeshrao] https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java#L386 We remove all the known partitions path (fetched from metastore) from the partitions path listed from the fileSystem. Since partition path fetched from metastore will always have lower case partition column name and the partition path listed from the fileSystem might have upper case column we might end up not removing the already present partition path. Eg: partition from metastore: /year=2020/month=3/day=2; partition from fileSystem: /Year=2020/Month=3/Day=2; Both these paths should be considered same and hence removed from *allPartDirs*. I guess HIVE-23347.3.patch doesn't handle that case. So i guess it is better to tackle this issue at place where the partition paths are fetched from fileSystem. > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch, > HIVE-23347.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-23282: - Attachment: (was: HIVE-23282.6.patch) > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch, > HIVE-23282.5.patch, HIVE-23282.6.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-23282: - Status: Patch Available (was: Open) > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch, > HIVE-23282.5.patch, HIVE-23282.6.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-23282: - Status: Open (was: Patch Available) > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch, > HIVE-23282.5.patch, HIVE-23282.6.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-23282: - Attachment: HIVE-23282.6.patch > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch, > HIVE-23282.5.patch, HIVE-23282.6.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-23282: - Attachment: HIVE-23282.6.patch > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch, > HIVE-23282.5.patch, HIVE-23282.6.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[
https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098885#comment-17098885
]
Hive QA commented on HIVE-23338:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001974/HIVE-23338.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17200 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22116/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22116/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22116/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001974 - PreCommit-HIVE-Build
> Bump jackson-databind version up to 2.9.10.4
>
>
> Key: HIVE-23338
> URL: https://issues.apache.org/jira/browse/HIVE-23338
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch,
> HIVE-23338.01.patch
>
>
> com.fasterxml.jackson.core:jackson-databind:2.9.9 is exploitable.
> And exclude a transitive dependency on
> com.fasterxml.jackson.core:jackson-databind:2.6.5, which is also exploitable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098874#comment-17098874 ] Peter Vary commented on HIVE-23340: --- [~pvargacl]: Please check if the performance change is measurable or not. Thanks, Peter > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao updated HIVE-23347: --- Status: Patch Available (was: Open) > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch, > HIVE-23347.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao updated HIVE-23347: --- Status: Open (was: Patch Available) > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch, > HIVE-23347.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao updated HIVE-23347: --- Attachment: HIVE-23347.3.patch > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch, > HIVE-23347.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-23358) MSCK REPAIR should remove all insignificant zeroes from partition values (for numeric datatypes) before creating the partitions
[
https://issues.apache.org/jira/browse/HIVE-23358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-23358:
-
Comment: was deleted
(was: [~adeshrao] are you reproducing this issue (including creating the
partitions) on the master branch? It should be fixed there with HIVE-10362)
> MSCK REPAIR should remove all insignificant zeroes from partition values (for
> numeric datatypes) before creating the partitions
> ---
>
> Key: HIVE-23358
> URL: https://issues.apache.org/jira/browse/HIVE-23358
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Minor
>
> For the following scenario
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1;
> {noformat}
> ++
> | partition |
> ++
> | year=2020/month=03/day=10 |
> | year=2020/month=03/day=11 |
> ++
> {noformat}
> 5.show table extended like 't1' partition (Year=2020, Month=03, Day=11);
> will throw an error:
> {noformat}
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10006]: Partition not found {year=2020, month=3, day=11}
> (state=42000,code=10006)
> {noformat}
> When the partition directory are created without the extra zeroes, this works
> fine.
> {noformat}
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=11
> {noformat}
> This happens because while searching for partitions, hive strips the extra
> "0" in month key and then queries the metastore
> (partSpec="year=2020/month=3/day=10") which returns no rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23358) MSCK REPAIR should remove all insignificant zeroes from partition values (for numeric datatypes) before creating the partitions
[
https://issues.apache.org/jira/browse/HIVE-23358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098870#comment-17098870
]
Karen Coppage commented on HIVE-23358:
--
[~adeshrao] are you reproducing this issue (including creating the partitions)
on the master branch? It should be fixed there with HIVE-10362
> MSCK REPAIR should remove all insignificant zeroes from partition values (for
> numeric datatypes) before creating the partitions
> ---
>
> Key: HIVE-23358
> URL: https://issues.apache.org/jira/browse/HIVE-23358
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Minor
>
> For the following scenario
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1;
> {noformat}
> ++
> | partition |
> ++
> | year=2020/month=03/day=10 |
> | year=2020/month=03/day=11 |
> ++
> {noformat}
> 5.show table extended like 't1' partition (Year=2020, Month=03, Day=11);
> will throw an error:
> {noformat}
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10006]: Partition not found {year=2020, month=3, day=11}
> (state=42000,code=10006)
> {noformat}
> When the partition directory are created without the extra zeroes, this works
> fine.
> {noformat}
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=11
> {noformat}
> This happens because while searching for partitions, hive strips the extra
> "0" in month key and then queries the metastore
> (partSpec="year=2020/month=3/day=10") which returns no rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[
https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098867#comment-17098867
]
Hive QA commented on HIVE-23338:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
9s{color} | {color:red} ptest2 in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
7s{color} | {color:red} ptest2 in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 7s{color}
| {color:red} ptest2 in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
6s{color} | {color:red} ptest2 in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22116/dev-support/hive-personality.sh
|
| git revision | master / 342f8fb |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22116/yetus/patch-mvninstall-testutils_ptest2.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22116/yetus/patch-compile-testutils_ptest2.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22116/yetus/patch-compile-testutils_ptest2.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22116/yetus/patch-javadoc-testutils_ptest2.txt
|
| modules | C: standalone-metastore upgrade-acid/pre-upgrade testutils/ptest2
U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22116/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bump jackson-databind version up to 2.9.10.4
>
>
> Key: HIVE-23338
> URL: https://issues.apache.org/jira/browse/HIVE-23338
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch,
> HIVE-23338.01.patch
>
>
> com.fasterxml.jackson.core:jackson-databind:2.9.9 is exploitable.
> And exclude a transitive dependency on
> com.fasterxml.jackson.core:jackson-databind:2.6.5, which is also exploitable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23325) Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23325: -- Attachment: HIVE-23325.3.patch > Clean up cleanup tasks for TxnHandler/CompactionTxnHandler > -- > > Key: HIVE-23325 > URL: https://issues.apache.org/jira/browse/HIVE-23325 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch, > HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch > > > Currently there are multiple ways to clean up ACID related stuff: > * AcidHouseKeeperService > * AcidWriteSetService > * AcidCompactionHistoryService > * Initiator > * Cleaner > * etc. > We should consolidate them where possible and improve logging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
