[jira] [Commented] (HIVE-23085) LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
[
https://issues.apache.org/jira/browse/HIVE-23085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068291#comment-17068291
]
Hive QA commented on HIVE-23085:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
50s{color} | {color:red} llap-server generated 1 new + 89 unchanged - 1 fixed =
90 total (was 90) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:llap-server |
| | Exceptional return value of java.io.File.delete() ignored in
org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(int,
int) At BuddyAllocator.java:ignored in
org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(int,
int) At BuddyAllocator.java:[line 880] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21291/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21291/yetus/new-findbugs-llap-server.html
|
| modules | C: common llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21291/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
>
>
> Key: HIVE-23085
> URL: https://issues.apache.org/jira/browse/HIVE-23085
> Project: Hive
> Issue Type: Improvement
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23085.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently we can configure only one SSD location while using SSD cache in
> LLAP. This highly undermines the
[jira] [Updated] (HIVE-22957) Add Predicate Filtering In MSCK REPAIR TABLE
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-22957: - Summary: Add Predicate Filtering In MSCK REPAIR TABLE (was: Predicate Filtering In MSCK REPAIR TABLE) > Add Predicate Filtering In MSCK REPAIR TABLE > > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22957) Add Predicate Filtering In MSCK REPAIR TABLE
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-22957: - Component/s: (was: Standalone Metastore) > Add Predicate Filtering In MSCK REPAIR TABLE > > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23079) Remove Calls to printStackTrace in Module hive-serde
[
https://issues.apache.org/jira/browse/HIVE-23079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068279#comment-17068279
]
Hive QA commented on HIVE-23079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997890/HIVE-23079.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18140 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21290/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21290/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21290/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997890 - PreCommit-HIVE-Build
> Remove Calls to printStackTrace in Module hive-serde
>
>
> Key: HIVE-23079
> URL: https://issues.apache.org/jira/browse/HIVE-23079
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23079.1.patch, HIVE-23079.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22842) Timestamp/date vectors in Arrow serializer should use correct calendar for value representation
[ https://issues.apache.org/jira/browse/HIVE-22842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-22842: - Attachment: HIVE-22842.05.patch > Timestamp/date vectors in Arrow serializer should use correct calendar for > value representation > --- > > Key: HIVE-22842 > URL: https://issues.apache.org/jira/browse/HIVE-22842 > Project: Hive > Issue Type: Improvement >Reporter: Jesus Camacho Rodriguez >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22842.01.patch, HIVE-22842.02.patch, > HIVE-22842.03.patch, HIVE-22842.04.patch, HIVE-22842.05.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23083) Enable fast serialization in xprod edge
[
https://issues.apache.org/jira/browse/HIVE-23083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068253#comment-17068253
]
Rajesh Balamohan edited comment on HIVE-23083 at 3/27/20, 4:19 AM:
---
Both test failures with following exceptions are unrelated to this patch.
{noformat}
java.lang.AssertionError: No thread with name metastore_task_thread_test_impl_3
found.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:49903/default: java.net.ConnectException: Connection
refused {noformat}
was (Author: rajesh.balamohan):
Both test failures with following exceptions are unrelated to this ticket
{noformat}
java.lang.AssertionError: No thread with name metastore_task_thread_test_impl_3
found.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:49903/default: java.net.ConnectException: Connection
refused {noformat}
> Enable fast serialization in xprod edge
> ---
>
> Key: HIVE-23083
> URL: https://issues.apache.org/jira/browse/HIVE-23083
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23083.1.patch, Screenshot 2020-03-26 at 2.28.34
> PM.png
>
>
> {noformat}
> select count(*) from store_sales, store, customer, customer_address where
> ss_store_sk = s_store_sk and s_market_id=10 and ss_customer_sk =
> c_customer_sk and c_birth_country <> upper(ca_country);
> {noformat}
> This uses "org/apache/hadoop/io/serializer/WritableSerialization" instead of
> TezBytesWritableSerialization.
>
> !Screenshot 2020-03-26 at 2.28.34 PM.png|width=812,height=488!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23083) Enable fast serialization in xprod edge
[
https://issues.apache.org/jira/browse/HIVE-23083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068253#comment-17068253
]
Rajesh Balamohan commented on HIVE-23083:
-
Both test failures with following exceptions are unrelated to this ticket
{noformat}
java.lang.AssertionError: No thread with name metastore_task_thread_test_impl_3
found.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:49903/default: java.net.ConnectException: Connection
refused {noformat}
> Enable fast serialization in xprod edge
> ---
>
> Key: HIVE-23083
> URL: https://issues.apache.org/jira/browse/HIVE-23083
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23083.1.patch, Screenshot 2020-03-26 at 2.28.34
> PM.png
>
>
> {noformat}
> select count(*) from store_sales, store, customer, customer_address where
> ss_store_sk = s_store_sk and s_market_id=10 and ss_customer_sk =
> c_customer_sk and c_birth_country <> upper(ca_country);
> {noformat}
> This uses "org/apache/hadoop/io/serializer/WritableSerialization" instead of
> TezBytesWritableSerialization.
>
> !Screenshot 2020-03-26 at 2.28.34 PM.png|width=812,height=488!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23079) Remove Calls to printStackTrace in Module hive-serde
[
https://issues.apache.org/jira/browse/HIVE-23079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068251#comment-17068251
]
Hive QA commented on HIVE-23079:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21290/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21290/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove Calls to printStackTrace in Module hive-serde
>
>
> Key: HIVE-23079
> URL: https://issues.apache.org/jira/browse/HIVE-23079
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23079.1.patch, HIVE-23079.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22957) Predicate Filtering In MSCK REPAIR TABLE
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-22957: - Summary: Predicate Filtering In MSCK REPAIR TABLE (was: Support For Filter Expression In MSCK Command) > Predicate Filtering In MSCK REPAIR TABLE > > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23077) Remove Calls to printStackTrace in Module hive-jdbc
[
https://issues.apache.org/jira/browse/HIVE-23077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068248#comment-17068248
]
Hive QA commented on HIVE-23077:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997891/HIVE-23077.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 18140 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=252)
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testAcidTablesReplLoadBootstrapIncr
(batchId=283)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21289/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21289/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21289/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997891 - PreCommit-HIVE-Build
> Remove Calls to printStackTrace in Module hive-jdbc
> ---
>
> Key: HIVE-23077
> URL: https://issues.apache.org/jira/browse/HIVE-23077
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23077.1.patch, HIVE-23077.1.patch
>
>
> Only one "tricky" change. Throw an Exception instead of {{printStackTrace}}
> in the static Driver loader as suggested from the reference here:
> https://github.com/mariadb-corporation/mariadb-connector-j/blob/3bc66153b51aca188afc50ff35a0123f16c099ed/src/main/java/org/mariadb/jdbc/Driver.java#L72
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23077) Remove Calls to printStackTrace in Module hive-jdbc
[
https://issues.apache.org/jira/browse/HIVE-23077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068226#comment-17068226
]
Hive QA commented on HIVE-23077:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21289/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21289/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove Calls to printStackTrace in Module hive-jdbc
> ---
>
> Key: HIVE-23077
> URL: https://issues.apache.org/jira/browse/HIVE-23077
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23077.1.patch, HIVE-23077.1.patch
>
>
> Only one "tricky" change. Throw an Exception instead of {{printStackTrace}}
> in the static Driver loader as suggested from the reference here:
> https://github.com/mariadb-corporation/mariadb-connector-j/blob/3bc66153b51aca188afc50ff35a0123f16c099ed/src/main/java/org/mariadb/jdbc/Driver.java#L72
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23084) Implement kill query in multiple HS2 environment
[
https://issues.apache.org/jira/browse/HIVE-23084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068216#comment-17068216
]
Hive QA commented on HIVE-23084:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997887/HIVE-23084.1.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18150 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.service.cli.thrift.TestMiniHS2StateWithNoZookeeper.openSessionAndClose
(batchId=287)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21288/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21288/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21288/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997887 - PreCommit-HIVE-Build
> Implement kill query in multiple HS2 environment
>
>
> Key: HIVE-23084
> URL: https://issues.apache.org/jira/browse/HIVE-23084
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23084.1.patch
>
>
> KILL command was implemented in:
> * https://issues.apache.org/jira/browse/HIVE-17483
> * https://issues.apache.org/jira/browse/HIVE-20549
> But it is not working in an environment where service discovery is enabled
> and more than one HS2 instance is running (except for manually sending the
> kill query to all HS2 instance).
> Solution:
> * If a HS2 instance can't kill a query locally, it should post a kill query
> request to the Zookeeper
> * Every HS2 should watch the Zookeeper for kill query requests and if its
> running on that instance kill it
> * Authorization of kill query should work the same
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23084) Implement kill query in multiple HS2 environment
[
https://issues.apache.org/jira/browse/HIVE-23084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068193#comment-17068193
]
Hive QA commented on HIVE-23084:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 1529 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} service in master has 50 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
48s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
40s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} service: The patch generated 5 new + 87 unchanged - 33
fixed = 92 total (was 120) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 14 new + 32
unchanged - 0 fixed = 46 total (was 32) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21288/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21288/yetus/patch-mvninstall-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21288/yetus/diff-checkstyle-service.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21288/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: common ql service itests/hive-unit itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21288/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement kill query in multiple HS2 environment
>
>
>
[jira] [Commented] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068172#comment-17068172
]
Hive QA commented on HIVE-22785:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997926/HIVE-22785.14.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 18141 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[topnkey_grouping_sets]
(batchId=1)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_all_non_partitioned]
(batchId=174)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation4 (batchId=295)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21287/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21287/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21287/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997926 - PreCommit-HIVE-Build
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.10.patch,
> HIVE-22785.11.patch, HIVE-22785.12.patch, HIVE-22785.13.patch,
> HIVE-22785.14.patch, HIVE-22785.2.patch, HIVE-22785.2.patch,
> HIVE-22785.3.patch, HIVE-22785.4.patch, HIVE-22785.5.patch,
> HIVE-22785.6.patch, HIVE-22785.7.patch, HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068145#comment-17068145
]
Hive QA commented on HIVE-22785:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1529 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 5 new + 540 unchanged - 12
fixed = 545 total (was 552) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
50s{color} | {color:red} ql generated 2 new + 1527 unchanged - 2 fixed = 1529
total (was 1529) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveSortExchange.getKeyExpressions()
may expose internal representation by returning
HiveSortExchange.keyExpressions At HiveSortExchange.java:by returning
HiveSortExchange.keyExpressions At HiveSortExchange.java:[line 102] |
| |
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveSortExchange.setKeyExpressions(ExprNodeDesc[])
may expose internal representation by storing an externally mutable object
into HiveSortExchange.keyExpressions At HiveSortExchange.java:by storing an
externally mutable object into HiveSortExchange.keyExpressions At
HiveSortExchange.java:[line 106] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21287/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21287/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21287/yetus/new-findbugs-ql.html
|
| modules | C: ql itests itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21287/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>
[jira] [Commented] (HIVE-23018) Provide a bulk API to fire multiple insert events
[ https://issues.apache.org/jira/browse/HIVE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068140#comment-17068140 ] Vihang Karajgaonkar commented on HIVE-23018: Patch 04 should fix the NPE issue in the log statement added in the patch. > Provide a bulk API to fire multiple insert events > - > > Key: HIVE-23018 > URL: https://issues.apache.org/jira/browse/HIVE-23018 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-23018.01.patch, HIVE-23018.02.patch, > HIVE-23018.03.patch, HIVE-23018.04.patch > > > Metastore provides a API to fire a listener event (currently only supports > INSERT event). The problem with that API is that it only takes in one > partition at a time. A typical query may insert data into multiple partitions > at a time. In such a case query engines like HS2 or Impala will have to issue > multiple RPCs to metastore sequentially to fire these events. This can show > up as a slowdown to the user if the query engines do not return the prompt to > the user until all the events are fired (In case of HS2 and Impala). It would > be great if we have bulk API which takes in multiple partitions for a table > so that metastore can generate many such events in one RPC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23018) Provide a bulk API to fire multiple insert events
[ https://issues.apache.org/jira/browse/HIVE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-23018: --- Attachment: HIVE-23018.04.patch > Provide a bulk API to fire multiple insert events > - > > Key: HIVE-23018 > URL: https://issues.apache.org/jira/browse/HIVE-23018 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-23018.01.patch, HIVE-23018.02.patch, > HIVE-23018.03.patch, HIVE-23018.04.patch > > > Metastore provides a API to fire a listener event (currently only supports > INSERT event). The problem with that API is that it only takes in one > partition at a time. A typical query may insert data into multiple partitions > at a time. In such a case query engines like HS2 or Impala will have to issue > multiple RPCs to metastore sequentially to fire these events. This can show > up as a slowdown to the user if the query engines do not return the prompt to > the user until all the events are fired (In case of HS2 and Impala). It would > be great if we have bulk API which takes in multiple partitions for a table > so that metastore can generate many such events in one RPC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[
https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068133#comment-17068133
]
Hive QA commented on HIVE-23058:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997878/HIVE_23058.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18141 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_ingest]
(batchId=185)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21286/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21286/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21286/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997878 - PreCommit-HIVE-Build
> Compaction task reattempt fails with FileAlreadyExistsException
> ---
>
> Key: HIVE-23058
> URL: https://issues.apache.org/jira/browse/HIVE-23058
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE_23058.1.patch, HIVE_23058.patch
>
>
> Issue occurs when compaction attempt is relaunched after first task attempt
> failure due to preemption by Scheduler or any other reason.
> Since _tmp directory was created by first attempt and was left uncleaned
> after task attempt failure. Second attempt of the the task fails with
> "FileAlreadyExistsException" exception.
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException):
>
> /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5
> for client 10.xx.xx.xxx already exists
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[
https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068096#comment-17068096
]
Hive QA commented on HIVE-23058:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 1529 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 1 new + 37 unchanged - 0 fixed
= 38 total (was 37) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 127
unchanged - 0 fixed = 129 total (was 127) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21286/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21286/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21286/yetus/diff-checkstyle-itests_hive-unit.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21286/yetus/whitespace-eol.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21286/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Compaction task reattempt fails with FileAlreadyExistsException
> ---
>
> Key: HIVE-23058
> URL: https://issues.apache.org/jira/browse/HIVE-23058
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE_23058.1.patch, HIVE_23058.patch
>
>
> Issue occurs when compaction attempt is relaunched after first task
[jira] [Updated] (HIVE-23030) Enable sketch union-s to be rolled up
[ https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23030: Attachment: HIVE-23030.04.patch > Enable sketch union-s to be rolled up > - > > Key: HIVE-23030 > URL: https://issues.apache.org/jira/browse/HIVE-23030 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23030.01.patch, HIVE-23030.02.patch, > HIVE-23030.03.patch, HIVE-23030.04.patch, HIVE-23030.04.patch > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Enabling rolling up sketch aggregates could enable the matching of > materialized views created for higher dimensions to be applied for lower > dimension cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22098) Data loss occurs when multiple tables are join with different bucket_version
[
https://issues.apache.org/jira/browse/HIVE-22098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067919#comment-17067919
]
Sam An edited comment on HIVE-22098 at 3/26/20, 9:58 PM:
-
[~luguangming], hope the message finds you well. There are some production use
case for your solution. Would you be able to work on this further at this time?
was (Author: samuelan):
[~luguangming], hope the message finds you well. There are some production use
case for your solution. Would you be able work on this further at this time?
> Data loss occurs when multiple tables are join with different bucket_version
>
>
> Key: HIVE-22098
> URL: https://issues.apache.org/jira/browse/HIVE-22098
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Assignee: LuGuangMing
>Priority: Blocker
> Labels: data-loss, wrongresults
> Attachments: HIVE-22098.1.patch, image-2019-08-12-18-45-15-771.png,
> join_test.sql, table_a_data.orc, table_b_data.orc, table_c_data.orc
>
>
> When different bucketVersion of tables do join and no of reducers is greater
> than 2, the result is incorrect (*data loss*).
> *Scenario 1*: Three tables join. The temporary result data of table_a in the
> first table and table_b in the second table joins result is recorded as
> tmp_a_b, When it joins with the third table, the bucket_version=2 of the
> table created by default after hive-3.0.0, temporary data tmp_a_b initialized
> the bucketVerison=-1, and then ReduceSinkOperator Verketison=-1 is joined. In
> the init method, the hash algorithm of selecting join column is selected
> according to bucketVersion. If bucketVersion = 2 and is not an acid
> operation, it will acquired the new algorithm of hash. Otherwise, the old
> algorithm of hash is acquired. Because of the inconsistency of the algorithm
> of hash, the partition of data allocation caused are different. At stage of
> Reducer, Data with the same key can not be paired resulting in data loss.
> *Scenario 2*: create two test tables, create table
> table_bucketversion_1(col_1 string, col_2 string) TBLPROPERTIES
> ('bucketing_version'='1'); table_bucketversion_2(col_1 string, col_2 string)
> TBLPROPERTIES ('bucketing_version'='2');
> when use table_bucketversion_1 to join table_bucketversion_2, partial result
> data will be loss due to bucketVerison is different.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22098) Data loss occurs when multiple tables are join with different bucket_version
[
https://issues.apache.org/jira/browse/HIVE-22098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067919#comment-17067919
]
Sam An edited comment on HIVE-22098 at 3/26/20, 9:57 PM:
-
[~luguangming], hope the message finds you well. There are some production use
case for your solution. Would you be able work on this further at this time?
was (Author: samuelan):
[~luguangming], hope the message finds you well. There are some production use
case for your solution. Would you like to complete this patch?
> Data loss occurs when multiple tables are join with different bucket_version
>
>
> Key: HIVE-22098
> URL: https://issues.apache.org/jira/browse/HIVE-22098
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Assignee: LuGuangMing
>Priority: Blocker
> Labels: data-loss, wrongresults
> Attachments: HIVE-22098.1.patch, image-2019-08-12-18-45-15-771.png,
> join_test.sql, table_a_data.orc, table_b_data.orc, table_c_data.orc
>
>
> When different bucketVersion of tables do join and no of reducers is greater
> than 2, the result is incorrect (*data loss*).
> *Scenario 1*: Three tables join. The temporary result data of table_a in the
> first table and table_b in the second table joins result is recorded as
> tmp_a_b, When it joins with the third table, the bucket_version=2 of the
> table created by default after hive-3.0.0, temporary data tmp_a_b initialized
> the bucketVerison=-1, and then ReduceSinkOperator Verketison=-1 is joined. In
> the init method, the hash algorithm of selecting join column is selected
> according to bucketVersion. If bucketVersion = 2 and is not an acid
> operation, it will acquired the new algorithm of hash. Otherwise, the old
> algorithm of hash is acquired. Because of the inconsistency of the algorithm
> of hash, the partition of data allocation caused are different. At stage of
> Reducer, Data with the same key can not be paired resulting in data loss.
> *Scenario 2*: create two test tables, create table
> table_bucketversion_1(col_1 string, col_2 string) TBLPROPERTIES
> ('bucketing_version'='1'); table_bucketversion_2(col_1 string, col_2 string)
> TBLPROPERTIES ('bucketing_version'='2');
> when use table_bucketversion_1 to join table_bucketversion_2, partial result
> data will be loss due to bucketVerison is different.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-18983) Add support for table properties inheritance in Create table like
[
https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067920#comment-17067920
]
Sam An edited comment on HIVE-18983 at 3/26/20, 9:57 PM:
-
[~ikryvenko] hope the message finds you well. There are some production use
case for your solution. Would you be able work on this further at this time? I
can see you have done quite a bit of coding and testing.
was (Author: samuelan):
[~ikryvenko] hope the message finds you well. There are some production use
case for your solution. Would you like to complete this patch? I can see you
have done quite a bit of coding and testing.
> Add support for table properties inheritance in Create table like
> -
>
> Key: HIVE-18983
> URL: https://issues.apache.org/jira/browse/HIVE-18983
> Project: Hive
> Issue Type: Improvement
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-18983.01.patch, HIVE-18983.02.patch,
> HIVE-18983.03.patch, HIVE-18983.04.patch, HIVE-18983.05.patch,
> HIVE-18983.06.patch, HIVE-18983.07.patch, HIVE-18983.08.patch,
> HIVE-18983.09.patch, HIVE-18983.10.patch, HIVE-18983.11.patch,
> HIVE-18983.12.patch
>
>
> Currently, Create table like support table properties.
> But it doesn't inherit table properties from original table.
> {code}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('comment'='comm');
> create table T like T1;
> show create table T;
> {code}
> *Output:*
> {code}
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
> 'maprfs:/user/hive/warehouse/t'
> TBLPROPERTIES (
> 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
> 'numFiles'='0',
> 'numRows'='0',
> 'rawDataSize'='0',
> 'totalSize'='0',
> 'transient_lastDdlTime'='1521230300')
> {code}
> It uses just default table properties and doesn't inherit properties from
> original table.
> It would be great if create table like will inherit origin table properties
> and will override them if they are specified in query .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-18983) Add support for table properties inheritance in Create table like
[
https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067920#comment-17067920
]
Sam An edited comment on HIVE-18983 at 3/26/20, 9:57 PM:
-
[~ikryvenko] hope the message finds you well. There are some production use
case for your solution. Would you be able to work on this further at this time?
I can see you have done quite a bit of coding and testing.
was (Author: samuelan):
[~ikryvenko] hope the message finds you well. There are some production use
case for your solution. Would you be able work on this further at this time? I
can see you have done quite a bit of coding and testing.
> Add support for table properties inheritance in Create table like
> -
>
> Key: HIVE-18983
> URL: https://issues.apache.org/jira/browse/HIVE-18983
> Project: Hive
> Issue Type: Improvement
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-18983.01.patch, HIVE-18983.02.patch,
> HIVE-18983.03.patch, HIVE-18983.04.patch, HIVE-18983.05.patch,
> HIVE-18983.06.patch, HIVE-18983.07.patch, HIVE-18983.08.patch,
> HIVE-18983.09.patch, HIVE-18983.10.patch, HIVE-18983.11.patch,
> HIVE-18983.12.patch
>
>
> Currently, Create table like support table properties.
> But it doesn't inherit table properties from original table.
> {code}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('comment'='comm');
> create table T like T1;
> show create table T;
> {code}
> *Output:*
> {code}
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
> 'maprfs:/user/hive/warehouse/t'
> TBLPROPERTIES (
> 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
> 'numFiles'='0',
> 'numRows'='0',
> 'rawDataSize'='0',
> 'totalSize'='0',
> 'transient_lastDdlTime'='1521230300')
> {code}
> It uses just default table properties and doesn't inherit properties from
> original table.
> It would be great if create table like will inherit origin table properties
> and will override them if they are specified in query .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23030) Enable sketch union-s to be rolled up
[
https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068078#comment-17068078
]
Hive QA commented on HIVE-23030:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997877/HIVE-23030.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 18141 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_ingest]
(batchId=185)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_orc_nested_column_pruning]
(batchId=172)
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
(batchId=239)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21285/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21285/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21285/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997877 - PreCommit-HIVE-Build
> Enable sketch union-s to be rolled up
> -
>
> Key: HIVE-23030
> URL: https://issues.apache.org/jira/browse/HIVE-23030
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23030.01.patch, HIVE-23030.02.patch,
> HIVE-23030.03.patch, HIVE-23030.04.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Enabling rolling up sketch aggregates could enable the matching of
> materialized views created for higher dimensions to be applied for lower
> dimension cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23051) Clean up BucketCodec
[
https://issues.apache.org/jira/browse/HIVE-23051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068072#comment-17068072
]
David Mollitor commented on HIVE-23051:
---
[~abstractdog] I have added a unit test. I don't really know what the
input/output should be, so the tests I wrote codify the current behavior.
Review please? :)
> Clean up BucketCodec
>
>
> Key: HIVE-23051
> URL: https://issues.apache.org/jira/browse/HIVE-23051
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23051.1.patch, HIVE-23051.2.patch,
> HIVE-23051.3.patch, HIVE-23051.4.patch, HIVE-23051.4.patch
>
>
> A couple of nagging things caught my eye with this class. The first thing:
> {code:java|title=BucketCodec.java}
> int statementId = options.getStatementId() >= 0 ?
> options.getStatementId() : 0;
> assert this.version >=0 && this.version <= MAX_VERSION
> : "Version out of range: " + version;
> if(!(options.getBucketId() >= 0 && options.getBucketId() <=
> MAX_BUCKET_ID)) {
> throw new IllegalArgumentException("bucketId out of range: " +
> options.getBucketId());
> }
> if(!(statementId >= 0 && statementId <= MAX_STATEMENT_ID)) {
> throw new IllegalArgumentException("statementId out of range: " +
> statementId);
> }
> {code}
> {{statementId}} gets capped, if it's less than 0, then it gets rounded up to
> 0. However, it later checks that the {{statementId}} is greater,... which is
> will always be 'true' since it's getting rounded.
> # Remove the rounding behavior.
> # Make better error message
> # Fail-fast in the constructor if the version is invalid
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[
https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068017#comment-17068017
]
Peter Vary edited comment on HIVE-23032 at 3/26/20, 9:30 PM:
-
Removed the comment as it was false alarm. Maybe next time I will wait until
morning :)
was (Author: pvary):
{code}
if (intLockId % batchSize != 0) {
pstmt.executeBatch();
}
{code}
should be, since intLockId incremented since the previous check:
{code}
if ((intLockId - 1) % batchSize != 0) {
pstmt.executeBatch();
}
{code}
I did the same mistake in my batching patch, but luckily I had a test with
enough transactions to show me my error :)
> Add batching in Lock generation
> ---
>
> Key: HIVE-23032
> URL: https://issues.apache.org/jira/browse/HIVE-23032
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch,
> HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch,
> HIVE-23032.6.patch, HIVE-23032.7.patch, HIVE-23032.8.patch
>
>
> Replace multi-row insert in Oracle with batching. Performance tests showed
> significant performance improvement after turning batching on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23030) Enable sketch union-s to be rolled up
[
https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068036#comment-17068036
]
Hive QA commented on HIVE-23030:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 1529 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 4 new + 174 unchanged - 0
fixed = 178 total (was 174) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21285/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21285/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21285/yetus/whitespace-eol.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21285/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable sketch union-s to be rolled up
> -
>
> Key: HIVE-23030
> URL: https://issues.apache.org/jira/browse/HIVE-23030
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23030.01.patch, HIVE-23030.02.patch,
> HIVE-23030.03.patch, HIVE-23030.04.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Enabling rolling up sketch aggregates could enable the matching of
> materialized views created for higher dimensions to be applied for lower
> dimension cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068029#comment-17068029 ] Denys Kuzmenko commented on HIVE-23032: --- [~pvary], I am not following why should I have (intLockId - 1)? it's 0 at the beginning, and after it's being incremented in a loop with (intLockId % batchSize == 0) check. > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch, HIVE-23032.8.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23032) Add batching in Lock generation
[
https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068017#comment-17068017
]
Peter Vary commented on HIVE-23032:
---
{code}
if (intLockId % batchSize != 0) {
pstmt.executeBatch();
}
{code}
should be, since intLockId incremented since the previous check:
{code}
if ((intLockId - 1) % batchSize != 0) {
pstmt.executeBatch();
}
{code}
I did the same mistake in my batching patch, but luckily I had a test with
enough transactions to show me my error :)
> Add batching in Lock generation
> ---
>
> Key: HIVE-23032
> URL: https://issues.apache.org/jira/browse/HIVE-23032
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch,
> HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch,
> HIVE-23032.6.patch, HIVE-23032.7.patch, HIVE-23032.8.patch
>
>
> Replace multi-row insert in Oracle with batching. Performance tests showed
> significant performance improvement after turning batching on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23076) Add batching for openTxn
[ https://issues.apache.org/jira/browse/HIVE-23076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23076: -- Attachment: HIVE-23076.3.patch > Add batching for openTxn > > > Key: HIVE-23076 > URL: https://issues.apache.org/jira/browse/HIVE-23076 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-23076.2.patch, HIVE-23076.3.patch, HIVE-23076.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23083) Enable fast serialization in xprod edge
[
https://issues.apache.org/jira/browse/HIVE-23083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068009#comment-17068009
]
Hive QA commented on HIVE-23083:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997872/HIVE-23083.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 18138 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=252)
org.apache.hive.jdbc.TestJdbcWithMiniHS2ErasureCoding.org.apache.hive.jdbc.TestJdbcWithMiniHS2ErasureCoding
(batchId=295)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21284/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21284/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21284/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997872 - PreCommit-HIVE-Build
> Enable fast serialization in xprod edge
> ---
>
> Key: HIVE-23083
> URL: https://issues.apache.org/jira/browse/HIVE-23083
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23083.1.patch, Screenshot 2020-03-26 at 2.28.34
> PM.png
>
>
> {noformat}
> select count(*) from store_sales, store, customer, customer_address where
> ss_store_sk = s_store_sk and s_market_id=10 and ss_customer_sk =
> c_customer_sk and c_birth_country <> upper(ca_country);
> {noformat}
> This uses "org/apache/hadoop/io/serializer/WritableSerialization" instead of
> TezBytesWritableSerialization.
>
> !Screenshot 2020-03-26 at 2.28.34 PM.png|width=812,height=488!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko edited comment on HIVE-23032 at 3/26/20, 8:18 PM: - MySQL ||Mode|batchSize|partCount|tblCount|Cnt|Score|Error|Units|| |original|x|30|4|100|251.485|± 5.201|ms/op| |patched|200|30|4|100|256.098|± 2.946|ms/op| |patched|1000|30|4|100|261.323|± 4.767|ms/op| |patched|1200|30|4|100|250.143|± 2.433|ms/op| |original|x|300|4|100|327.434|± 6.124|ms/op| |patched|200|300|4|100|336.495|± 3.688|ms/op| |patched|1000|300|4|100|338.376|± 3.474|ms/op| |patched|1200|300|4|100|333.510|± 4.098|ms/op| |original|x|600|4|100|472.850|± 9.435|ms/op| |patched|200|600|4|100|478.784|± 8.744|ms/op| |patched|1000|600|4|100|515.727|± 12.683|ms/op| |patched|1200|600|4|100|500.296|± 9.335|ms/op| |original|x|1200|4|100|726.753|± 41.900|ms/op| |patched|200|1200|4|100|705.487|± 9.658|ms/op| |patched|1000|1200|4|100|752.484|± 19.530|ms/op| |patched|1200|1200|4|100|691.993|± 11.539|ms/op| Oracle ||Mode|batchSize|partCount|tblCount|Cnt|Score|Error|Units|| |original|x|30|4|100|775.042|± 60.616|ms/op| |patched|200|30|4|100|632.782|± 42.980|ms/op| |patched|1000|30|4|100|434.803|± 38.732|ms/op| |patched|1200|30|4|100|616.095|± 41.783|ms/op| |original|x|300|4|100|11539.710|± 580.909|ms/op| |patched|200|300|4|100|1186.042|± 74.027|ms/op| |patched|1000|300|4|100|870.643|± 32.118|ms/op| |patched|1200|300|4|100|1207.119|± 79.411|ms/op| |original|x|600|4|100|22671.069|± 949.307|ms/op| |patched|200|600|4|100|2060.261|± 209.748|ms/op| |patched|1000|600|4|100|1443.972|± 110.546|ms/op| |patched|1200|600|4|100|2053.340|± 88.405|ms/op| |original|x|1200|4|100|47114.679|± 1577.975|ms/op| |patched|200|1200|4|100|3362.730|± 121.145|ms/op| |patched|1000|1200|4|100|2499.806|± 100.270|ms/op| |patched|1200|1200|4|100|3509.726|± 113.270|ms/op| Postgres ||Mode|batchSize|partCount|tblCount|Cnt|Score|Error|Units|| |original|x|30|4|100|310.203|± 10.585|ms/op| |patched|200|30|4|100|290.962|± 9.010|ms/op| |patched|1000|30|4|100|281.426|± 2.209|ms/op| |patched|1200|30|4|100|306.250|± 5.894|ms/op| |original|x|300|4|100|453.817|± 11.402|ms/op| |patched|200|300|4|100|510.549|± 33.378|ms/op| |patched|1000|300|4|100|320.018|± 6.612|ms/op| |patched|1200|300|4|100|426.354|± 41.727|ms/op| |original|x|600|4|100|1148.439|± 120.601|ms/op| |patched|200|600|4|100|904.347|± 126.828|ms/op| |patched|1000|600|4|100|445.005|± 22.069|ms/op| |patched|1200|600|4|100|808.863|± 152.863|ms/op| |original|x|1200|4|100|1366.978|± 380.341|ms/op| |patched|200|1200|4|100|1187.517|± 242.108|ms/op| |patched|1000|1200|4|100|662.381|± 16.256|ms/op| |patched|1200|1200|4|100|2137.430|± 543.652|ms/op| was (Author: dkuzmenko): MySQL ||Mode|batchSize|partCount|tblCount|Cnt|Score|Error|Units|| |original|x|30|4|100|251.485|± 5.201|ms/op| |patched|1000|30|4|100|261.323|± 4.767|ms/op| |patched|200|30|4|100|256.098|± 2.946|ms/op| |original|x|300|4|100|327.434|± 6.124|ms/op| |patched|1000|300|4|100|338.376|± 3.474|ms/op| |patched|200|300|4|100|336.495|± 3.688|ms/op| |original|x|600|4|100|472.850|± 9.435|ms/op| |patched|1000|600|4|100|515.727|± 12.683|ms/op| |patched|200|600|4|100|478.784|± 8.744|ms/op| |original|x|1200|4|100|726.753|± 41.900|ms/op| |patched|1000|1200|4|100|752.484|± 19.530|ms/op| |patched|200|1200|4|100|705.487|± 9.658|ms/op| Oracle ||Mode|batchSize|partCount|tblCount|Cnt|Score|Error|Units|| |original|x|30|4|100|775.042|± 60.616|ms/op| |patched|1000|30|4|100|434.803|± 38.732|ms/op| |patched|200|30|4|100|632.782|± 42.980|ms/op| |original|x|300|4|100|11539.710|± 580.909|ms/op| |patched|1000|300|4|100|870.643|± 32.118|ms/op| |patched|200|300|4|100|1186.042|± 74.027|ms/op| |original|x|600|4|100|22671.069|± 949.307|ms/op| |patched|1000|600|4|100|1443.972|± 110.546|ms/op| |patched|200|600|4|100|2060.261|± 209.748|ms/op| |original|x|1200|4|100|47114.679|± 1577.975|ms/op| |patched|1000|1200|4|100|2499.806|± 100.270|ms/op| |patched|200|1200|4|100|3362.730|± 121.145|ms/op| Postgres ||Mode|batchSize|partCount|tblCount|Cnt|Score|Error|Units|| |original|x|30|4|100|310.203|± 10.585|ms/op| |patched|1000|30|4|100|281.426|± 2.209|ms/op| |patched|200|30|4|100|290.962|± 9.010|ms/op| |original|x|300|4|100|453.817|± 11.402|ms/op| |patched|1000|300|4|100|320.018|± 6.612|ms/op| |patched|200|300|4|100|510.549|± 33.378|ms/op| |original|x|600|4|100|1148.439|± 120.601|ms/op| |patched|1000|600|4|100|445.005|± 22.069|ms/op| |patched|200|600|4|100|904.347|± 126.828|ms/op| |original|x|1200|4|100|1366.978|± 380.341|ms/op| |patched|1000|1200|4|100|662.381|± 16.256|ms/op| |patched|200|1200|4|100|1187.517|± 242.108|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Patch Available (was: Open)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.10.patch,
> HIVE-22785.11.patch, HIVE-22785.12.patch, HIVE-22785.13.patch,
> HIVE-22785.14.patch, HIVE-22785.2.patch, HIVE-22785.2.patch,
> HIVE-22785.3.patch, HIVE-22785.4.patch, HIVE-22785.5.patch,
> HIVE-22785.6.patch, HIVE-22785.7.patch, HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Open (was: Patch Available)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.10.patch,
> HIVE-22785.11.patch, HIVE-22785.12.patch, HIVE-22785.13.patch,
> HIVE-22785.14.patch, HIVE-22785.2.patch, HIVE-22785.2.patch,
> HIVE-22785.3.patch, HIVE-22785.4.patch, HIVE-22785.5.patch,
> HIVE-22785.6.patch, HIVE-22785.7.patch, HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Attachment: HIVE-22785.14.patch
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.10.patch,
> HIVE-22785.11.patch, HIVE-22785.12.patch, HIVE-22785.13.patch,
> HIVE-22785.14.patch, HIVE-22785.2.patch, HIVE-22785.2.patch,
> HIVE-22785.3.patch, HIVE-22785.4.patch, HIVE-22785.5.patch,
> HIVE-22785.6.patch, HIVE-22785.7.patch, HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko edited comment on HIVE-23032 at 3/26/20, 7:41 PM: - MySQL ||Mode|batchSize|partCount|tblCount|Cnt|Score|Error|Units|| |original|x|30|4|100|251.485|± 5.201|ms/op| |patched|1000|30|4|100|261.323|± 4.767|ms/op| |patched|200|30|4|100|256.098|± 2.946|ms/op| |original|x|300|4|100|327.434|± 6.124|ms/op| |patched|1000|300|4|100|338.376|± 3.474|ms/op| |patched|200|300|4|100|336.495|± 3.688|ms/op| |original|x|600|4|100|472.850|± 9.435|ms/op| |patched|1000|600|4|100|515.727|± 12.683|ms/op| |patched|200|600|4|100|478.784|± 8.744|ms/op| |original|x|1200|4|100|726.753|± 41.900|ms/op| |patched|1000|1200|4|100|752.484|± 19.530|ms/op| |patched|200|1200|4|100|705.487|± 9.658|ms/op| Oracle ||Mode|batchSize|partCount|tblCount|Cnt|Score|Error|Units|| |original|x|30|4|100|775.042|± 60.616|ms/op| |patched|1000|30|4|100|434.803|± 38.732|ms/op| |patched|200|30|4|100|632.782|± 42.980|ms/op| |original|x|300|4|100|11539.710|± 580.909|ms/op| |patched|1000|300|4|100|870.643|± 32.118|ms/op| |patched|200|300|4|100|1186.042|± 74.027|ms/op| |original|x|600|4|100|22671.069|± 949.307|ms/op| |patched|1000|600|4|100|1443.972|± 110.546|ms/op| |patched|200|600|4|100|2060.261|± 209.748|ms/op| |original|x|1200|4|100|47114.679|± 1577.975|ms/op| |patched|1000|1200|4|100|2499.806|± 100.270|ms/op| |patched|200|1200|4|100|3362.730|± 121.145|ms/op| Postgres ||Mode|batchSize|partCount|tblCount|Cnt|Score|Error|Units|| |original|x|30|4|100|310.203|± 10.585|ms/op| |patched|1000|30|4|100|281.426|± 2.209|ms/op| |patched|200|30|4|100|290.962|± 9.010|ms/op| |original|x|300|4|100|453.817|± 11.402|ms/op| |patched|1000|300|4|100|320.018|± 6.612|ms/op| |patched|200|300|4|100|510.549|± 33.378|ms/op| |original|x|600|4|100|1148.439|± 120.601|ms/op| |patched|1000|600|4|100|445.005|± 22.069|ms/op| |patched|200|600|4|100|904.347|± 126.828|ms/op| |original|x|1200|4|100|1366.978|± 380.341|ms/op| |patched|1000|1200|4|100|662.381|± 16.256|ms/op| |patched|200|1200|4|100|1187.517|± 242.108|ms/op| was (Author: dkuzmenko): MySQL ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|251.485|± 5.201|ms/op| |patched|30|4|100|261.323|± 4.767|ms/op| |original|300|4|100|327.434|± 6.124|ms/op| |patched|300|4|100|338.376|± 3.474|ms/op| |original|600|4|100|472.850|± 9.435|ms/op| |patched|600|4|100|515.727|± 12.683|ms/op| |original|1200|4|100|726.753|± 41.900|ms/op| |patched|1200|4|100|752.484|± 19.530|ms/op| Oracle ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|775.042|± 60.616|ms/op| |patched|30|4|100|434.803|± 38.732|ms/op| |original|300|4|100|11539.710|± 580.909|ms/op| |patched|300|4|100|870.643|± 32.118|ms/op| |original|600|4|100|22671.069|± 949.307|ms/op| |patched|600|4|100|1443.972|± 110.546|ms/op| |original|1200|4|100|47114.679|± 1577.975|ms/op| |patched|1200|4|100|2499.806|± 100.270|ms/op| Postgres ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|310.203|± 10.585|ms/op| |patched|30|4|100|281.426|± 2.209|ms/op| |original|300|4|100|453.817|± 11.402|ms/op| |patched|300|4|100|320.018|± 6.612|ms/op| |original|600|4|100|1148.439|± 120.601|ms/op| |patched|600|4|100|445.005|± 22.069|ms/op| |original|1200|4|100|1366.978|± 380.341|ms/op| |patched|1200|4|100|662.381|± 16.256|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch, HIVE-23032.8.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23083) Enable fast serialization in xprod edge
[
https://issues.apache.org/jira/browse/HIVE-23083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067979#comment-17067979
]
Hive QA commented on HIVE-23083:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 1529 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21284/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21284/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable fast serialization in xprod edge
> ---
>
> Key: HIVE-23083
> URL: https://issues.apache.org/jira/browse/HIVE-23083
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23083.1.patch, Screenshot 2020-03-26 at 2.28.34
> PM.png
>
>
> {noformat}
> select count(*) from store_sales, store, customer, customer_address where
> ss_store_sk = s_store_sk and s_market_id=10 and ss_customer_sk =
> c_customer_sk and c_birth_country <> upper(ca_country);
> {noformat}
> This uses "org/apache/hadoop/io/serializer/WritableSerialization" instead of
> TezBytesWritableSerialization.
>
> !Screenshot 2020-03-26 at 2.28.34 PM.png|width=812,height=488!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23076) Add batching for openTxn
[
https://issues.apache.org/jira/browse/HIVE-23076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067953#comment-17067953
]
Hive QA commented on HIVE-23076:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997860/HIVE-23076.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 18140 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=177)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMigrationNoAutogather.testRetryFailure
(batchId=276)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testRenameTableScenariosWithReplacePolicyDMLOperattion
(batchId=265)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21282/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21282/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21282/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997860 - PreCommit-HIVE-Build
> Add batching for openTxn
>
>
> Key: HIVE-23076
> URL: https://issues.apache.org/jira/browse/HIVE-23076
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23076.2.patch, HIVE-23076.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23080) Clean Up HivePreparedStatement
[ https://issues.apache.org/jira/browse/HIVE-23080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067938#comment-17067938 ] Naveen Gangam commented on HIVE-23080: -- +1. > Clean Up HivePreparedStatement > -- > > Key: HIVE-23080 > URL: https://issues.apache.org/jira/browse/HIVE-23080 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Attachments: HIVE-23080.1.patch > > > * Remove auto-generated code > * Fix check styles warnings -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko edited comment on HIVE-23032 at 3/26/20, 6:29 PM: - MySQL ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|251.485|± 5.201|ms/op| |patched|30|4|100|261.323|± 4.767|ms/op| |original|300|4|100|327.434|± 6.124|ms/op| |patched|300|4|100|338.376|± 3.474|ms/op| |original|600|4|100|472.850|± 9.435|ms/op| |patched|600|4|100|515.727|± 12.683|ms/op| |original|1200|4|100|726.753|± 41.900|ms/op| |patched|1200|4|100|752.484|± 19.530|ms/op| Oracle ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|775.042|± 60.616|ms/op| |patched|30|4|100|434.803|± 38.732|ms/op| |original|300|4|100|11539.710|± 580.909|ms/op| |patched|300|4|100|870.643|± 32.118|ms/op| |original|600|4|100|22671.069|± 949.307|ms/op| |patched|600|4|100|1443.972|± 110.546|ms/op| |original|1200|4|100|47114.679|± 1577.975|ms/op| |patched|1200|4|100|2499.806|± 100.270|ms/op| Postgres ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|310.203|± 10.585|ms/op| |patched|30|4|100|281.426|± 2.209|ms/op| |original|300|4|100|453.817|± 11.402|ms/op| |patched|300|4|100|320.018|± 6.612|ms/op| |original|600|4|100|1148.439|± 120.601|ms/op| |patched|600|4|100|445.005|± 22.069|ms/op| |original|1200|4|100|1366.978|± 380.341|ms/op| |patched|1200|4|100|662.381|± 16.256|ms/op| was (Author: dkuzmenko): MySQL ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|251.485|± 5.201|ms/op| |patched|30|4|100|261.323|± 4.767|ms/op| |original|300|4|100|327.434|± 6.124|ms/op| |patched|300|4|100|338.376|± 3.474|ms/op| |original|600|4|100|472.850|± 9.435|ms/op| |patched|600|4|100|515.727|± 12.683|ms/op| |original|1200|4|100|726.753|± 41.900|ms/op| |patched|1200|4|100|752.484|± 19.530|ms/op| Oracle ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|775.042|± 60.616|ms/op| |patched|30|4|100|434.803|± 38.732|ms/op| |original|300|4|100|11539.710|± 580.909|ms/op| |patched|300|4|100|870.643|± 32.118|ms/op| |original|600|4|100|22671.069|± 949.307|ms/op| |patched|600|4|100|1443.972|± 110.546|ms/op| |original|1200|4|100|47114.679|± 1577.975|ms/op| |patched|1200|4|100|2499.806|± 100.270|ms/op| Postgres ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|||ms/op| |patched|30|4|100|281.426|± 2.209|ms/op| |original|300|4|100|||ms/op| |patched|300|4|100|320.018|± 6.612|ms/op| |original|600|4|100|||ms/op| |patched|600|4|100|445.005|± 22.069|ms/op| |original|1200|4|100|||ms/op| |patched|1200|4|100|662.381|± 16.256|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch, HIVE-23032.8.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko edited comment on HIVE-23032 at 3/26/20, 6:23 PM: - MySQL ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|251.485|± 5.201|ms/op| |patched|30|4|100|261.323|± 4.767|ms/op| |original|300|4|100|327.434|± 6.124|ms/op| |patched|300|4|100|338.376|± 3.474|ms/op| |original|600|4|100|472.850|± 9.435|ms/op| |patched|600|4|100|515.727|± 12.683|ms/op| |original|1200|4|100|726.753|± 41.900|ms/op| |patched|1200|4|100|752.484|± 19.530|ms/op| Oracle ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|775.042|± 60.616|ms/op| |patched|30|4|100|434.803|± 38.732|ms/op| |original|300|4|100|11539.710|± 580.909|ms/op| |patched|300|4|100|870.643|± 32.118|ms/op| |original|600|4|100|22671.069|± 949.307|ms/op| |patched|600|4|100|1443.972|± 110.546|ms/op| |original|1200|4|100|47114.679|± 1577.975|ms/op| |patched|1200|4|100|2499.806|± 100.270|ms/op| Postgres ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|||ms/op| |patched|30|4|100|281.426|± 2.209|ms/op| |original|300|4|100|||ms/op| |patched|300|4|100|320.018|± 6.612|ms/op| |original|600|4|100|||ms/op| |patched|600|4|100|445.005|± 22.069|ms/op| |original|1200|4|100|||ms/op| |patched|1200|4|100|662.381|± 16.256|ms/op| was (Author: dkuzmenko): ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|||ms/op| |patched|30|4|100|261.323|± 4.767|ms/op| |original|300|4|100|||ms/op| |patched|300|4|100|338.376|± 3.474|ms/op| |original|600|4|100|||ms/op| |patched|600|4|100|515.727|± 12.683|ms/op| |original|1200|4|100|||ms/op| |patched|1200|4|100|752.484|±19.530|ms/op| Oracle ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|||ms/op| |patched|30|4|100|434.803|± 38.732|ms/op| |original|300|4|100|||ms/op| |patched|300|4|100|870.643|± 32.118|ms/op| |original|600|4|100|||ms/op| |patched|600|4|100|1443.972|± 110.546|ms/op| |original|1200|4|100|||ms/op| |patched|1200|4|100|2499.806|± 100.270|ms/op| Postgres ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|||ms/op| |patched|30|4|100|281.426|± 2.209|ms/op| |original|300|4|100|||ms/op| |patched|300|4|100|320.018|± 6.612|ms/op| |original|600|4|100|||ms/op| |patched|600|4|100|445.005|± 22.069|ms/op| |original|1200|4|100|||ms/op| |patched|1200|4|100|662.381|± 16.256|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch, HIVE-23032.8.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23086) Two tests fail on branch-2.3
[
https://issues.apache.org/jira/browse/HIVE-23086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-23086:
--
Attachment: HIVE-23086.patch
> Two tests fail on branch-2.3
>
>
> Key: HIVE-23086
> URL: https://issues.apache.org/jira/browse/HIVE-23086
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 2.3.6
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-23086.patch
>
>
> TestPerfCli query88.q and TestMiniLlapLocalCliDriver union_fast_stats.q fail
> on the 2.3 branch.
> The TestMiniLlapLocalCliDriver failure is fixed in HIVE-14977 where
> union_fast_stats.q is removed from the list of tests run with
> TestMiniLlapLocalCliDriver.
> The TestPerfCli failure is fixed HIVE-16602 where one line is added to
> query88.q to allow cartesian products:
> {{set hive.strict.checks.cartesian.product=false;}}
> I propose to backport these two changes to branch-2.3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18983) Add support for table properties inheritance in Create table like
[
https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067920#comment-17067920
]
Sam An commented on HIVE-18983:
---
[~ikryvenko] hope the message finds you well. There are some production use
case for your solution. Would you like to complete this patch? I can see you
have done quite a bit of coding and testing.
> Add support for table properties inheritance in Create table like
> -
>
> Key: HIVE-18983
> URL: https://issues.apache.org/jira/browse/HIVE-18983
> Project: Hive
> Issue Type: Improvement
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-18983.01.patch, HIVE-18983.02.patch,
> HIVE-18983.03.patch, HIVE-18983.04.patch, HIVE-18983.05.patch,
> HIVE-18983.06.patch, HIVE-18983.07.patch, HIVE-18983.08.patch,
> HIVE-18983.09.patch, HIVE-18983.10.patch, HIVE-18983.11.patch,
> HIVE-18983.12.patch
>
>
> Currently, Create table like support table properties.
> But it doesn't inherit table properties from original table.
> {code}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('comment'='comm');
> create table T like T1;
> show create table T;
> {code}
> *Output:*
> {code}
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
> 'maprfs:/user/hive/warehouse/t'
> TBLPROPERTIES (
> 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
> 'numFiles'='0',
> 'numRows'='0',
> 'rawDataSize'='0',
> 'totalSize'='0',
> 'transient_lastDdlTime'='1521230300')
> {code}
> It uses just default table properties and doesn't inherit properties from
> original table.
> It would be great if create table like will inherit origin table properties
> and will override them if they are specified in query .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23086) Two tests fail on branch-2.3
[
https://issues.apache.org/jira/browse/HIVE-23086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-23086:
--
Status: Patch Available (was: Open)
Attached patch to backport select changes from HIVE-14977 and HIVE-16602
> Two tests fail on branch-2.3
>
>
> Key: HIVE-23086
> URL: https://issues.apache.org/jira/browse/HIVE-23086
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 2.3.6
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-23086.patch
>
>
> TestPerfCli query88.q and TestMiniLlapLocalCliDriver union_fast_stats.q fail
> on the 2.3 branch.
> The TestMiniLlapLocalCliDriver failure is fixed in HIVE-14977 where
> union_fast_stats.q is removed from the list of tests run with
> TestMiniLlapLocalCliDriver.
> The TestPerfCli failure is fixed HIVE-16602 where one line is added to
> query88.q to allow cartesian products:
> {{set hive.strict.checks.cartesian.product=false;}}
> I propose to backport these two changes to branch-2.3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22098) Data loss occurs when multiple tables are join with different bucket_version
[
https://issues.apache.org/jira/browse/HIVE-22098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067919#comment-17067919
]
Sam An commented on HIVE-22098:
---
[~luguangming], hope the message finds you well. There are some production use
case for your solution. Would you like to complete this patch?
> Data loss occurs when multiple tables are join with different bucket_version
>
>
> Key: HIVE-22098
> URL: https://issues.apache.org/jira/browse/HIVE-22098
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Assignee: LuGuangMing
>Priority: Blocker
> Labels: data-loss, wrongresults
> Attachments: HIVE-22098.1.patch, image-2019-08-12-18-45-15-771.png,
> join_test.sql, table_a_data.orc, table_b_data.orc, table_c_data.orc
>
>
> When different bucketVersion of tables do join and no of reducers is greater
> than 2, the result is incorrect (*data loss*).
> *Scenario 1*: Three tables join. The temporary result data of table_a in the
> first table and table_b in the second table joins result is recorded as
> tmp_a_b, When it joins with the third table, the bucket_version=2 of the
> table created by default after hive-3.0.0, temporary data tmp_a_b initialized
> the bucketVerison=-1, and then ReduceSinkOperator Verketison=-1 is joined. In
> the init method, the hash algorithm of selecting join column is selected
> according to bucketVersion. If bucketVersion = 2 and is not an acid
> operation, it will acquired the new algorithm of hash. Otherwise, the old
> algorithm of hash is acquired. Because of the inconsistency of the algorithm
> of hash, the partition of data allocation caused are different. At stage of
> Reducer, Data with the same key can not be paired resulting in data loss.
> *Scenario 2*: create two test tables, create table
> table_bucketversion_1(col_1 string, col_2 string) TBLPROPERTIES
> ('bucketing_version'='1'); table_bucketversion_2(col_1 string, col_2 string)
> TBLPROPERTIES ('bucketing_version'='2');
> when use table_bucketversion_1 to join table_bucketversion_2, partial result
> data will be loss due to bucketVerison is different.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23086) Two tests fail on branch-2.3
[
https://issues.apache.org/jira/browse/HIVE-23086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates reassigned HIVE-23086:
-
> Two tests fail on branch-2.3
>
>
> Key: HIVE-23086
> URL: https://issues.apache.org/jira/browse/HIVE-23086
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 2.3.6
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>
> TestPerfCli query88.q and TestMiniLlapLocalCliDriver union_fast_stats.q fail
> on the 2.3 branch.
> The TestMiniLlapLocalCliDriver failure is fixed in HIVE-14977 where
> union_fast_stats.q is removed from the list of tests run with
> TestMiniLlapLocalCliDriver.
> The TestPerfCli failure is fixed HIVE-16602 where one line is added to
> query88.q to allow cartesian products:
> {{set hive.strict.checks.cartesian.product=false;}}
> I propose to backport these two changes to branch-2.3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23076) Add batching for openTxn
[
https://issues.apache.org/jira/browse/HIVE-23076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067893#comment-17067893
]
Hive QA commented on HIVE-23076:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
17s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 185 unchanged - 1 fixed = 185 total (was 186) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21282/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21282/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add batching for openTxn
>
>
> Key: HIVE-23076
> URL: https://issues.apache.org/jira/browse/HIVE-23076
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23076.2.patch, HIVE-23076.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23080) Clean Up HivePreparedStatement
[ https://issues.apache.org/jira/browse/HIVE-23080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067889#comment-17067889 ] David Mollitor commented on HIVE-23080: --- [~ngangam] Review? Should be pretty trivial though it's a lot of code. I just changed formatting and got rid of comments. > Clean Up HivePreparedStatement > -- > > Key: HIVE-23080 > URL: https://issues.apache.org/jira/browse/HIVE-23080 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Attachments: HIVE-23080.1.patch > > > * Remove auto-generated code > * Fix check styles warnings -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23085) LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
[
https://issues.apache.org/jira/browse/HIVE-23085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067877#comment-17067877
]
Gopal Vijayaraghavan commented on HIVE-23085:
-
Here's the actual implementation known to work instead of using multiple
independent disks (on-prem the independent disks are marginally better, since
we can keep limping using partial hardware - on-cloud, it is better to give up
and get a better box ASAP).
{code}
# mount nvme SSDs as LVM to /data when available for specific
instance types
# There are max 4 nvme slots in r5 instances. /dev/nvme0n1 is root
EBS volume.
# concat all existing nvme slots to logical volume creation
#
https://stackoverflow.com/questions/44458304/how-to-escape-in-cloudformations-fnsub
if [ -e /dev/nvme1n1 ]
then
for i in $(seq 1 4); do
if [ -e /dev/nvme${!i}n1 ]
then
x+=" /dev/nvme${!i}n1"
fi
done
yes | sudo pvcreate $x
yes | sudo vgcreate nvmevg $x
yes | sudo lvcreate -l 100%FREE nvmevg -n nvmelv
yes | sudo mkfs.ext4 /dev/nvmevg/nvmelv
sudo mkdir -p /data
sudo mount /dev/nvmevg/nvmelv /data
fi
{code}
> LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
>
>
> Key: HIVE-23085
> URL: https://issues.apache.org/jira/browse/HIVE-23085
> Project: Hive
> Issue Type: Improvement
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23085.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently we can configure only one SSD location while using SSD cache in
> LLAP. This highly undermines the capacity of some machines to use its disk
> capacity to the fullest. For example *AWS* provides *r5d.4x large* series
> which comes with *2 * 300 GB NVme SSD disk* with the current design only one
> of the mounted *NVme SSD* disk can be used for caching. Hence adding support
> for caching data at multiple ssd mounted locations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22842) Timestamp/date vectors in Arrow serializer should use correct calendar for value representation
[
https://issues.apache.org/jira/browse/HIVE-22842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067874#comment-17067874
]
Hive QA commented on HIVE-22842:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997786/HIVE-22842.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18161 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead
(batchId=292)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21281/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21281/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21281/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997786 - PreCommit-HIVE-Build
> Timestamp/date vectors in Arrow serializer should use correct calendar for
> value representation
> ---
>
> Key: HIVE-22842
> URL: https://issues.apache.org/jira/browse/HIVE-22842
> Project: Hive
> Issue Type: Improvement
>Reporter: Jesus Camacho Rodriguez
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22842.01.patch, HIVE-22842.02.patch,
> HIVE-22842.03.patch, HIVE-22842.04.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23052: -- Attachment: HIVE-23052.7.patch > Optimize lock enqueueing in TxnHandler > -- > > Key: HIVE-23052 > URL: https://issues.apache.org/jira/browse/HIVE-23052 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23052.1.patch, HIVE-23052.2.patch, > HIVE-23052.3.patch, HIVE-23052.4.patch, HIVE-23052.5.patch, > HIVE-23052.6.patch, HIVE-23052.7.patch > > > * Reduce scope of next_lock_id select-for-update by moving the txn_component > inserts before the S4U + inserting the hive_locks entries before the S4U > (first with a temp ID, which will be replaced later in a single update). This > helps decrease the overall time that the next_lock_id table is locked, > thereby increasing concurrency > * Insert txn_components in a batch instead of one-by-one (also in > TxnHandler::addDynamicPartition) > * Increment next_lock_id and update hive_locks table in a single batch > statement > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Attachment: HIVE-22785.13.patch
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.10.patch,
> HIVE-22785.11.patch, HIVE-22785.12.patch, HIVE-22785.13.patch,
> HIVE-22785.2.patch, HIVE-22785.2.patch, HIVE-22785.3.patch,
> HIVE-22785.4.patch, HIVE-22785.5.patch, HIVE-22785.6.patch,
> HIVE-22785.7.patch, HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Open (was: Patch Available)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.10.patch,
> HIVE-22785.11.patch, HIVE-22785.12.patch, HIVE-22785.13.patch,
> HIVE-22785.2.patch, HIVE-22785.2.patch, HIVE-22785.3.patch,
> HIVE-22785.4.patch, HIVE-22785.5.patch, HIVE-22785.6.patch,
> HIVE-22785.7.patch, HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Patch Available (was: Open)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.10.patch,
> HIVE-22785.11.patch, HIVE-22785.12.patch, HIVE-22785.13.patch,
> HIVE-22785.2.patch, HIVE-22785.2.patch, HIVE-22785.3.patch,
> HIVE-22785.4.patch, HIVE-22785.5.patch, HIVE-22785.6.patch,
> HIVE-22785.7.patch, HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23032: -- Attachment: HIVE-23032.8.patch > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch, HIVE-23032.8.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22842) Timestamp/date vectors in Arrow serializer should use correct calendar for value representation
[
https://issues.apache.org/jira/browse/HIVE-22842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067820#comment-17067820
]
Hive QA commented on HIVE-22842:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} storage-api in master has 51 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 1529 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} storage-api: The patch generated 1 new + 3 unchanged -
0 fixed = 4 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 210 unchanged - 2
fixed = 211 total (was 212) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21281/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21281/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21281/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21281/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: storage-api common ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21281/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Timestamp/date vectors in Arrow serializer should use correct calendar for
> value representation
>
[jira] [Commented] (HIVE-23085) LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
[ https://issues.apache.org/jira/browse/HIVE-23085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067801#comment-17067801 ] Gopal Vijayaraghavan commented on HIVE-23085: - This was originally done and it caused serious stability issues, when one of the disks filled up and other didn't - the full disk throws a SIGBUS error. bq. *AWS* provides *r5d.4x large* series which comes with *2 * 300 GB NVme SSD disk* That is the exact hardware I have in prod, which uses an LVM volume mount as /grid/ssd instead of 2 disk balancing problems. > LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache > > > Key: HIVE-23085 > URL: https://issues.apache.org/jira/browse/HIVE-23085 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23085.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently we can configure only one SSD location while using SSD cache in > LLAP. This highly undermines the capacity of some machines to use its disk > capacity to the fullest. For example *AWS* provides *r5d.4x large* series > which comes with *2 * 300 GB NVme SSD disk* with the current design only one > of the mounted *NVme SSD* disk can be used for caching. Hence adding support > for caching data at multiple ssd mounted locations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067797#comment-17067797
]
Hive QA commented on HIVE-22995:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/1299/HIVE-22995.11.patch
{color:green}SUCCESS:{color} +1 due to 10 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 18145 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=196)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_materialized]
(batchId=184)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterTable
(batchId=233)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testAlterTable
(batchId=235)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreCustomAuth.testAlterTable
(batchId=239)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTable
(batchId=234)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTable
(batchId=241)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterTable
(batchId=231)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testAlterTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testAlterTable
(batchId=240)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testRenamePartitionedTableAcrossDatabases
(batchId=261)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testRenameTableAcrossDatabases
(batchId=261)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenamePartitionedTableAcrossDatabases
(batchId=270)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenameTableAcrossDatabases
(batchId=270)
org.apache.hive.service.server.TestInformationSchemaWithPrivilegeZookeeperPlain.test
(batchId=290)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21280/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21280/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21280/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 1299 - PreCommit-HIVE-Build
> Add support for location for managed tables on database
> ---
>
> Key: HIVE-22995
> URL: https://issues.apache.org/jira/browse/HIVE-22995
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22995.1.patch, HIVE-22995.10.patch,
> HIVE-22995.11.patch, HIVE-22995.2.patch, HIVE-22995.3.patch,
> HIVE-22995.4.patch, HIVE-22995.5.patch, HIVE-22995.6.patch,
> HIVE-22995.7.patch, HIVE-22995.8.patch, HIVE-22995.9.patch, Hive Metastore
> Support for Tenant-based storage heirarchy.pdf
>
>
> I have attached the initial spec to this jira.
> Default location for database would be the external table base directory.
> Managed location can be optionally specified.
> {code}
> CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
> [COMMENT database_comment]
> [LOCATION hdfs_path]
> [MANAGEDLOCATION hdfs_path]
> [WITH DBPROPERTIES (property_name=property_value, ...)];
> ALTER (DATABASE|SCHEMA) database_name SET
> MANAGEDLOCATION
> hdfs_path;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23078) Remove HiveDriver SecurityManager Check
[
https://issues.apache.org/jira/browse/HIVE-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067794#comment-17067794
]
Naveen Gangam commented on HIVE-23078:
--
whoa .. this code has existed for over 7 years now via HIVE-2935 (part of some
intial core changes for the driver). Not clear what the original intent of it
was. My guess is to check if the application had write access on the file
system, maybe to be able to write some tmp files?
[~ashutoshc] You were one of the reviewers. Do you have any context to this?
Thanks
> Remove HiveDriver SecurityManager Check
> ---
>
> Key: HIVE-23078
> URL: https://issues.apache.org/jira/browse/HIVE-23078
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23078.1.patch
>
>
> {code:java|title=HiveDriver.java}
> public HiveDriver() {
> // TODO Auto-generated constructor stub
> SecurityManager security = System.getSecurityManager();
> if (security != null) {
> security.checkWrite("foobah");
> }
> }
> {code}
> Not sure why it needs to write a file called "foobah" but I checked out some
> other JDBC drivers and they do nothing like this. Remove this check; remove
> the constructor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko edited comment on HIVE-23032 at 3/26/20, 3:36 PM: - ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|||ms/op| |patched|30|4|100|261.323|± 4.767|ms/op| |original|300|4|100|||ms/op| |patched|300|4|100|338.376|± 3.474|ms/op| |original|600|4|100|||ms/op| |patched|600|4|100|515.727|± 12.683|ms/op| |original|1200|4|100|||ms/op| |patched|1200|4|100|752.484|±19.530|ms/op| Oracle ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|||ms/op| |patched|30|4|100|434.803|± 38.732|ms/op| |original|300|4|100|||ms/op| |patched|300|4|100|870.643|± 32.118|ms/op| |original|600|4|100|||ms/op| |patched|600|4|100|1443.972|± 110.546|ms/op| |original|1200|4|100|||ms/op| |patched|1200|4|100|2499.806|± 100.270|ms/op| Postgres ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |original|30|4|100|||ms/op| |patched|30|4|100|281.426|± 2.209|ms/op| |original|300|4|100|||ms/op| |patched|300|4|100|320.018|± 6.612|ms/op| |original|600|4|100|||ms/op| |patched|600|4|100|445.005|± 22.069|ms/op| |original|1200|4|100|||ms/op| |patched|1200|4|100|662.381|± 16.256|ms/op| was (Author: dkuzmenko): ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |patched|30|4|100|261.323|± 4.767|ms/op| |patched|300|4|100|338.376|± 3.474|ms/op| |patched|600|4|100|515.727|± 12.683|ms/op| |patched|1200|4|100|752.484|±19.530|ms/op| Oracle ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |patched|30|4|100|434.803|± 38.732|ms/op| |patched|300|4|100|870.643|± 32.118|ms/op| |patched|600|4|100|1443.972|± 110.546|ms/op| |patched|1200|4|100|2499.806|± 100.270|ms/op| Postgres ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |patched|30|4|100|281.426|± 2.209|ms/op| |patched|300|4|100|320.018|± 6.612|ms/op| |patched|600|4|100|445.005|± 22.069|ms/op| |patched|1200|4|100|662.381|± 16.256|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23082) PK/FK stat rescale doesn't work in some cases
[
https://issues.apache.org/jira/browse/HIVE-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067773#comment-17067773
]
Zoltan Haindrich commented on HIVE-23082:
-
this boils down to something going wrong with the stat computation; a bigint
column with values in [1:70] has 73 distinct values :D
{code}
data_type bigint
min1
max70
num_nulls 72
distinct_count 73
{code}
> PK/FK stat rescale doesn't work in some cases
> -
>
> Key: HIVE-23082
> URL: https://issues.apache.org/jira/browse/HIVE-23082
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> As a result in Joins may retain the original estimate; see MAPJOIN_33 in this
> plan ; which retained the estimate of SEL_32
> {code}
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Map 1 <- Map 2 (BROADCAST_EDGE)|
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:12 |
> | Stage-1|
> | Map 1 vectorized |
> | File Output Operator [FS_36] |
> | Limit [LIM_35] (rows=12 width=4) |
> | Number of rows:12|
> | Select Operator [SEL_34] (rows=5040 width=4) |
> | Output:["_col0"] |
> | Map Join Operator [MAPJOIN_33] (rows=5040 width=8) |
> | Conds:SEL_32._col0=RS_30._col0(Inner) |
> | <-Map 2 [BROADCAST_EDGE] vectorized|
> | BROADCAST [RS_30]|
> | PartitionCols:_col0|
> | Select Operator [SEL_29] (rows=1 width=8) |
> | Output:["_col0"] |
> | Filter Operator [FIL_28] (rows=1 width=108) |
> | predicate:((r_reason_id = 'reason 66') and r_reason_sk
> is not null) |
> | TableScan [TS_3] (rows=2 width=108) |
> |
> default@rx0,reason,Tbl:COMPLETE,Col:COMPLETE,Output:["r_reason_id","r_reason_sk"]
> |
> | <-Select Operator [SEL_32] (rows=5040 width=7) |
> | Output:["_col0"] |
> | Filter Operator [FIL_31] (rows=5040 width=7) |
> | predicate:sr_reason_sk is not null |
> | TableScan [TS_0] (rows=5112 width=7) |
> |
> default@sr0,store_returns,Tbl:COMPLETE,Col:COMPLETE,Output:["sr_reason_sk"] |
> ||
> ++
> {code}
> repro:
> {code}
> set hive.query.results.cache.enabled=false;
> set hive.explain.user=true;
> drop table if exists default.rx0;
> drop table if exists default.sr0;
> create table rx0 (r_reason_id string, r_reason_sk bigint);
> create table sr0 (sr_reason_sk bigint);
> insert into rx0 values ('',1),('GEAA',70);
> insert into sr0 values (NULL),(1),(2),(3),(4),(5),(6),(7),(8),(9),(10),
> (11),(12),(13),(14),(15),(16),(17),(18),(19),(20),(21),(22),(23),(24),(25),
> (26),(27),(28),(29),(30),(31),(32),(33),(34),(35),(36),(37),(38),(39),(40),
> (41),(42),(43),(44),(45),(46),(47),(48),(49),(50),(51),(52),(53),(54),(55),
> (56),(57),(58),(59),(60),(61),(62),(63),(64),(65),(66),(67),(68),(69),(70);
> insert into sr0 select a.* from sr0 a,sr0 b;
> -- |sr0| ~ 5112
> explain select 1
> from default.sr0 store_returns , default.rx0 reason
> where sr_reason_sk = r_reason_sk
> and r_reason_id = 'reason 66'
> limit 12;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067771#comment-17067771
]
Hive QA commented on HIVE-22995:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
44s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
55s{color} | {color:blue} parser in master has 3 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 1529 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} hcatalog/core in master has 37 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 1 new + 98 unchanged - 0 fixed = 99 total (was 98) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
29s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 23 new + 1266 unchanged - 9 fixed = 1289 total (was 1275) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 8 new + 161 unchanged - 0
fixed = 169 total (was 161) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} itests/hive-unit: The patch generated 34 new + 285
unchanged - 0 fixed = 319 total (was 285) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21280/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Updated] (HIVE-22534) ACID: Improve Compactor thread logging
[ https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Pintér updated HIVE-22534: - Attachment: HIVE-22534.11.patch > ACID: Improve Compactor thread logging > -- > > Key: HIVE-22534 > URL: https://issues.apache.org/jira/browse/HIVE-22534 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Minor > Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch, > HIVE-22534.03.patch, HIVE-22534.04.patch, HIVE-22534.05.patch, > HIVE-22534.06.patch, HIVE-22534.07.patch, HIVE-22534.08.patch, > HIVE-22534.09.patch, HIVE-22534.10.patch, HIVE-22534.11.patch > > > Make sure that it is easy to find issues when one of the compactor thread > fails. > Maybe: > * MDC - with iteration / threadname - so we can easily grep the logs for a > given run > * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23046) Separate housekeeping thread from initiator flag
[ https://issues.apache.org/jira/browse/HIVE-23046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Pintér updated HIVE-23046: - Attachment: HIVE-23046.01.patch > Separate housekeeping thread from initiator flag > > > Key: HIVE-23046 > URL: https://issues.apache.org/jira/browse/HIVE-23046 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Attachments: HIVE-23046.01.patch > > > If hive.compactor.initiator.on=false, the housekeeping thread is not started, > therefore certain resources are not cleaned up. HK thread should be > configurable from another parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23046) Separate housekeeping thread from initiator flag
[ https://issues.apache.org/jira/browse/HIVE-23046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Pintér updated HIVE-23046: - Status: Patch Available (was: Open) > Separate housekeeping thread from initiator flag > > > Key: HIVE-23046 > URL: https://issues.apache.org/jira/browse/HIVE-23046 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Attachments: HIVE-23046.01.patch > > > If hive.compactor.initiator.on=false, the housekeeping thread is not started, > therefore certain resources are not cleaned up. HK thread should be > configurable from another parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko edited comment on HIVE-23032 at 3/26/20, 3:07 PM: - ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |patched|30|4|100|261.323|± 4.767|ms/op| |patched|300|4|100|338.376|± 3.474|ms/op| |patched|600|4|100|515.727|± 12.683|ms/op| |patched|1200|4|100|752.484|±19.530|ms/op| Oracle ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |patched|30|4|100|434.803|± 38.732|ms/op| |patched|300|4|100|870.643|± 32.118|ms/op| |patched|600|4|100|1443.972|± 110.546|ms/op| |patched|1200|4|100|2499.806|± 100.270|ms/op| Postgres ||Mode|partCount|tblCount|Cnt|Score|Error|Units|| |patched|30|4|100|281.426|± 2.209|ms/op| |patched|300|4|100|320.018|± 6.612|ms/op| |patched|600|4|100|445.005|± 22.069|ms/op| |patched|1200|4|100|662.381|± 16.256|ms/op| was (Author: dkuzmenko): ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |Oracle|patched|30|4|100|1038.296|± 36.946|ms/op| |Oracle|patched|300|4|100|2670.009|± 135.603|ms/op| |Oracle|patched|600|4|100|4657.155|± 280.535|ms/op| |Oracle|patched|1200|4|100|8740.191|± 322.856|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22760) Add Clock caching eviction based strategy
[ https://issues.apache.org/jira/browse/HIVE-22760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22760: -- Attachment: HIVE-22760.3.patch > Add Clock caching eviction based strategy > - > > Key: HIVE-22760 > URL: https://issues.apache.org/jira/browse/HIVE-22760 > Project: Hive > Issue Type: New Feature > Components: llap >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22760.2.patch, HIVE-22760.3.patch, > HIVE-22760.3.patch, HIVE-22760.patch, HIVE-22760.patch > > > LRFU is the current default right now. > The main issue with such Strategy is that it has a very high memory overhead, > in addition to that, most of the accounting has to happen under locks thus > can be source of contentions. > Add Simpler policy like clock, can help with both issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23073) Upgrade to netty 4.1.48.Final
[ https://issues.apache.org/jira/browse/HIVE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23073: Attachment: HIVE-23073.02.patch > Upgrade to netty 4.1.48.Final > - > > Key: HIVE-23073 > URL: https://issues.apache.org/jira/browse/HIVE-23073 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23073.01.patch, HIVE-23073.02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23073) Upgrade to netty 4.1.48.Final
[ https://issues.apache.org/jira/browse/HIVE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23073: Attachment: (was: HIVE-23073.02.patch) > Upgrade to netty 4.1.48.Final > - > > Key: HIVE-23073 > URL: https://issues.apache.org/jira/browse/HIVE-23073 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23073.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23080) Clean Up HivePreparedStatement
[
https://issues.apache.org/jira/browse/HIVE-23080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067713#comment-17067713
]
Hive QA commented on HIVE-23080:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997769/HIVE-23080.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18140 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21279/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21279/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21279/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997769 - PreCommit-HIVE-Build
> Clean Up HivePreparedStatement
> --
>
> Key: HIVE-23080
> URL: https://issues.apache.org/jira/browse/HIVE-23080
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23080.1.patch
>
>
> * Remove auto-generated code
> * Fix check styles warnings
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23073) Upgrade to netty 4.1.48.Final
[ https://issues.apache.org/jira/browse/HIVE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23073: Attachment: HIVE-23073.02.patch > Upgrade to netty 4.1.48.Final > - > > Key: HIVE-23073 > URL: https://issues.apache.org/jira/browse/HIVE-23073 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23073.01.patch, HIVE-23073.02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23073) Upgrade to netty 4.1.48.Final
[ https://issues.apache.org/jira/browse/HIVE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23073: Attachment: (was: HIVE-23073.02.patch) > Upgrade to netty 4.1.48.Final > - > > Key: HIVE-23073 > URL: https://issues.apache.org/jira/browse/HIVE-23073 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23073.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23073) Upgrade to netty 4.1.48.Final
[ https://issues.apache.org/jira/browse/HIVE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23073: Attachment: HIVE-23073.02.patch > Upgrade to netty 4.1.48.Final > - > > Key: HIVE-23073 > URL: https://issues.apache.org/jira/browse/HIVE-23073 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23073.01.patch, HIVE-23073.02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state
[
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pablo Junge updated HIVE-22566:
---
Attachment: (was: HIVE-22566.04.patch)
> Drop table involved in materialized view leaves the table in inconsistent
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.05.patch,
> HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created
> materialized view, the table is not dropped, which is the desired state as it
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did
> not succeed, leaving it in an inconsistent state.
>
> Repro:
> ---
> 1) Create tables:
>
> {code:java}
> CREATE TABLE emps ( empid INT, deptno INT, name VARCHAR(256), salary
> FLOAT, hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES
> ('transactional'='true');
> CREATE TABLE depts ( deptno INT, deptname VARCHAR(256), locationid
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>
> 2) Create the VM:
>
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps
> JOIN depts ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>
> 3) Following is in backend database at this point:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | 81 | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>
> 4) Let's drop the 'emps' table:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO : Compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 0.05 seconds
> INFO : Executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 10.281 seconds
> INFO : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>
> 5) List tables:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO : Compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.041 seconds
> INFO : Executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.016 seconds
> INFO : OK
> +---+
> | tab_name |
> +---+
> | depts |
> | emps |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>
> 6) Now, from the backend-db point of view:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | NULL | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>
> 7) From Metastore.log
>
> {code:java}
>
[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state
[
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pablo Junge updated HIVE-22566:
---
Attachment: (was: HIVE-22566.03.patch)
> Drop table involved in materialized view leaves the table in inconsistent
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.05.patch,
> HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created
> materialized view, the table is not dropped, which is the desired state as it
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did
> not succeed, leaving it in an inconsistent state.
>
> Repro:
> ---
> 1) Create tables:
>
> {code:java}
> CREATE TABLE emps ( empid INT, deptno INT, name VARCHAR(256), salary
> FLOAT, hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES
> ('transactional'='true');
> CREATE TABLE depts ( deptno INT, deptname VARCHAR(256), locationid
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>
> 2) Create the VM:
>
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps
> JOIN depts ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>
> 3) Following is in backend database at this point:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | 81 | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>
> 4) Let's drop the 'emps' table:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO : Compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 0.05 seconds
> INFO : Executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 10.281 seconds
> INFO : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>
> 5) List tables:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO : Compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.041 seconds
> INFO : Executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.016 seconds
> INFO : OK
> +---+
> | tab_name |
> +---+
> | depts |
> | emps |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>
> 6) Now, from the backend-db point of view:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | NULL | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>
> 7) From Metastore.log
>
> {code:java}
>
[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state
[
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pablo Junge updated HIVE-22566:
---
Status: Open (was: Patch Available)
> Drop table involved in materialized view leaves the table in inconsistent
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.03.patch,
> HIVE-22566.04.patch, HIVE-22566.05.patch, HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created
> materialized view, the table is not dropped, which is the desired state as it
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did
> not succeed, leaving it in an inconsistent state.
>
> Repro:
> ---
> 1) Create tables:
>
> {code:java}
> CREATE TABLE emps ( empid INT, deptno INT, name VARCHAR(256), salary
> FLOAT, hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES
> ('transactional'='true');
> CREATE TABLE depts ( deptno INT, deptname VARCHAR(256), locationid
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>
> 2) Create the VM:
>
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps
> JOIN depts ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>
> 3) Following is in backend database at this point:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | 81 | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>
> 4) Let's drop the 'emps' table:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO : Compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 0.05 seconds
> INFO : Executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 10.281 seconds
> INFO : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>
> 5) List tables:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO : Compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.041 seconds
> INFO : Executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.016 seconds
> INFO : OK
> +---+
> | tab_name |
> +---+
> | depts |
> | emps |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>
> 6) Now, from the backend-db point of view:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | NULL | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>
> 7) From
[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state
[
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pablo Junge updated HIVE-22566:
---
Attachment: HIVE-22566.05.patch
Status: Patch Available (was: Open)
> Drop table involved in materialized view leaves the table in inconsistent
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.03.patch,
> HIVE-22566.04.patch, HIVE-22566.05.patch, HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created
> materialized view, the table is not dropped, which is the desired state as it
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did
> not succeed, leaving it in an inconsistent state.
>
> Repro:
> ---
> 1) Create tables:
>
> {code:java}
> CREATE TABLE emps ( empid INT, deptno INT, name VARCHAR(256), salary
> FLOAT, hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES
> ('transactional'='true');
> CREATE TABLE depts ( deptno INT, deptname VARCHAR(256), locationid
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>
> 2) Create the VM:
>
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps
> JOIN depts ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>
> 3) Following is in backend database at this point:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | 81 | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>
> 4) Let's drop the 'emps' table:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO : Compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 0.05 seconds
> INFO : Executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 10.281 seconds
> INFO : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>
> 5) List tables:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO : Compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.041 seconds
> INFO : Executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.016 seconds
> INFO : OK
> +---+
> | tab_name |
> +---+
> | depts |
> | emps |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>
> 6) Now, from the backend-db point of view:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | NULL | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in
[jira] [Updated] (HIVE-23085) LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
[ https://issues.apache.org/jira/browse/HIVE-23085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-23085: - Attachment: HIVE-23085.01.patch Status: Patch Available (was: Open) > LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache > > > Key: HIVE-23085 > URL: https://issues.apache.org/jira/browse/HIVE-23085 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23085.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently we can configure only one SSD location while using SSD cache in > LLAP. This highly undermines the capacity of some machines to use its disk > capacity to the fullest. For example *AWS* provides *r5d.4x large* series > which comes with *2 * 300 GB NVme SSD disk* with the current design only one > of the mounted *NVme SSD* disk can be used for caching. Hence adding support > for caching data at multiple ssd mounted locations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23085) LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
[ https://issues.apache.org/jira/browse/HIVE-23085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23085: -- Labels: pull-request-available (was: ) > LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache > > > Key: HIVE-23085 > URL: https://issues.apache.org/jira/browse/HIVE-23085 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Currently we can configure only one SSD location while using SSD cache in > LLAP. This highly undermines the capacity of some machines to use its disk > capacity to the fullest. For example *AWS* provides *r5d.4x large* series > which comes with *2 * 300 GB NVme SSD disk* with the current design only one > of the mounted *NVme SSD* disk can be used for caching. Hence adding support > for caching data at multiple ssd mounted locations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23085) LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
[ https://issues.apache.org/jira/browse/HIVE-23085?focusedWorklogId=410267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410267 ] ASF GitHub Bot logged work on HIVE-23085: - Author: ASF GitHub Bot Created on: 26/Mar/20 13:42 Start Date: 26/Mar/20 13:42 Worklog Time Spent: 10m Work Description: shameersss1 commented on pull request #962: HIVE-23085: Support Multiple NVMe-SSD disk Locations While Using SSD Cache In LLAP URL: https://github.com/apache/hive/pull/962 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 410267) Remaining Estimate: 0h Time Spent: 10m > LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache > > > Key: HIVE-23085 > URL: https://issues.apache.org/jira/browse/HIVE-23085 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently we can configure only one SSD location while using SSD cache in > LLAP. This highly undermines the capacity of some machines to use its disk > capacity to the fullest. For example *AWS* provides *r5d.4x large* series > which comes with *2 * 300 GB NVme SSD disk* with the current design only one > of the mounted *NVme SSD* disk can be used for caching. Hence adding support > for caching data at multiple ssd mounted locations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23078) Remove HiveDriver SecurityManager Check
[
https://issues.apache.org/jira/browse/HIVE-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067684#comment-17067684
]
David Mollitor commented on HIVE-23078:
---
[~ngangam] Any thoughts on this?
> Remove HiveDriver SecurityManager Check
> ---
>
> Key: HIVE-23078
> URL: https://issues.apache.org/jira/browse/HIVE-23078
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23078.1.patch
>
>
> {code:java|title=HiveDriver.java}
> public HiveDriver() {
> // TODO Auto-generated constructor stub
> SecurityManager security = System.getSecurityManager();
> if (security != null) {
> security.checkWrite("foobah");
> }
> }
> {code}
> Not sure why it needs to write a file called "foobah" but I checked out some
> other JDBC drivers and they do nothing like this. Remove this check; remove
> the constructor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23077) Remove Calls to printStackTrace in Module hive-jdbc
[
https://issues.apache.org/jira/browse/HIVE-23077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23077:
--
Attachment: HIVE-23077.1.patch
> Remove Calls to printStackTrace in Module hive-jdbc
> ---
>
> Key: HIVE-23077
> URL: https://issues.apache.org/jira/browse/HIVE-23077
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23077.1.patch, HIVE-23077.1.patch
>
>
> Only one "tricky" change. Throw an Exception instead of {{printStackTrace}}
> in the static Driver loader as suggested from the reference here:
> https://github.com/mariadb-corporation/mariadb-connector-j/blob/3bc66153b51aca188afc50ff35a0123f16c099ed/src/main/java/org/mariadb/jdbc/Driver.java#L72
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23079) Remove Calls to printStackTrace in Module hive-serde
[ https://issues.apache.org/jira/browse/HIVE-23079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23079: -- Attachment: HIVE-23079.1.patch > Remove Calls to printStackTrace in Module hive-serde > > > Key: HIVE-23079 > URL: https://issues.apache.org/jira/browse/HIVE-23079 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23079.1.patch, HIVE-23079.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23085) LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
[ https://issues.apache.org/jira/browse/HIVE-23085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman reassigned HIVE-23085: > LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache > > > Key: HIVE-23085 > URL: https://issues.apache.org/jira/browse/HIVE-23085 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Fix For: 4.0.0 > > > Currently we can configure only one SSD location while using SSD cache in > LLAP. This highly undermines the capacity of some machines to use its disk > capacity to the fullest. For example *AWS* provides *r5d.4x large* series > which comes with *2 * 300 GB NVme SSD disk* with the current design only one > of the mounted *NVme SSD* disk can be used for caching. Hence adding support > for caching data at multiple ssd mounted locations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko edited comment on HIVE-23032 at 3/26/20, 1:24 PM: - ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |Oracle|patched|30|4|100|1038.296|± 36.946|ms/op| |Oracle|patched|300|4|100|2670.009|± 135.603|ms/op| |Oracle|patched|600|4|100|4657.155|± 280.535|ms/op| |Oracle|patched|1200|4|100|8740.191|± 322.856|ms/op| was (Author: dkuzmenko): ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |Oracle|patched|30|4|100|1038.296|± 36.946|ms/op| |Oracle|patched|300|4|100|2670.009|± 135.603|ms/op| |Oracle|patched|600|4|100|4657.155|± 280.535|ms/op| |Oracle|patched|1200|4|100|8740.191|± 322.856|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko edited comment on HIVE-23032 at 3/26/20, 1:23 PM: - ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| |Oracle|patched|30|4|100|1038.296|± 36.946|ms/op| |Oracle|patched|300|4|100|2670.009|± 135.603|ms/op| |Oracle|patched|600|4|100|4657.155|± 280.535|ms/op| |Oracle|patched|1200|4|100|8740.191|± 322.856|ms/op| was (Author: dkuzmenko): ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| |Oracle|patched|30|4|100|1038.296|± 36.946|ms/op| |Oracle|patched|300|4|100|2670.009|± 135.603|ms/op| |Oracle|patched|600|4|100|4657.155|± 280.535|ms/op| |Oracle|patched|1200|4|100|8740.191|± 322.856|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko edited comment on HIVE-23032 at 3/26/20, 1:23 PM: - ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |Oracle|patched|30|4|100|1038.296|± 36.946|ms/op| |Oracle|patched|300|4|100|2670.009|± 135.603|ms/op| |Oracle|patched|600|4|100|4657.155|± 280.535|ms/op| |Oracle|patched|1200|4|100|8740.191|± 322.856|ms/op| was (Author: dkuzmenko): ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| |Oracle|patched|30|4|100|1038.296|± 36.946|ms/op| |Oracle|patched|300|4|100|2670.009|± 135.603|ms/op| |Oracle|patched|600|4|100|4657.155|± 280.535|ms/op| |Oracle|patched|1200|4|100|8740.191|± 322.856|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko edited comment on HIVE-23032 at 3/26/20, 1:23 PM: - ||Benchmark|Mode|partCount|tblCount|Cnt|Score|Error|Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| |Oracle|patched|30|4|100|1038.296|± 36.946|ms/op| |Oracle|patched|300|4|100|2670.009|± 135.603|ms/op| |Oracle|patched|600|4|100|4657.155|± 280.535|ms/op| |Oracle|patched|1200|4|100|8740.191|± 322.856|ms/op| was (Author: dkuzmenko): ||Benchmark|Mode|(partCount)|(tblCount)|Cnt|Score|Error|Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23080) Clean Up HivePreparedStatement
[
https://issues.apache.org/jira/browse/HIVE-23080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067669#comment-17067669
]
Hive QA commented on HIVE-23080:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} jdbc: The patch generated 0 new + 11 unchanged - 3
fixed = 11 total (was 14) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21279/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21279/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean Up HivePreparedStatement
> --
>
> Key: HIVE-23080
> URL: https://issues.apache.org/jira/browse/HIVE-23080
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23080.1.patch
>
>
> * Remove auto-generated code
> * Fix check styles warnings
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23084) Implement kill query in multiple HS2 environment
[ https://issues.apache.org/jira/browse/HIVE-23084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23084: --- Attachment: HIVE-23084.1.patch > Implement kill query in multiple HS2 environment > > > Key: HIVE-23084 > URL: https://issues.apache.org/jira/browse/HIVE-23084 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23084.1.patch > > > KILL command was implemented in: > * https://issues.apache.org/jira/browse/HIVE-17483 > * https://issues.apache.org/jira/browse/HIVE-20549 > But it is not working in an environment where service discovery is enabled > and more than one HS2 instance is running (except for manually sending the > kill query to all HS2 instance). > Solution: > * If a HS2 instance can't kill a query locally, it should post a kill query > request to the Zookeeper > * Every HS2 should watch the Zookeeper for kill query requests and if its > running on that instance kill it > * Authorization of kill query should work the same -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23084) Implement kill query in multiple HS2 environment
[ https://issues.apache.org/jira/browse/HIVE-23084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23084: --- Status: Patch Available (was: In Progress) > Implement kill query in multiple HS2 environment > > > Key: HIVE-23084 > URL: https://issues.apache.org/jira/browse/HIVE-23084 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23084.1.patch > > > KILL command was implemented in: > * https://issues.apache.org/jira/browse/HIVE-17483 > * https://issues.apache.org/jira/browse/HIVE-20549 > But it is not working in an environment where service discovery is enabled > and more than one HS2 instance is running (except for manually sending the > kill query to all HS2 instance). > Solution: > * If a HS2 instance can't kill a query locally, it should post a kill query > request to the Zookeeper > * Every HS2 should watch the Zookeeper for kill query requests and if its > running on that instance kill it > * Authorization of kill query should work the same -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23051) Clean up BucketCodec
[
https://issues.apache.org/jira/browse/HIVE-23051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067650#comment-17067650
]
Hive QA commented on HIVE-23051:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997770/HIVE-23051.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18147 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21278/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21278/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21278/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997770 - PreCommit-HIVE-Build
> Clean up BucketCodec
>
>
> Key: HIVE-23051
> URL: https://issues.apache.org/jira/browse/HIVE-23051
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23051.1.patch, HIVE-23051.2.patch,
> HIVE-23051.3.patch, HIVE-23051.4.patch, HIVE-23051.4.patch
>
>
> A couple of nagging things caught my eye with this class. The first thing:
> {code:java|title=BucketCodec.java}
> int statementId = options.getStatementId() >= 0 ?
> options.getStatementId() : 0;
> assert this.version >=0 && this.version <= MAX_VERSION
> : "Version out of range: " + version;
> if(!(options.getBucketId() >= 0 && options.getBucketId() <=
> MAX_BUCKET_ID)) {
> throw new IllegalArgumentException("bucketId out of range: " +
> options.getBucketId());
> }
> if(!(statementId >= 0 && statementId <= MAX_STATEMENT_ID)) {
> throw new IllegalArgumentException("statementId out of range: " +
> statementId);
> }
> {code}
> {{statementId}} gets capped, if it's less than 0, then it gets rounded up to
> 0. However, it later checks that the {{statementId}} is greater,... which is
> will always be 'true' since it's getting rounded.
> # Remove the rounding behavior.
> # Make better error message
> # Fail-fast in the constructor if the version is invalid
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Patch Available (was: Open)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.10.patch,
> HIVE-22785.11.patch, HIVE-22785.12.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch,
> HIVE-22785.5.patch, HIVE-22785.6.patch, HIVE-22785.7.patch,
> HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Attachment: HIVE-22785.12.patch
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.10.patch,
> HIVE-22785.11.patch, HIVE-22785.12.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch,
> HIVE-22785.5.patch, HIVE-22785.6.patch, HIVE-22785.7.patch,
> HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Open (was: Patch Available)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.10.patch,
> HIVE-22785.11.patch, HIVE-22785.2.patch, HIVE-22785.2.patch,
> HIVE-22785.3.patch, HIVE-22785.4.patch, HIVE-22785.5.patch,
> HIVE-22785.6.patch, HIVE-22785.7.patch, HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko edited comment on HIVE-23032 at 3/26/20, 12:25 PM: -- ||Benchmark|Mode|(partCount)|(tblCount)|Cnt|Score|Error|Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| was (Author: dkuzmenko): ||Benchmark|Mode|(partCount)|(tblCount)|Cnt|Score|Error Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067634#comment-17067634 ] Denys Kuzmenko commented on HIVE-23032: --- ||Benchmark|Mode|(partCount)|(tblCount)|Cnt|Score|Error Units|| |MySQL|patched|30|4|100|265.502|± 2.449|ms/op| |MySQL|patched|300|4|100|497.645|± 6.788|ms/op| |MySQL|patched|600|4|100|783.228|± 8.720|ms/op| |MySQL|patched|1200|4|100|1548.145|± 18.028|ms/op| > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23051) Clean up BucketCodec
[
https://issues.apache.org/jira/browse/HIVE-23051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067619#comment-17067619
]
Hive QA commented on HIVE-23051:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 1529 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 1 new + 1 unchanged - 7 fixed
= 2 total (was 8) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21278/dev-support/hive-personality.sh
|
| git revision | master / 8d71480 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21278/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21278/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean up BucketCodec
>
>
> Key: HIVE-23051
> URL: https://issues.apache.org/jira/browse/HIVE-23051
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23051.1.patch, HIVE-23051.2.patch,
> HIVE-23051.3.patch, HIVE-23051.4.patch, HIVE-23051.4.patch
>
>
> A couple of nagging things caught my eye with this class. The first thing:
> {code:java|title=BucketCodec.java}
> int statementId = options.getStatementId() >= 0 ?
> options.getStatementId() : 0;
> assert this.version >=0 && this.version <= MAX_VERSION
> : "Version out of range: " + version;
> if(!(options.getBucketId() >= 0 && options.getBucketId() <=
> MAX_BUCKET_ID)) {
> throw new IllegalArgumentException("bucketId out of range: " +
> options.getBucketId());
> }
> if(!(statementId >= 0 && statementId <= MAX_STATEMENT_ID)) {
> throw new IllegalArgumentException("statementId out of range: " +
> statementId);
> }
> {code}
> {{statementId}} gets capped, if it's less than 0, then it gets rounded up to
> 0. However, it later checks that the {{statementId}} is greater,... which is
> will always be 'true' since it's getting rounded.
> # Remove the rounding behavior.
> # Make better error message
> # Fail-fast in the constructor if the version is invalid
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067613#comment-17067613 ] László Pintér commented on HIVE-23058: -- +1 (pending tests) > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
