[jira] [Updated] (HIVE-19927) Last Repl ID set by bootstrap dump is incorrect and may cause data loss if have ACID/MM tables.
[ https://issues.apache.org/jira/browse/HIVE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19927: Resolution: Fixed Status: Resolved (was: Patch Available) > Last Repl ID set by bootstrap dump is incorrect and may cause data loss if > have ACID/MM tables. > --- > > Key: HIVE-19927 > URL: https://issues.apache.org/jira/browse/HIVE-19927 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl, Transactions >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19927.01-branch-3.patch, HIVE-19927.01.patch, > HIVE-19927.02.patch, HIVE-19927.03.patch, HIVE-19927.04.patch > > > During bootstrap dump of ACID tables, let's consider the below sequence. > - Current session (REPL DUMP), Open txn (Txn1) - Event-10 > - Another session (Session-2), Open txn (Txn2) - Event-11 > - Session-2 -> Insert data (T1.D1) to ACID table. - Event-12 > - Get lastReplId = last event ID logged. (Event-12) > - Session-2 -> Commit Txn (Txn2) - Event-13 > - Dump ACID tables based on validTxnList based on Txn1. --> This step skips > all the data written by txns > Txn1. So, T1.D1 will be missing. > - Commit Txn (Txn1) > - REPL LOAD from bootstrap dump will skip T1.D1. > - Incremental REPL DUMP will start from Event-13 and hence lose Txn2 which is > opened after Txn1. So, data T1.D1 will be lost for ever. > Proposed to capture the lastReplId of bootstrap before opening current txn > (Txn1) and store it in Driver context and use it for dump. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19927) Last Repl ID set by bootstrap dump is incorrect and may cause data loss if have ACID/MM tables.
[ https://issues.apache.org/jira/browse/HIVE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19927: Fix Version/s: 3.2.0 > Last Repl ID set by bootstrap dump is incorrect and may cause data loss if > have ACID/MM tables. > --- > > Key: HIVE-19927 > URL: https://issues.apache.org/jira/browse/HIVE-19927 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl, Transactions >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19927.01-branch-3.patch, HIVE-19927.01.patch, > HIVE-19927.02.patch, HIVE-19927.03.patch, HIVE-19927.04.patch > > > During bootstrap dump of ACID tables, let's consider the below sequence. > - Current session (REPL DUMP), Open txn (Txn1) - Event-10 > - Another session (Session-2), Open txn (Txn2) - Event-11 > - Session-2 -> Insert data (T1.D1) to ACID table. - Event-12 > - Get lastReplId = last event ID logged. (Event-12) > - Session-2 -> Commit Txn (Txn2) - Event-13 > - Dump ACID tables based on validTxnList based on Txn1. --> This step skips > all the data written by txns > Txn1. So, T1.D1 will be missing. > - Commit Txn (Txn1) > - REPL LOAD from bootstrap dump will skip T1.D1. > - Incremental REPL DUMP will start from Event-13 and hence lose Txn2 which is > opened after Txn1. So, data T1.D1 will be lost for ever. > Proposed to capture the lastReplId of bootstrap before opening current txn > (Txn1) and store it in Driver context and use it for dump. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19927) Last Repl ID set by bootstrap dump is incorrect and may cause data loss if have ACID/MM tables.
[ https://issues.apache.org/jira/browse/HIVE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560632#comment-16560632 ] Sankar Hariappan commented on HIVE-19927: - Test failures are irrelevant. 01-branch-3.patch is committed to branch-3. > Last Repl ID set by bootstrap dump is incorrect and may cause data loss if > have ACID/MM tables. > --- > > Key: HIVE-19927 > URL: https://issues.apache.org/jira/browse/HIVE-19927 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl, Transactions >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19927.01-branch-3.patch, HIVE-19927.01.patch, > HIVE-19927.02.patch, HIVE-19927.03.patch, HIVE-19927.04.patch > > > During bootstrap dump of ACID tables, let's consider the below sequence. > - Current session (REPL DUMP), Open txn (Txn1) - Event-10 > - Another session (Session-2), Open txn (Txn2) - Event-11 > - Session-2 -> Insert data (T1.D1) to ACID table. - Event-12 > - Get lastReplId = last event ID logged. (Event-12) > - Session-2 -> Commit Txn (Txn2) - Event-13 > - Dump ACID tables based on validTxnList based on Txn1. --> This step skips > all the data written by txns > Txn1. So, T1.D1 will be missing. > - Commit Txn (Txn1) > - REPL LOAD from bootstrap dump will skip T1.D1. > - Incremental REPL DUMP will start from Event-13 and hence lose Txn2 which is > opened after Txn1. So, data T1.D1 will be lost for ever. > Proposed to capture the lastReplId of bootstrap before opening current txn > (Txn1) and store it in Driver context and use it for dump. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19846) Removed Deprecated Calls From FileUtils-getJarFilesByPath
[
https://issues.apache.org/jira/browse/HIVE-19846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560623#comment-16560623
]
Hive QA commented on HIVE-19846:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} common: The patch generated 0 new + 27 unchanged - 1
fixed = 27 total (was 28) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12913/dev-support/hive-personality.sh
|
| git revision | master / 188f4ea |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12913/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Removed Deprecated Calls From FileUtils-getJarFilesByPath
> -
>
> Key: HIVE-19846
> URL: https://issues.apache.org/jira/browse/HIVE-19846
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-19846.1.patch, HIVE-19846.2.patch,
> HIVE-19846.3.patch, HIVE-19846.4.patch, HIVE-19846.5.patch.txt,
> HIVE-19846.6.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20213) Upgrade Calcite to 1.17.0
[ https://issues.apache.org/jira/browse/HIVE-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560621#comment-16560621 ] Ashutosh Chauhan commented on HIVE-20213: - +1 > Upgrade Calcite to 1.17.0 > - > > Key: HIVE-20213 > URL: https://issues.apache.org/jira/browse/HIVE-20213 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20213.01.patch, HIVE-20213.02.patch, > HIVE-20213.03.patch, HIVE-20213.03.patch, HIVE-20213.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18038) org.apache.hadoop.hive.ql.session.OperationLog - Review
[
https://issues.apache.org/jira/browse/HIVE-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560615#comment-16560615
]
Hive QA commented on HIVE-18038:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933425/HIVE-18038.10.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14813 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12912/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12912/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12912/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12933425 - PreCommit-HIVE-Build
> org.apache.hadoop.hive.ql.session.OperationLog - Review
> ---
>
> Key: HIVE-18038
> URL: https://issues.apache.org/jira/browse/HIVE-18038
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18038.1.patch, HIVE-18038.10.patch,
> HIVE-18038.2.patch, HIVE-18038.3.patch, HIVE-18038.4.patch,
> HIVE-18038.5.patch, HIVE-18038.6.patch, HIVE-18038.7.patch,
> HIVE-18038.8.patch, HIVE-18038.9.patch
>
>
> Simplifications, improve readability
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18038) org.apache.hadoop.hive.ql.session.OperationLog - Review
[
https://issues.apache.org/jira/browse/HIVE-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560598#comment-16560598
]
Hive QA commented on HIVE-18038:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 2297 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} ql: The patch generated 0 new + 4 unchanged - 5
fixed = 4 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} ql generated 0 new + 2296 unchanged - 1 fixed = 2296
total (was 2297) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12912/dev-support/hive-personality.sh
|
| git revision | master / 188f4ea |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12912/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> org.apache.hadoop.hive.ql.session.OperationLog - Review
> ---
>
> Key: HIVE-18038
> URL: https://issues.apache.org/jira/browse/HIVE-18038
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18038.1.patch, HIVE-18038.10.patch,
> HIVE-18038.2.patch, HIVE-18038.3.patch, HIVE-18038.4.patch,
> HIVE-18038.5.patch, HIVE-18038.6.patch, HIVE-18038.7.patch,
> HIVE-18038.8.patch, HIVE-18038.9.patch
>
>
> Simplifications, improve readability
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560591#comment-16560591
]
Hive QA commented on HIVE-17684:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933424/HIVE-17684.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 661 failed/errored test(s), 14813 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join11] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join14] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join15] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join33] (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_without_localtask]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_5]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin9]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer7]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[empty_join] (batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_convert_join]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join25] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join26] (batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join33] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join38] (batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin1] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_filter_on_outerjoin]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_subquery2]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonblock_op_deduplicate]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_join]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_transform_hint]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin] (batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sort_merge_join_desc_8]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_include_no_sel]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_left_outer_join]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join0]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join3]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin3]
(batchId=12)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[add_part_multiple]
(batchId=140)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[alter_merge_orc]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[alter_merge_stats_orc]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[annotate_stats_join]
(batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join0]
(batchId=148)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join10]
(batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join11]
(batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join12]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join13]
(batchId=145)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join14]
(batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join15]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join16]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join17]
(batchId=146)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join18]
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join18_multi_distinct]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join19]
(batchId=137)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join1]
(batchId=144)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join20]
(batchId=149)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join21]
(batchId=146)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join22]
[jira] [Updated] (HIVE-20213) Upgrade Calcite to 1.17.0
[ https://issues.apache.org/jira/browse/HIVE-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20213: --- Attachment: HIVE-20213.03.patch > Upgrade Calcite to 1.17.0 > - > > Key: HIVE-20213 > URL: https://issues.apache.org/jira/browse/HIVE-20213 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20213.01.patch, HIVE-20213.02.patch, > HIVE-20213.03.patch, HIVE-20213.03.patch, HIVE-20213.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560576#comment-16560576
]
Hive QA commented on HIVE-17684:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2297 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 31 unchanged - 0 fixed
= 32 total (was 31) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
29s{color} | {color:red} ql generated 4 new + 2296 unchanged - 1 fixed = 2300
total (was 2297) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Incorrect lazy initialization and update of static field
org.apache.hadoop.hive.ql.exec.Operator.hiveGcTimeMonitor in
org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:of static field
org.apache.hadoop.hive.ql.exec.Operator.hiveGcTimeMonitor in
org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:[lines 430-434] |
| | Write to static field
org.apache.hadoop.hive.ql.exec.Operator.criticalGcTimePercentage from instance
method org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:from instance method
org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:[line 432] |
| | Write to static field
org.apache.hadoop.hive.ql.exec.Operator.hiveGcTimeMonitor from instance method
org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:from instance method
org.apache.hadoop.hive.ql.exec.Operator.initialize(Configuration,
ObjectInspector[]) At Operator.java:[line 434] |
| | Write to static field
org.apache.hadoop.hive.ql.exec.Operator.lastAlertGcTimePercentage from instance
method
org.apache.hadoop.hive.ql.exec.Operator$HiveGcTimeMonitor$1.alert(GcTimeMonitor$GcData)
At Operator.java:from instance method
org.apache.hadoop.hive.ql.exec.Operator$HiveGcTimeMonitor$1.alert(GcTimeMonitor$GcData)
At Operator.java:[line 141] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12911/dev-support/hive-personality.sh
|
| git revision | master / 188f4ea |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12911/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12911/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
[jira] [Commented] (HIVE-20258) Should Syncronize getInstance in ReplChangeManager
[
https://issues.apache.org/jira/browse/HIVE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560565#comment-16560565
]
Hive QA commented on HIVE-20258:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933422/HIVE-20258.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14813 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12910/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12910/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12910/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12933422 - PreCommit-HIVE-Build
> Should Syncronize getInstance in ReplChangeManager
> --
>
> Key: HIVE-20258
> URL: https://issues.apache.org/jira/browse/HIVE-20258
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20258.1.patch
>
>
> {code:java}
> public static ReplChangeManager getInstance(Configuration conf) throws
> MetaException {
> if (instance == null) {
> instance = new ReplChangeManager(conf);
> }
> return instance;
> }
> {code}
> This method needs to be synchronized or two different threads may see a
> 'null' value and each create their own manager.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18729) Druid Time column type
[ https://issues.apache.org/jira/browse/HIVE-18729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18729: --- Attachment: HIVE-18729.01.patch > Druid Time column type > -- > > Key: HIVE-18729 > URL: https://issues.apache.org/jira/browse/HIVE-18729 > Project: Hive > Issue Type: Task > Components: Druid integration >Reporter: slim bouguerra >Assignee: Jesus Camacho Rodriguez >Priority: Blocker > Attachments: HIVE-18729.01.branch-3.patch, HIVE-18729.01.patch, > HIVE-18729.branch-3.patch, HIVE-18729.patch > > > I have talked Offline with [~jcamachorodriguez] about this and agreed that > the best way to go is to support both cases where Druid time column can be > Timestamp or Timestamp with local time zone. > In fact, for the Hive-Druid internal table, this makes perfect sense since we > have Hive metadata about the time column during the CTAS statement then we > can handle both cases as we do for another type of storage eg ORC. > For the Druid external tables, we can have a default type and allow the user > to override that via table properties. > CC [~ashutoshc] and [~nishantbangarwa]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18729) Druid Time column type
[ https://issues.apache.org/jira/browse/HIVE-18729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18729: --- Status: Open (was: Patch Available) > Druid Time column type > -- > > Key: HIVE-18729 > URL: https://issues.apache.org/jira/browse/HIVE-18729 > Project: Hive > Issue Type: Task > Components: Druid integration >Reporter: slim bouguerra >Assignee: Jesus Camacho Rodriguez >Priority: Blocker > Attachments: HIVE-18729.01.branch-3.patch, HIVE-18729.branch-3.patch, > HIVE-18729.patch > > > I have talked Offline with [~jcamachorodriguez] about this and agreed that > the best way to go is to support both cases where Druid time column can be > Timestamp or Timestamp with local time zone. > In fact, for the Hive-Druid internal table, this makes perfect sense since we > have Hive metadata about the time column during the CTAS statement then we > can handle both cases as we do for another type of storage eg ORC. > For the Druid external tables, we can have a default type and allow the user > to override that via table properties. > CC [~ashutoshc] and [~nishantbangarwa]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-18729) Druid Time column type
[ https://issues.apache.org/jira/browse/HIVE-18729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18729 started by Jesus Camacho Rodriguez. -- > Druid Time column type > -- > > Key: HIVE-18729 > URL: https://issues.apache.org/jira/browse/HIVE-18729 > Project: Hive > Issue Type: Task > Components: Druid integration >Reporter: slim bouguerra >Assignee: Jesus Camacho Rodriguez >Priority: Blocker > Attachments: HIVE-18729.01.branch-3.patch, HIVE-18729.branch-3.patch, > HIVE-18729.patch > > > I have talked Offline with [~jcamachorodriguez] about this and agreed that > the best way to go is to support both cases where Druid time column can be > Timestamp or Timestamp with local time zone. > In fact, for the Hive-Druid internal table, this makes perfect sense since we > have Hive metadata about the time column during the CTAS statement then we > can handle both cases as we do for another type of storage eg ORC. > For the Druid external tables, we can have a default type and allow the user > to override that via table properties. > CC [~ashutoshc] and [~nishantbangarwa]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560556#comment-16560556
]
BELUGA BEHR commented on HIVE-20238:
A lot of overlap with {{HiveStringUtils}}
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch, HIVE-20238.2.patch
>
>
> Remove the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20238:
---
Status: Patch Available (was: Open)
Taking another crack at it
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch, HIVE-20238.2.patch
>
>
> Remove the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20238:
---
Attachment: HIVE-20238.2.patch
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch, HIVE-20238.2.patch
>
>
> Remove the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20238:
---
Status: Open (was: Patch Available)
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch, HIVE-20238.2.patch
>
>
> Remove the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20258) Should Syncronize getInstance in ReplChangeManager
[
https://issues.apache.org/jira/browse/HIVE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560551#comment-16560551
]
Hive QA commented on HIVE-20258:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12910/dev-support/hive-personality.sh
|
| git revision | master / 188f4ea |
| Default Java | 1.8.0_111 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12910/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12910/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12910/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Should Syncronize getInstance in ReplChangeManager
> --
>
> Key: HIVE-20258
> URL: https://issues.apache.org/jira/browse/HIVE-20258
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20258.1.patch
>
>
> {code:java}
> public static ReplChangeManager getInstance(Configuration conf) throws
> MetaException {
> if (instance == null) {
> instance = new ReplChangeManager(conf);
> }
> return instance;
> }
> {code}
> This method needs to be synchronized or two different threads may see a
> 'null' value and each create their own manager.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20257) Improvements to Hive.java
[
https://issues.apache.org/jira/browse/HIVE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20257:
---
Attachment: HIVE-20257.2.patch
> Improvements to Hive.java
> -
>
> Key: HIVE-20257
> URL: https://issues.apache.org/jira/browse/HIVE-20257
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20257.1.patch, HIVE-20257.2.patch
>
>
> Various fixes to {{Hive.java}}
> * Use Log4J parameters in logging statements
> * Fix check styles
> * Make code more concise
> * Remove "log and throw" code
> * Replaced calls to deprecated code
> * Removed superfluous calls to {{toString}}
>
> "log and throw" is considered and anti-pattern. Only the highest level catch
> should be providing detailed logging otherwise we print the same stack trace
> to the logs several times and with different context (for example when an
> exception is wrapped, we get two different logging events).
>
> https://community.oracle.com/docs/DOC-983543#logAndThrow
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20257) Improvements to Hive.java
[
https://issues.apache.org/jira/browse/HIVE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20257:
---
Status: Patch Available (was: Open)
Fix check-style warnings
> Improvements to Hive.java
> -
>
> Key: HIVE-20257
> URL: https://issues.apache.org/jira/browse/HIVE-20257
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20257.1.patch, HIVE-20257.2.patch
>
>
> Various fixes to {{Hive.java}}
> * Use Log4J parameters in logging statements
> * Fix check styles
> * Make code more concise
> * Remove "log and throw" code
> * Replaced calls to deprecated code
> * Removed superfluous calls to {{toString}}
>
> "log and throw" is considered and anti-pattern. Only the highest level catch
> should be providing detailed logging otherwise we print the same stack trace
> to the logs several times and with different context (for example when an
> exception is wrapped, we get two different logging events).
>
> https://community.oracle.com/docs/DOC-983543#logAndThrow
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20257) Improvements to Hive.java
[
https://issues.apache.org/jira/browse/HIVE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20257:
---
Status: Open (was: Patch Available)
> Improvements to Hive.java
> -
>
> Key: HIVE-20257
> URL: https://issues.apache.org/jira/browse/HIVE-20257
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20257.1.patch
>
>
> Various fixes to {{Hive.java}}
> * Use Log4J parameters in logging statements
> * Fix check styles
> * Make code more concise
> * Remove "log and throw" code
> * Replaced calls to deprecated code
> * Removed superfluous calls to {{toString}}
>
> "log and throw" is considered and anti-pattern. Only the highest level catch
> should be providing detailed logging otherwise we print the same stack trace
> to the logs several times and with different context (for example when an
> exception is wrapped, we get two different logging events).
>
> https://community.oracle.com/docs/DOC-983543#logAndThrow
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560539#comment-16560539
]
Hive QA commented on HIVE-20238:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
30s{color} | {color:blue} standalone-metastore/metastore-common in master has 9
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
7s{color} | {color:red} standalone-metastore_metastore-common generated 4 new +
4 unchanged - 0 fixed = 8 total (was 4) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12909/dev-support/hive-personality.sh
|
| git revision | master / 188f4ea |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12909/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12909/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12909/yetus/diff-javadoc-javadoc-standalone-metastore_metastore-common.txt
|
| modules | C: standalone-metastore/metastore-common metastore
standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12909/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch
>
>
> Remove the method {{stringifyException}}
>
[jira] [Commented] (HIVE-20257) Improvements to Hive.java
[
https://issues.apache.org/jira/browse/HIVE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560529#comment-16560529
]
Hive QA commented on HIVE-20257:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933414/HIVE-20257.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14813 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12908/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12908/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12908/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12933414 - PreCommit-HIVE-Build
> Improvements to Hive.java
> -
>
> Key: HIVE-20257
> URL: https://issues.apache.org/jira/browse/HIVE-20257
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20257.1.patch
>
>
> Various fixes to {{Hive.java}}
> * Use Log4J parameters in logging statements
> * Fix check styles
> * Make code more concise
> * Remove "log and throw" code
> * Replaced calls to deprecated code
> * Removed superfluous calls to {{toString}}
>
> "log and throw" is considered and anti-pattern. Only the highest level catch
> should be providing detailed logging otherwise we print the same stack trace
> to the logs several times and with different context (for example when an
> exception is wrapped, we get two different logging events).
>
> https://community.oracle.com/docs/DOC-983543#logAndThrow
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20261) Expose inputPartitionList in QueryPlan
[ https://issues.apache.org/jira/browse/HIVE-20261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HIVE-20261: -- Attachment: HIVE-20261.1.patch > Expose inputPartitionList in QueryPlan > -- > > Key: HIVE-20261 > URL: https://issues.apache.org/jira/browse/HIVE-20261 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Zheng Shao >Assignee: Zheng Shao >Priority: Minor > Attachments: HIVE-20261.1.patch > > > Having access to the list of input partitions for all historical Hive queries > in a system provides a great opportunity to insights on data access frequency > and potential storage tiering. > This task aims to expose that via QueryPlan so that a Hive Hook can pick it > up and store the information for analysis later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20261) Expose inputPartitionList in QueryPlan
[ https://issues.apache.org/jira/browse/HIVE-20261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao reassigned HIVE-20261: - > Expose inputPartitionList in QueryPlan > -- > > Key: HIVE-20261 > URL: https://issues.apache.org/jira/browse/HIVE-20261 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Zheng Shao >Assignee: Zheng Shao >Priority: Minor > > Having access to the list of input partitions for all historical Hive queries > in a system provides a great opportunity to insights on data access frequency > and potential storage tiering. > This task aims to expose that via QueryPlan so that a Hive Hook can pick it > up and store the information for analysis later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20257) Improvements to Hive.java
[
https://issues.apache.org/jira/browse/HIVE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560516#comment-16560516
]
Hive QA commented on HIVE-20257:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2297 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 9 new + 181 unchanged - 34
fixed = 190 total (was 215) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12908/dev-support/hive-personality.sh
|
| git revision | master / 188f4ea |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12908/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12908/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12908/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improvements to Hive.java
> -
>
> Key: HIVE-20257
> URL: https://issues.apache.org/jira/browse/HIVE-20257
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20257.1.patch
>
>
> Various fixes to {{Hive.java}}
> * Use Log4J parameters in logging statements
> * Fix check styles
> * Make code more concise
> * Remove "log and throw" code
> * Replaced calls to deprecated code
> * Removed superfluous calls to {{toString}}
>
> "log and throw" is considered and anti-pattern. Only the highest level catch
> should be providing detailed logging otherwise we print the same stack trace
> to the logs several times and with different context (for example when an
> exception is wrapped, we get two different logging events).
>
> https://community.oracle.com/docs/DOC-983543#logAndThrow
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20196) Separate MetastoreConf into common and server parts
[ https://issues.apache.org/jira/browse/HIVE-20196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560507#comment-16560507 ] Alexander Kolbasov commented on HIVE-20196: --- [~alangates] Initially I just replaced all instances of X.class.getName() with literal strings. This is ok but my concern is that this may silently break of someone renames something, so I decided to do it differently. If you think that this is not a big deal I can revert to the old approach of just using string literals. [~vihangk1] What do you think about this? > Separate MetastoreConf into common and server parts > --- > > Key: HIVE-20196 > URL: https://issues.apache.org/jira/browse/HIVE-20196 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20196.01.patch, HIVE-20196.02.patch > > > MetastoreConf has knowledge about some server-specific classes. We need to > separate these into a separate server-specific class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20241) Support partitioning spec in CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-20241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20241:
---
Attachment: HIVE-20241.01.patch
> Support partitioning spec in CTAS statements
>
>
> Key: HIVE-20241
> URL: https://issues.apache.org/jira/browse/HIVE-20241
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20241.01.patch, HIVE-20241.01.patch,
> HIVE-20241.01.patch, HIVE-20241.patch
>
>
> Currently, for partitioned tables we will declare the table and insert the
> data in different operations. This issue is to extend CTAS statement to
> support specifying partition columns.
> For instance:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20221) Increase column width for partition_params
[ https://issues.apache.org/jira/browse/HIVE-20221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560499#comment-16560499 ] Alan Gates commented on HIVE-20221: --- Have you run the install and upgrade tests for all the db types? Assuming those pass, +1. > Increase column width for partition_params > -- > > Key: HIVE-20221 > URL: https://issues.apache.org/jira/browse/HIVE-20221 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20221.01.patch, HIVE-20221.01.patch > > > HIVE-12274 have addressed almost all metastore columns; however it have left > out PARTITION_PARAMS; so in case of a partitioned tables the limits are still > there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20241) Support partitioning spec in CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-20241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560497#comment-16560497
]
Hive QA commented on HIVE-20241:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933407/HIVE-20241.01.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14810 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
[druidmini_dynamic_partition.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test1.q,druidmini_test_insert.q]
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
(batchId=244)
org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=248)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12907/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12907/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12907/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12933407 - PreCommit-HIVE-Build
> Support partitioning spec in CTAS statements
>
>
> Key: HIVE-20241
> URL: https://issues.apache.org/jira/browse/HIVE-20241
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20241.01.patch, HIVE-20241.01.patch,
> HIVE-20241.patch
>
>
> Currently, for partitioned tables we will declare the table and insert the
> data in different operations. This issue is to extend CTAS statement to
> support specifying partition columns.
> For instance:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20196) Separate MetastoreConf into common and server parts
[ https://issues.apache.org/jira/browse/HIVE-20196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560496#comment-16560496 ] Alan Gates commented on HIVE-20196: --- I don't think this is a good approach. You're copying a lot of code in order to separate out a few config variables. What exactly is the value of not having these present on the client. If the only concern is X.class.getName() messing up your package divisions just hardwire in the name, ie "org.apache.hadoop.hive.metastore.X" instead. > Separate MetastoreConf into common and server parts > --- > > Key: HIVE-20196 > URL: https://issues.apache.org/jira/browse/HIVE-20196 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20196.01.patch, HIVE-20196.02.patch > > > MetastoreConf has knowledge about some server-specific classes. We need to > separate these into a separate server-specific class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20260) NDV of a column shouldn't be scaled when row count is changed by filter on another column
[
https://issues.apache.org/jira/browse/HIVE-20260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-20260:
---
Description:
HIVE-17465 introduced progressive scaling of rowcounts in presence of multiple
filters. HIVE-19500 improved on that by also scaling col stats (NDV) in such
scenario. However, it should pay attention to column used in filter expression
and not scale for all filters. eg.,
consider filter a = 1 and b = 2 ndv of column b should not be scaled down by
row count changes caused by a = 1
Other way to say this that ndv of a particular column should be updated at the
end of computation of row count for that operator.
Here are the possible cases where our estimates can be accurate (or close to)
{code}
case 1 - (d_year = 2001 and d_moy=1)
case 2 - (d_year = 2001 and d_year IN (2001, 2002))
case 3 - (d_year = 2001 and d_moy = 1 and d_dom = 1)
case 4 - (d_date IN ('1999-01-02', '1999-01-02'))
case 5 - (d_date = '1999-01-01')
{code}
was:
HIVE-17465 introduced progressive scaling of rowcounts in presence of multiple
filters. HIVE-19500 improved on that by also scaling col stats (NDV) in such
scenario. However, it should pay attention to column used in filter expression
and not scale for all filters. eg.,
consider filter a = 1 and b = 2 ndv of column b should not be scaled down by
row count changes caused by a = 1
Other way to say this that ndv of a particular column should be updated at the
end of computation of row count for that operator.
> NDV of a column shouldn't be scaled when row count is changed by filter on
> another column
> -
>
> Key: HIVE-20260
> URL: https://issues.apache.org/jira/browse/HIVE-20260
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Ashutosh Chauhan
>Priority: Major
>
> HIVE-17465 introduced progressive scaling of rowcounts in presence of
> multiple filters. HIVE-19500 improved on that by also scaling col stats (NDV)
> in such scenario. However, it should pay attention to column used in filter
> expression and not scale for all filters. eg.,
> consider filter a = 1 and b = 2 ndv of column b should not be scaled down by
> row count changes caused by a = 1
> Other way to say this that ndv of a particular column should be updated at
> the end of computation of row count for that operator.
> Here are the possible cases where our estimates can be accurate (or close to)
> {code}
> case 1 - (d_year = 2001 and d_moy=1)
> case 2 - (d_year = 2001 and d_year IN (2001, 2002))
> case 3 - (d_year = 2001 and d_moy = 1 and d_dom = 1)
> case 4 - (d_date IN ('1999-01-02', '1999-01-02'))
> case 5 - (d_date = '1999-01-01')
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-20215) Hive unable to plan/compile query containing subquery with multiple same name columns
[
https://issues.apache.org/jira/browse/HIVE-20215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg resolved HIVE-20215.
Resolution: Duplicate
> Hive unable to plan/compile query containing subquery with multiple same name
> columns
> -
>
> Key: HIVE-20215
> URL: https://issues.apache.org/jira/browse/HIVE-20215
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> *Reproducer*
> ==
> {code:sql}
> >create table t1(c1 int)
> >explain select count(*) from (select c1, c1 from t1) subq
> {code}
> {noformat}
> FAILED: SemanticException [Error 10007]: Ambiguous column reference c1 in subq
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20215) Hive unable to plan/compile query containing subquery with multiple same name columns
[
https://issues.apache.org/jira/browse/HIVE-20215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560460#comment-16560460
]
Vineet Garg commented on HIVE-20215:
This will be fixe in HIVE-19770
> Hive unable to plan/compile query containing subquery with multiple same name
> columns
> -
>
> Key: HIVE-20215
> URL: https://issues.apache.org/jira/browse/HIVE-20215
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> *Reproducer*
> ==
> {code:sql}
> >create table t1(c1 int)
> >explain select count(*) from (select c1, c1 from t1) subq
> {code}
> {noformat}
> FAILED: SemanticException [Error 10007]: Ambiguous column reference c1 in subq
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20241) Support partitioning spec in CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-20241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560459#comment-16560459
]
Hive QA commented on HIVE-20241:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
11s{color} | {color:blue} ql in master has 2297 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 10 new + 832 unchanged - 14
fixed = 842 total (was 846) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
19s{color} | {color:red} ql generated 3 new + 2294 unchanged - 3 fixed = 2297
total (was 2297) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA230 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46716-46729] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA274 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
47066-47079] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA289 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
47202-47215] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12907/dev-support/hive-personality.sh
|
| git revision | master / 188f4ea |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12907/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12907/yetus/new-findbugs-ql.html
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12907/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support partitioning spec in CTAS statements
>
>
> Key: HIVE-20241
> URL: https://issues.apache.org/jira/browse/HIVE-20241
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20241.01.patch, HIVE-20241.01.patch,
> HIVE-20241.patch
>
>
> Currently, for partitioned tables we will declare the table
[jira] [Updated] (HIVE-19770) Support for CBO for queries with multiple same columns in select
[
https://issues.apache.org/jira/browse/HIVE-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19770:
---
Attachment: HIVE-19770.7.patch
> Support for CBO for queries with multiple same columns in select
>
>
> Key: HIVE-19770
> URL: https://issues.apache.org/jira/browse/HIVE-19770
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19770.1.patch, HIVE-19770.2.patch,
> HIVE-19770.3.patch, HIVE-19770.4.patch, HIVE-19770.5.patch,
> HIVE-19770.6.patch, HIVE-19770.7.patch
>
>
> Currently queries such as {code:sql} select a,a from t1 where b > 10 {code}
> are not supported for CBO.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19770) Support for CBO for queries with multiple same columns in select
[
https://issues.apache.org/jira/browse/HIVE-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19770:
---
Status: Patch Available (was: Open)
> Support for CBO for queries with multiple same columns in select
>
>
> Key: HIVE-19770
> URL: https://issues.apache.org/jira/browse/HIVE-19770
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19770.1.patch, HIVE-19770.2.patch,
> HIVE-19770.3.patch, HIVE-19770.4.patch, HIVE-19770.5.patch,
> HIVE-19770.6.patch, HIVE-19770.7.patch
>
>
> Currently queries such as {code:sql} select a,a from t1 where b > 10 {code}
> are not supported for CBO.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19770) Support for CBO for queries with multiple same columns in select
[
https://issues.apache.org/jira/browse/HIVE-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19770:
---
Status: Open (was: Patch Available)
> Support for CBO for queries with multiple same columns in select
>
>
> Key: HIVE-19770
> URL: https://issues.apache.org/jira/browse/HIVE-19770
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19770.1.patch, HIVE-19770.2.patch,
> HIVE-19770.3.patch, HIVE-19770.4.patch, HIVE-19770.5.patch,
> HIVE-19770.6.patch, HIVE-19770.7.patch
>
>
> Currently queries such as {code:sql} select a,a from t1 where b > 10 {code}
> are not supported for CBO.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560444#comment-16560444 ] Jason Dere commented on HIVE-20259: --- Yeah, I had originally proposed using the ClearDanglingScratchDir functionality for cleanup, though [~hagleitn] had some reservations with this approach as this depends on very specific behavior of HDFS (file leases). Another possible option rather than using the file lease, is to simply periodically write a file with a known naming convention (.cleanup.timestamp_val?) to the cache directory. A cleanup thread would look for the .cleanup file in the base cache directory, and delete any cache directories where the age of the file is too old. As long as the Hive process is still alive and creating new versions of the cleanup file, its cache directory would not be cleaned up by the cleanup thread. > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560433#comment-16560433
]
Hive QA commented on HIVE-20233:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933397/HIVE-20233.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14783 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=160)
[insert_values_non_partitioned.q,retry_failure_oom.q,colstats_date_min_max.q,vectorized_math_funcs.q,vectorization_4.q,vectorization_2.q,subquery_multi.q,check_constraint.q,cbo_simple_select.q,tez_union_decimal.q,vector_parquet_nested_two_level_complex.q,multi_column_in.q,groupby1.q,order_null.q,cbo_rp_udf_udaf.q,cbo_rp_views.q,vector_decimal_aggregate.q,vector_date_1.q,lvj_mapjoin.q,update_all_types.q,retry_failure_stat_changes.q,kill_query.q,vector_llap_text_1.q,vector_decimal_3.q,update_two_cols.q,update_where_no_match.q,vectorized_nested_mapjoin.q,authorization_2.q,schema_evol_text_nonvec_part.q,enforce_constraint_notnull.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12906/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12906/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12906/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12933397 - PreCommit-HIVE-Build
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20212) Hiveserver2 in http mode emitting metric default.General.open_connections incorrectly
[ https://issues.apache.org/jira/browse/HIVE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20212: --- Resolution: Fixed Fix Version/s: 3.2.0 4.0.0 Status: Resolved (was: Patch Available) Pushed to master, branch-3. Thanks for reviewing [~kgyrtkirk] > Hiveserver2 in http mode emitting metric default.General.open_connections > incorrectly > - > > Key: HIVE-20212 > URL: https://issues.apache.org/jira/browse/HIVE-20212 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20212.01.patch, HIVE-20212.01.patch, > HIVE-20212.02.patch, HIVE-20212.03.patch, HIVE-20212.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20162) Do Not Print StackTraces to STDERR in AbstractJoinTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-20162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated HIVE-20162:
Attachment: HIVE-20162.1.patch
Status: Patch Available (was: Open)
Removed printStrackTrace() and passed the exception object to SemanticException.
> Do Not Print StackTraces to STDERR in AbstractJoinTaskDispatcher
>
>
> Key: HIVE-20162
> URL: https://issues.apache.org/jira/browse/HIVE-20162
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
> Attachments: HIVE-20162.1.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AbstractJoinTaskDispatcher.java
> {code}
> } catch (Exception e) {
> e.printStackTrace();
> throw new SemanticException("Generate Map Join Task Error: " +
> e.getMessage());
> }
> {code}
> Remove the call to {{printStackTrace}} and just throw the error. If the
> stack trace really is needed (doubtful), then pass it to the
> {{SemanticException}} constructor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20162) Do Not Print StackTraces to STDERR in AbstractJoinTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-20162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada reassigned HIVE-20162:
---
Assignee: Anurag Mantripragada
> Do Not Print StackTraces to STDERR in AbstractJoinTaskDispatcher
>
>
> Key: HIVE-20162
> URL: https://issues.apache.org/jira/browse/HIVE-20162
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AbstractJoinTaskDispatcher.java
> {code}
> } catch (Exception e) {
> e.printStackTrace();
> throw new SemanticException("Generate Map Join Task Error: " +
> e.getMessage());
> }
> {code}
> Remove the call to {{printStackTrace}} and just throw the error. If the
> stack trace really is needed (doubtful), then pass it to the
> {{SemanticException}} constructor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560401#comment-16560401 ] Gopal V edited comment on HIVE-20259 at 7/27/18 10:10 PM: -- Integrate into https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/ClearDanglingScratchDir.java was (Author: gopalv): Integrate into https://github.com/apache/hive/blob/master/bin/ext/cleardanglingscratchdir.sh ? > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560401#comment-16560401 ] Gopal V commented on HIVE-20259: Integrate into https://github.com/apache/hive/blob/master/bin/ext/cleardanglingscratchdir.sh ? > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560392#comment-16560392
]
Hive QA commented on HIVE-20233:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2297 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 20 unchanged - 5 fixed
= 21 total (was 25) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12906/dev-support/hive-personality.sh
|
| git revision | master / 804b125 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12906/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12906/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-20259: - > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20212) Hiveserver2 in http mode emitting metric default.General.open_connections incorrectly
[
https://issues.apache.org/jira/browse/HIVE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560358#comment-16560358
]
Hive QA commented on HIVE-20212:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933399/HIVE-20212.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14813 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12905/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12905/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12905/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12933399 - PreCommit-HIVE-Build
> Hiveserver2 in http mode emitting metric default.General.open_connections
> incorrectly
> -
>
> Key: HIVE-20212
> URL: https://issues.apache.org/jira/browse/HIVE-20212
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-20212.01.patch, HIVE-20212.01.patch,
> HIVE-20212.02.patch, HIVE-20212.03.patch, HIVE-20212.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20239) Do Not Print StackTraces to STDERR in MapJoinProcessor
[
https://issues.apache.org/jira/browse/HIVE-20239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated HIVE-20239:
Attachment: HIVE-20239.2.patch
Status: Patch Available (was: Open)
You are right, it does take exception object as a parameter. Submitted new
patch with the change.
> Do Not Print StackTraces to STDERR in MapJoinProcessor
> --
>
> Key: HIVE-20239
> URL: https://issues.apache.org/jira/browse/HIVE-20239
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20239.1.patch, HIVE-20239.2.patch
>
>
> {code:java|title=MapJoinProcessor.java}
> } catch (Exception e) {
> e.printStackTrace();
> throw new SemanticException("Failed to generate new mapJoin operator " +
> "by exception : " + e.getMessage());
> }
> {code}
> Please change to... something like...
> {code}
> } catch (Exception e) {
> throw new SemanticException("Failed to generate new mapJoin operator",
> e);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20239) Do Not Print StackTraces to STDERR in MapJoinProcessor
[
https://issues.apache.org/jira/browse/HIVE-20239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated HIVE-20239:
Status: Open (was: Patch Available)
> Do Not Print StackTraces to STDERR in MapJoinProcessor
> --
>
> Key: HIVE-20239
> URL: https://issues.apache.org/jira/browse/HIVE-20239
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20239.1.patch
>
>
> {code:java|title=MapJoinProcessor.java}
> } catch (Exception e) {
> e.printStackTrace();
> throw new SemanticException("Failed to generate new mapJoin operator " +
> "by exception : " + e.getMessage());
> }
> {code}
> Please change to... something like...
> {code}
> } catch (Exception e) {
> throw new SemanticException("Failed to generate new mapJoin operator",
> e);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20258) Should Syncronize getInstance in ReplChangeManager
[
https://issues.apache.org/jira/browse/HIVE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20258:
---
Description:
{code:java}
public static ReplChangeManager getInstance(Configuration conf) throws
MetaException {
if (instance == null) {
instance = new ReplChangeManager(conf);
}
return instance;
}
{code}
This method needs to be synchronized or two different threads may see a 'null'
value and each create their own manager.
was:
{code:java}
public static ReplChangeManager getInstance(Configuration conf) throws
MetaException {
if (instance == null) {
instance = new ReplChangeManager(conf);
}
return instance;
}
{code}
This method needs to be synchronized or two different callers will see a 'null'
value and each create their own manager.
> Should Syncronize getInstance in ReplChangeManager
> --
>
> Key: HIVE-20258
> URL: https://issues.apache.org/jira/browse/HIVE-20258
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20258.1.patch
>
>
> {code:java}
> public static ReplChangeManager getInstance(Configuration conf) throws
> MetaException {
> if (instance == null) {
> instance = new ReplChangeManager(conf);
> }
> return instance;
> }
> {code}
> This method needs to be synchronized or two different threads may see a
> 'null' value and each create their own manager.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19489) Disable stats autogather for external tables
[ https://issues.apache.org/jira/browse/HIVE-19489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560351#comment-16560351 ] BELUGA BEHR commented on HIVE-19489: There is already such a flag and it is mentioned in [HIVE-18743]. My suggestions would be to use this flag (though rename it, I dislike the "do_not_" prefix). Users could manually set it, but by default it would be set to 'true' for managed tables and 'false' for external tables. > Disable stats autogather for external tables > > > Key: HIVE-19489 > URL: https://issues.apache.org/jira/browse/HIVE-19489 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > Hive auto-gather of table statistics can result in incorrect generation of > stats (and the stats being marked as accurate) in the case of external tables > where the data is being written by external apps. > To avoid this issue, stats autogather will be disabled on external tables > when loading/inserting into a table with existing data, if > HIVE_DISABLE_UNSAFE_EXTERNALTABLE_OPERATIONS is enabled. In this situation, > users should rely on explicitly calling ANALYZE TABLE on their external > tables to make sure the stats are kept up-to-date. > Autogather of stats will still be allowed to occur on external tables in the > case of INSERT OVERWRITE or LOAD DATA OVERWRITE, since the existing data is > being removed and so the stats calculated on the inserted/loaded data should > be accurate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19489) Disable stats autogather for external tables
[ https://issues.apache.org/jira/browse/HIVE-19489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560351#comment-16560351 ] BELUGA BEHR edited comment on HIVE-19489 at 7/27/18 9:23 PM: - There is already such a flag and it is mentioned in [HIVE-18743]. My suggestions would be to use this flag (though rename it, I dislike the "do_not_" prefix). Users could manually set it at the table properties level, but by default it would be set to 'true' for managed tables and 'false' for external tables. was (Author: belugabehr): There is already such a flag and it is mentioned in [HIVE-18743]. My suggestions would be to use this flag (though rename it, I dislike the "do_not_" prefix). Users could manually set it, but by default it would be set to 'true' for managed tables and 'false' for external tables. > Disable stats autogather for external tables > > > Key: HIVE-19489 > URL: https://issues.apache.org/jira/browse/HIVE-19489 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > Hive auto-gather of table statistics can result in incorrect generation of > stats (and the stats being marked as accurate) in the case of external tables > where the data is being written by external apps. > To avoid this issue, stats autogather will be disabled on external tables > when loading/inserting into a table with existing data, if > HIVE_DISABLE_UNSAFE_EXTERNALTABLE_OPERATIONS is enabled. In this situation, > users should rely on explicitly calling ANALYZE TABLE on their external > tables to make sure the stats are kept up-to-date. > Autogather of stats will still be allowed to occur on external tables in the > case of INSERT OVERWRITE or LOAD DATA OVERWRITE, since the existing data is > being removed and so the stats calculated on the inserted/loaded data should > be accurate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20240) Semijoin Reduction : Use local variable to check for external table condition
[ https://issues.apache.org/jira/browse/HIVE-20240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20240: -- Attachment: HIVE-20240.01-branch-3.patch > Semijoin Reduction : Use local variable to check for external table condition > - > > Key: HIVE-20240 > URL: https://issues.apache.org/jira/browse/HIVE-20240 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20240.01-branch-3.patch, HIVE-20240.1.patch > > > This condition, > > semiJoin = semiJoin && > !disableSemiJoinOptDueToExternalTable(parseContext.getConf(), ts, ctx); > > may set semiJoin to false if an external table is encountered and will remain > false for subsequent cases. It should only disable it for that particular > case. > > cc [~jdere] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20240) Semijoin Reduction : Use local variable to check for external table condition
[ https://issues.apache.org/jira/browse/HIVE-20240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560341#comment-16560341 ] Deepak Jaiswal commented on HIVE-20240: --- Thanks [~jdere] for the review. Committed to master. Preparing for branch-3 > Semijoin Reduction : Use local variable to check for external table condition > - > > Key: HIVE-20240 > URL: https://issues.apache.org/jira/browse/HIVE-20240 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20240.01-branch-3.patch, HIVE-20240.1.patch > > > This condition, > > semiJoin = semiJoin && > !disableSemiJoinOptDueToExternalTable(parseContext.getConf(), ts, ctx); > > may set semiJoin to false if an external table is encountered and will remain > false for subsequent cases. It should only disable it for that particular > case. > > cc [~jdere] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20239) Do Not Print StackTraces to STDERR in MapJoinProcessor
[
https://issues.apache.org/jira/browse/HIVE-20239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560328#comment-16560328
]
BELUGA BEHR commented on HIVE-20239:
[~anuragmantri]
{code:java|title=diff}
throw new SemanticException("Failed to generate new mapJoin operator " +
- "by exception : " + e.getMessage());
+ "by exception : " + e);
{code}
Right now, the Exception's {{toString}} is appended to the error message. It
would have to be:
{code}
throw new SemanticException("Failed to generate new mapJoin operator",
e);
{code}
I just looked at the SemanticException code and that should be acceptable.
> Do Not Print StackTraces to STDERR in MapJoinProcessor
> --
>
> Key: HIVE-20239
> URL: https://issues.apache.org/jira/browse/HIVE-20239
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20239.1.patch
>
>
> {code:java|title=MapJoinProcessor.java}
> } catch (Exception e) {
> e.printStackTrace();
> throw new SemanticException("Failed to generate new mapJoin operator " +
> "by exception : " + e.getMessage());
> }
> {code}
> Please change to... something like...
> {code}
> } catch (Exception e) {
> throw new SemanticException("Failed to generate new mapJoin operator",
> e);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20239) Do Not Print StackTraces to STDERR in MapJoinProcessor
[
https://issues.apache.org/jira/browse/HIVE-20239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560321#comment-16560321
]
Anurag Mantripragada commented on HIVE-20239:
-
SemanticException accepted the exception object in constructor. Looks like all
the test passed.
> Do Not Print StackTraces to STDERR in MapJoinProcessor
> --
>
> Key: HIVE-20239
> URL: https://issues.apache.org/jira/browse/HIVE-20239
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20239.1.patch
>
>
> {code:java|title=MapJoinProcessor.java}
> } catch (Exception e) {
> e.printStackTrace();
> throw new SemanticException("Failed to generate new mapJoin operator " +
> "by exception : " + e.getMessage());
> }
> {code}
> Please change to... something like...
> {code}
> } catch (Exception e) {
> throw new SemanticException("Failed to generate new mapJoin operator",
> e);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20153) Count and Sum UDF consume more memory in Hive 2+
[ https://issues.apache.org/jira/browse/HIVE-20153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-20153: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Thanks [~gopalv] for reviewing. Pushed to master. [~szehon] It should solve your issue, but let me know if you still see it. > Count and Sum UDF consume more memory in Hive 2+ > > > Key: HIVE-20153 > URL: https://issues.apache.org/jira/browse/HIVE-20153 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 2.3.2 >Reporter: Szehon Ho >Assignee: Aihua Xu >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20153.1.patch, Screen Shot 2018-07-12 at 6.41.28 > PM.png > > > While playing with Hive2, we noticed that queries with a lot of count() and > sum() aggregations run out of memory on Hadoop side where they worked before > in Hive1. > In many queries, we have to double the Mapper Memory settings (in our > particular case mapreduce.map.java.opts from -Xmx2000M to -Xmx4000M), it > makes it not so easy to upgrade to Hive 2. > Taking heap dump, we see one of the main culprit is the field 'uniqueObjects' > in GeneraicUDAFSum and GenericUDAFCount, which was added to support Window > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20239) Do Not Print StackTraces to STDERR in MapJoinProcessor
[
https://issues.apache.org/jira/browse/HIVE-20239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560313#comment-16560313
]
BELUGA BEHR commented on HIVE-20239:
If the Semantic Exception does not accept an Exception object in the
constructor, then please keep the current format and just remove the
{{printStackTrace}} call.
> Do Not Print StackTraces to STDERR in MapJoinProcessor
> --
>
> Key: HIVE-20239
> URL: https://issues.apache.org/jira/browse/HIVE-20239
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20239.1.patch
>
>
> {code:java|title=MapJoinProcessor.java}
> } catch (Exception e) {
> e.printStackTrace();
> throw new SemanticException("Failed to generate new mapJoin operator " +
> "by exception : " + e.getMessage());
> }
> {code}
> Please change to... something like...
> {code}
> } catch (Exception e) {
> throw new SemanticException("Failed to generate new mapJoin operator",
> e);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20212) Hiveserver2 in http mode emitting metric default.General.open_connections incorrectly
[
https://issues.apache.org/jira/browse/HIVE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560309#comment-16560309
]
Hive QA commented on HIVE-20212:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12905/dev-support/hive-personality.sh
|
| git revision | master / 0ad7112 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12905/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hiveserver2 in http mode emitting metric default.General.open_connections
> incorrectly
> -
>
> Key: HIVE-20212
> URL: https://issues.apache.org/jira/browse/HIVE-20212
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-20212.01.patch, HIVE-20212.01.patch,
> HIVE-20212.02.patch, HIVE-20212.03.patch, HIVE-20212.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19846) Removed Deprecated Calls From FileUtils-getJarFilesByPath
[ https://issues.apache.org/jira/browse/HIVE-19846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-19846: --- Status: Patch Available (was: Open) Attaching the same page... again... > Removed Deprecated Calls From FileUtils-getJarFilesByPath > - > > Key: HIVE-19846 > URL: https://issues.apache.org/jira/browse/HIVE-19846 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-19846.1.patch, HIVE-19846.2.patch, > HIVE-19846.3.patch, HIVE-19846.4.patch, HIVE-19846.5.patch.txt, > HIVE-19846.6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19846) Removed Deprecated Calls From FileUtils-getJarFilesByPath
[ https://issues.apache.org/jira/browse/HIVE-19846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-19846: --- Attachment: HIVE-19846.6.patch > Removed Deprecated Calls From FileUtils-getJarFilesByPath > - > > Key: HIVE-19846 > URL: https://issues.apache.org/jira/browse/HIVE-19846 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-19846.1.patch, HIVE-19846.2.patch, > HIVE-19846.3.patch, HIVE-19846.4.patch, HIVE-19846.5.patch.txt, > HIVE-19846.6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19846) Removed Deprecated Calls From FileUtils-getJarFilesByPath
[ https://issues.apache.org/jira/browse/HIVE-19846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-19846: --- Status: Open (was: Patch Available) > Removed Deprecated Calls From FileUtils-getJarFilesByPath > - > > Key: HIVE-19846 > URL: https://issues.apache.org/jira/browse/HIVE-19846 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-19846.1.patch, HIVE-19846.2.patch, > HIVE-19846.3.patch, HIVE-19846.4.patch, HIVE-19846.5.patch.txt, > HIVE-19846.6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18882) Do Not Hide Exception in Hive Metastore Client Connection
[
https://issues.apache.org/jira/browse/HIVE-18882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560292#comment-16560292
]
BELUGA BEHR commented on HIVE-18882:
[~ngangam] Fair point.
We do print the exception at the end of the method if we are not successful
with connecting.
Perhaps just change it to:
{code}
LOG.debug("Failed to connect to the MetaStore Server...", e);
LOG.warn("Failed to connect to the MetaStore Server...");
{code}
> Do Not Hide Exception in Hive Metastore Client Connection
> -
>
> Key: HIVE-18882
> URL: https://issues.apache.org/jira/browse/HIVE-18882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Manoj Narayanan
>Priority: Minor
> Labels: noob
> Attachments: HIVE-18882.1.patch, HIVE-18882.2.patch, HIVE-18882.patch
>
>
> [https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L526-L531]
>
> {code:java}
> if (LOG.isDebugEnabled()) {
> LOG.warn("Failed to connect to the MetaStore Server...", e);
> } else {
> // Don't print full exception trace if DEBUG is not on.
> LOG.warn("Failed to connect to the MetaStore Server...");
> }
> {code}
> I do not understand the logic here. I always want to see the reason for the
> failure. Otherwise, I do not know why it is failing unless I restart the
> server with debug logging enabled. By that point, the error may have
> cleared. Please just use the Exception in the WARN output without adding
> confusing logging for debugging. This is never an expected behavior... that
> enabling debug would change a _warn_ level log message.
> Also... please remove the ellipsis, they add no value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18038) org.apache.hadoop.hive.ql.session.OperationLog - Review
[ https://issues.apache.org/jira/browse/HIVE-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-18038: --- Attachment: HIVE-18038.10.patch > org.apache.hadoop.hive.ql.session.OperationLog - Review > --- > > Key: HIVE-18038 > URL: https://issues.apache.org/jira/browse/HIVE-18038 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HIVE-18038.1.patch, HIVE-18038.10.patch, > HIVE-18038.2.patch, HIVE-18038.3.patch, HIVE-18038.4.patch, > HIVE-18038.5.patch, HIVE-18038.6.patch, HIVE-18038.7.patch, > HIVE-18038.8.patch, HIVE-18038.9.patch > > > Simplifications, improve readability -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18038) org.apache.hadoop.hive.ql.session.OperationLog - Review
[ https://issues.apache.org/jira/browse/HIVE-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-18038: --- Status: Open (was: Patch Available) > org.apache.hadoop.hive.ql.session.OperationLog - Review > --- > > Key: HIVE-18038 > URL: https://issues.apache.org/jira/browse/HIVE-18038 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HIVE-18038.1.patch, HIVE-18038.10.patch, > HIVE-18038.2.patch, HIVE-18038.3.patch, HIVE-18038.4.patch, > HIVE-18038.5.patch, HIVE-18038.6.patch, HIVE-18038.7.patch, > HIVE-18038.8.patch, HIVE-18038.9.patch > > > Simplifications, improve readability -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18038) org.apache.hadoop.hive.ql.session.OperationLog - Review
[ https://issues.apache.org/jira/browse/HIVE-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-18038: --- Status: Patch Available (was: Open) Putting up the same patch again to re-run unit tests > org.apache.hadoop.hive.ql.session.OperationLog - Review > --- > > Key: HIVE-18038 > URL: https://issues.apache.org/jira/browse/HIVE-18038 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HIVE-18038.1.patch, HIVE-18038.10.patch, > HIVE-18038.2.patch, HIVE-18038.3.patch, HIVE-18038.4.patch, > HIVE-18038.5.patch, HIVE-18038.6.patch, HIVE-18038.7.patch, > HIVE-18038.8.patch, HIVE-18038.9.patch > > > Simplifications, improve readability -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560284#comment-16560284
]
Hive QA commented on HIVE-19097:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933398/HIVE-19097.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 78 failed/errored test(s), 14814 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_filter]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_simple_select]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_simple_select]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic3]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_intervals]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_partition_skip_default]
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_cond_pushdown]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_eq_with_case_when]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_query_multiskew_2]
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcr] (batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup2]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup3] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_transform]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[remove_exprs_stats]
(batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_related_col]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_non_constant_in_expr]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_case]
(batchId=61)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketpruning1]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_7]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_simple_select]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[kryo]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_decimal64_reader]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_ssb]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_ssb_2]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_between_in]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_string_decimal]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_multipartitioning]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_navfn]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_case]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=187)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join19]
(batchId=137)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
(batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_simple_select]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[pcr] (batchId=135)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_transform]
(batchId=143)
[jira] [Commented] (HIVE-20256) Remove unused classes from Hive QL
[ https://issues.apache.org/jira/browse/HIVE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560285#comment-16560285 ] Vineet Garg commented on HIVE-20256: [~mgergely] Any particular reason this is being fixed for 3.0.1? 4.0 looks better choice to me since this is just refactoring patch. > Remove unused classes from Hive QL > -- > > Key: HIVE-20256 > URL: https://issues.apache.org/jira/browse/HIVE-20256 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20256.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560281#comment-16560281
]
Hive QA commented on HIVE-19097:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 35m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2297 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 1 new + 44 unchanged - 3 fixed
= 45 total (was 47) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12904/dev-support/hive-personality.sh
|
| git revision | master / 0ad7112 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12904/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12904/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch, HIVE-19097.02.patch,
> HIVE-19097.03.patch, HIVE-19097.04.patch, HIVE-19097.partial.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data
[jira] [Commented] (HIVE-20156) Printing Stacktrace to STDERR
[
https://issues.apache.org/jira/browse/HIVE-20156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560278#comment-16560278
]
Andrew Sherman commented on HIVE-20156:
---
Thanks [~ngangam]
> Printing Stacktrace to STDERR
> -
>
> Key: HIVE-20156
> URL: https://issues.apache.org/jira/browse/HIVE-20156
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20156.1.patch
>
>
> Class {{org.apache.hadoop.hive.ql.exec.JoinOperator}} has the following code:
> {code}
> } catch (Exception e) {
> e.printStackTrace();
> throw new HiveException(e);
> }
> {code}
> Do not print the stack trace to STDERR with a call to {{printStackTrace()}}.
> Please remove that line and let the code catching the {{HiveException}} worry
> about printing any messages through a logger.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18929) The method humanReadableInt in HiveStringUtils.java has a race condition.
[
https://issues.apache.org/jira/browse/HIVE-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560277#comment-16560277
]
Andrew Sherman commented on HIVE-18929:
---
Thanks [~aihuaxu]
> The method humanReadableInt in HiveStringUtils.java has a race condition.
> -
>
> Key: HIVE-18929
> URL: https://issues.apache.org/jira/browse/HIVE-18929
> Project: Hive
> Issue Type: Bug
> Components: API
>Affects Versions: 2.3.2
>Reporter: Chaiyong Ragkhitwetsagul
>Assignee: Andrew Sherman
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-18929.1.patch
>
>
> I found that the {{humanReadableInt(long number)}} method in the
> hive/common/src/java/org/apache/hive/common/util/HiveStringUtils.java file
> contains code which has a race condition as shown in Hadoop (issue tracking
> ID HADOOP-9252: https://issues.apache.org/jira/browse/HADOOP-9252). The fix
> can also be seen in the Hadoop code base.
> I couldn't find a call to the method anywhere else in the code. But it might
> be worth to fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HIVE-17684:
--
Status: Patch Available (was: In Progress)
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HIVE-17684:
--
Attachment: HIVE-17684.04.patch
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HIVE-17684:
--
Status: In Progress (was: Patch Available)
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20258) Should Syncronize getInstance in ReplChangeManager
[
https://issues.apache.org/jira/browse/HIVE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20258:
---
Status: Patch Available (was: Open)
> Should Syncronize getInstance in ReplChangeManager
> --
>
> Key: HIVE-20258
> URL: https://issues.apache.org/jira/browse/HIVE-20258
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20258.1.patch
>
>
> {code:java}
> public static ReplChangeManager getInstance(Configuration conf) throws
> MetaException {
> if (instance == null) {
> instance = new ReplChangeManager(conf);
> }
> return instance;
> }
> {code}
> This method needs to be synchronized or two different callers will see a
> 'null' value and each create their own manager.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20258) Should Syncronize getInstance in ReplChangeManager
[
https://issues.apache.org/jira/browse/HIVE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20258:
---
Attachment: HIVE-20258.1.patch
> Should Syncronize getInstance in ReplChangeManager
> --
>
> Key: HIVE-20258
> URL: https://issues.apache.org/jira/browse/HIVE-20258
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20258.1.patch
>
>
> {code:java}
> public static ReplChangeManager getInstance(Configuration conf) throws
> MetaException {
> if (instance == null) {
> instance = new ReplChangeManager(conf);
> }
> return instance;
> }
> {code}
> This method needs to be synchronized or two different callers will see a
> 'null' value and each create their own manager.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20258) Should Syncronize getInstance in ReplChangeManager
[
https://issues.apache.org/jira/browse/HIVE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-20258:
--
Assignee: BELUGA BEHR
> Should Syncronize getInstance in ReplChangeManager
> --
>
> Key: HIVE-20258
> URL: https://issues.apache.org/jira/browse/HIVE-20258
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
>
> {code:java}
> public static ReplChangeManager getInstance(Configuration conf) throws
> MetaException {
> if (instance == null) {
> instance = new ReplChangeManager(conf);
> }
> return instance;
> }
> {code}
> This method needs to be synchronized or two different callers will see a
> 'null' value and each create their own manager.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20238:
---
Status: Patch Available (was: Open)
Submitted a patch.
I reviewed the entire StringUtils class. The biggest issue was that it is a
bit inconsistent about when it returns 'null' v.s. an Empty collection. I
strongly suggest that it returns empty collections to avoid NPE.
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch
>
>
> Remove the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20238:
---
Attachment: HIVE-20238.1.patch
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch
>
>
> Remove the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-20238:
--
Assignee: BELUGA BEHR
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
>
> Remove the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20256) Remove unused classes from Hive QL
[
https://issues.apache.org/jira/browse/HIVE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560166#comment-16560166
]
Hive QA commented on HIVE-20256:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933395/HIVE-20256.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14813 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12903/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12903/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12903/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12933395 - PreCommit-HIVE-Build
> Remove unused classes from Hive QL
> --
>
> Key: HIVE-20256
> URL: https://issues.apache.org/jira/browse/HIVE-20256
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 3.0.1
>
> Attachments: HIVE-20256.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20257) Improvements to Hive.java
[
https://issues.apache.org/jira/browse/HIVE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20257:
---
Attachment: HIVE-20257.1.patch
> Improvements to Hive.java
> -
>
> Key: HIVE-20257
> URL: https://issues.apache.org/jira/browse/HIVE-20257
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20257.1.patch
>
>
> Various fixes to {{Hive.java}}
> * Use Log4J parameters in logging statements
> * Fix check styles
> * Make code more concise
> * Remove "log and throw" code
> * Replaced calls to deprecated code
> * Removed superfluous calls to {{toString}}
>
> "log and throw" is considered and anti-pattern. Only the highest level catch
> should be providing detailed logging otherwise we print the same stack trace
> to the logs several times and with different context (for example when an
> exception is wrapped, we get two different logging events).
>
> https://community.oracle.com/docs/DOC-983543#logAndThrow
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20257) Improvements to Hive.java
[
https://issues.apache.org/jira/browse/HIVE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20257:
---
Status: Patch Available (was: Open)
> Improvements to Hive.java
> -
>
> Key: HIVE-20257
> URL: https://issues.apache.org/jira/browse/HIVE-20257
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20257.1.patch
>
>
> Various fixes to {{Hive.java}}
> * Use Log4J parameters in logging statements
> * Fix check styles
> * Make code more concise
> * Remove "log and throw" code
> * Replaced calls to deprecated code
> * Removed superfluous calls to {{toString}}
>
> "log and throw" is considered and anti-pattern. Only the highest level catch
> should be providing detailed logging otherwise we print the same stack trace
> to the logs several times and with different context (for example when an
> exception is wrapped, we get two different logging events).
>
> https://community.oracle.com/docs/DOC-983543#logAndThrow
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20257) Improvements to Hive.java
[
https://issues.apache.org/jira/browse/HIVE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-20257:
--
Assignee: BELUGA BEHR
> Improvements to Hive.java
> -
>
> Key: HIVE-20257
> URL: https://issues.apache.org/jira/browse/HIVE-20257
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20257.1.patch
>
>
> Various fixes to {{Hive.java}}
> * Use Log4J parameters in logging statements
> * Fix check styles
> * Make code more concise
> * Remove "log and throw" code
> * Replaced calls to deprecated code
> * Removed superfluous calls to {{toString}}
>
> "log and throw" is considered and anti-pattern. Only the highest level catch
> should be providing detailed logging otherwise we print the same stack trace
> to the logs several times and with different context (for example when an
> exception is wrapped, we get two different logging events).
>
> https://community.oracle.com/docs/DOC-983543#logAndThrow
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20256) Remove unused classes from Hive QL
[
https://issues.apache.org/jira/browse/HIVE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560127#comment-16560127
]
Hive QA commented on HIVE-20256:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 2297 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} ql: The patch generated 0 new + 0 unchanged - 61
fixed = 0 total (was 61) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} ql generated 0 new + 2291 unchanged - 6 fixed = 2291
total (was 2297) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12903/dev-support/hive-personality.sh
|
| git revision | master / 9aac806 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12903/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove unused classes from Hive QL
> --
>
> Key: HIVE-20256
> URL: https://issues.apache.org/jira/browse/HIVE-20256
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 3.0.1
>
> Attachments: HIVE-20256.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20153) Count and Sum UDF consume more memory in Hive 2+
[ https://issues.apache.org/jira/browse/HIVE-20153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560102#comment-16560102 ] Szehon Ho commented on HIVE-20153: -- Thanks Aihua for the fix. Yes I can test it, I am out of town at the moment so need to wait to get back, and hope I can do it sometime next week. If you dont want to wait, feel free to go ahead, I can comment my findings afterwards. > Count and Sum UDF consume more memory in Hive 2+ > > > Key: HIVE-20153 > URL: https://issues.apache.org/jira/browse/HIVE-20153 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 2.3.2 >Reporter: Szehon Ho >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-20153.1.patch, Screen Shot 2018-07-12 at 6.41.28 > PM.png > > > While playing with Hive2, we noticed that queries with a lot of count() and > sum() aggregations run out of memory on Hadoop side where they worked before > in Hive1. > In many queries, we have to double the Mapper Memory settings (in our > particular case mapreduce.map.java.opts from -Xmx2000M to -Xmx4000M), it > makes it not so easy to upgrade to Hive 2. > Taking heap dump, we see one of the main culprit is the field 'uniqueObjects' > in GeneraicUDAFSum and GenericUDAFCount, which was added to support Window > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20241) Support partitioning spec in CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-20241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560091#comment-16560091
]
Jesus Camacho Rodriguez commented on HIVE-20241:
[~ashutoshc], can you take a look?
https://reviews.apache.org/r/68086/
Thanks
> Support partitioning spec in CTAS statements
>
>
> Key: HIVE-20241
> URL: https://issues.apache.org/jira/browse/HIVE-20241
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20241.01.patch, HIVE-20241.01.patch,
> HIVE-20241.patch
>
>
> Currently, for partitioned tables we will declare the table and insert the
> data in different operations. This issue is to extend CTAS statement to
> support specifying partition columns.
> For instance:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20241) Support partitioning spec in CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-20241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20241:
---
Attachment: HIVE-20241.01.patch
> Support partitioning spec in CTAS statements
>
>
> Key: HIVE-20241
> URL: https://issues.apache.org/jira/browse/HIVE-20241
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20241.01.patch, HIVE-20241.01.patch,
> HIVE-20241.patch
>
>
> Currently, for partitioned tables we will declare the table and insert the
> data in different operations. This issue is to extend CTAS statement to
> support specifying partition columns.
> For instance:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20241) Support partitioning spec in CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-20241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560090#comment-16560090
]
Hive QA commented on HIVE-20241:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933391/HIVE-20241.01.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14815 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestServiceDiscovery.org.apache.hive.jdbc.TestServiceDiscovery
(batchId=251)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12902/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12902/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12902/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12933391 - PreCommit-HIVE-Build
> Support partitioning spec in CTAS statements
>
>
> Key: HIVE-20241
> URL: https://issues.apache.org/jira/browse/HIVE-20241
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20241.01.patch, HIVE-20241.patch
>
>
> Currently, for partitioned tables we will declare the table and insert the
> data in different operations. This issue is to extend CTAS statement to
> support specifying partition columns.
> For instance:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HIVE-20226) HMS getNextNotification will throw exception when request maxEvents exceed table's max_rows
[ https://issues.apache.org/jira/browse/HIVE-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan reopened HIVE-20226: -- > HMS getNextNotification will throw exception when request maxEvents exceed > table's max_rows > --- > > Key: HIVE-20226 > URL: https://issues.apache.org/jira/browse/HIVE-20226 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0, 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20226.1.patch, HIVE-20226.2.patch, > HIVE-20226.3.patch > > > When Sentry call getNextNotification(), its request max_events will always be > Integer.MAX_VALUE. > query.setRange(0, maxEvents) of ObjectStore will throw exception when > maxEvents exceed table's max_rows. > Ex: assumed max_rows = 50,000,000 in a mysql table > java.sql.SQLException: setMaxRows() out of range. 2147483647 > 5000. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20256) Remove unused classes from Hive QL
[ https://issues.apache.org/jira/browse/HIVE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560074#comment-16560074 ] Gopal V edited comment on HIVE-20256 at 7/27/18 6:00 PM: - VectorColumnAssignFactory is not being used? Public classes needs Deprecation and rebuild things which depend on Hive downstream entirely before removing wholesale (like storage-api -> orc -> llap-server). Specifically - ORC & Hive compiles against an existing storage-api release, so the builds won't break until the next release happens. was (Author: gopalv): VectorColumnAssignFactory is not being used? Some of this stuff needs Deprecation and rebuild things which depend on Hive downstream entirely before removing wholesale (like storage-api -> orc -> llap-server). Specifically - ORC & Hive compiles against an existing storage-api release, so the builds won't break until the next release happens. > Remove unused classes from Hive QL > -- > > Key: HIVE-20256 > URL: https://issues.apache.org/jira/browse/HIVE-20256 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20256.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20256) Remove unused classes from Hive QL
[ https://issues.apache.org/jira/browse/HIVE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560074#comment-16560074 ] Gopal V edited comment on HIVE-20256 at 7/27/18 5:57 PM: - VectorColumnAssignFactory is not being used? Some of this stuff needs Deprecation and rebuild things which depend on Hive downstream entirely before removing wholesale (like storage-api -> orc -> llap-server). Specifically - ORC & Hive compiles against an existing storage-api release, so the builds won't break until the next release happens. was (Author: gopalv): VectorColumnAssignFactory is not being used? Some of this stuff needs Deprecation and rebuild things which depend on Hive downstream entirely before removing wholesale (like storage-api -> orc -> llap-server). > Remove unused classes from Hive QL > -- > > Key: HIVE-20256 > URL: https://issues.apache.org/jira/browse/HIVE-20256 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20256.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20256) Remove unused classes from Hive QL
[ https://issues.apache.org/jira/browse/HIVE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560074#comment-16560074 ] Gopal V commented on HIVE-20256: VectorColumnAssignFactory is not being used? Some of this stuff needs Deprecation and rebuild things which depend on Hive downstream entirely before removing wholesale (like storage-api -> orc -> llap-server). > Remove unused classes from Hive QL > -- > > Key: HIVE-20256 > URL: https://issues.apache.org/jira/browse/HIVE-20256 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20256.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20226) HMS getNextNotification will throw exception when request maxEvents exceed table's max_rows
[ https://issues.apache.org/jira/browse/HIVE-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560070#comment-16560070 ] Alice Fan commented on HIVE-20226: -- Thank you [~ychena]! > HMS getNextNotification will throw exception when request maxEvents exceed > table's max_rows > --- > > Key: HIVE-20226 > URL: https://issues.apache.org/jira/browse/HIVE-20226 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0, 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20226.1.patch, HIVE-20226.2.patch, > HIVE-20226.3.patch > > > When Sentry call getNextNotification(), its request max_events will always be > Integer.MAX_VALUE. > query.setRange(0, maxEvents) of ObjectStore will throw exception when > maxEvents exceed table's max_rows. > Ex: assumed max_rows = 50,000,000 in a mysql table > java.sql.SQLException: setMaxRows() out of range. 2147483647 > 5000. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20226) HMS getNextNotification will throw exception when request maxEvents exceed table's max_rows
[ https://issues.apache.org/jira/browse/HIVE-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20226: - Resolution: Fixed Status: Resolved (was: Patch Available) > HMS getNextNotification will throw exception when request maxEvents exceed > table's max_rows > --- > > Key: HIVE-20226 > URL: https://issues.apache.org/jira/browse/HIVE-20226 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0, 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20226.1.patch, HIVE-20226.2.patch, > HIVE-20226.3.patch > > > When Sentry call getNextNotification(), its request max_events will always be > Integer.MAX_VALUE. > query.setRange(0, maxEvents) of ObjectStore will throw exception when > maxEvents exceed table's max_rows. > Ex: assumed max_rows = 50,000,000 in a mysql table > java.sql.SQLException: setMaxRows() out of range. 2147483647 > 5000. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20241) Support partitioning spec in CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-20241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560050#comment-16560050
]
Hive QA commented on HIVE-20241:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2297 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 10 new + 832 unchanged - 14
fixed = 842 total (was 846) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
22s{color} | {color:red} ql generated 3 new + 2294 unchanged - 3 fixed = 2297
total (was 2297) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA230 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46716-46729] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA274 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
47066-47079] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA289 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
47202-47215] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12902/dev-support/hive-personality.sh
|
| git revision | master / 9aac806 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12902/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12902/yetus/new-findbugs-ql.html
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12902/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support partitioning spec in CTAS statements
>
>
> Key: HIVE-20241
> URL: https://issues.apache.org/jira/browse/HIVE-20241
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20241.01.patch, HIVE-20241.patch
>
>
> Currently, for partitioned tables we will declare the table and insert the
> data
[jira] [Updated] (HIVE-20212) Hiveserver2 in http mode emitting metric default.General.open_connections incorrectly
[ https://issues.apache.org/jira/browse/HIVE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20212: --- Attachment: HIVE-20212.03.patch > Hiveserver2 in http mode emitting metric default.General.open_connections > incorrectly > - > > Key: HIVE-20212 > URL: https://issues.apache.org/jira/browse/HIVE-20212 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-20212.01.patch, HIVE-20212.01.patch, > HIVE-20212.02.patch, HIVE-20212.03.patch, HIVE-20212.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19097:
Attachment: HIVE-19097.04.patch
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch, HIVE-19097.02.patch,
> HIVE-19097.03.patch, HIVE-19097.04.patch, HIVE-19097.partial.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
