[jira] [Commented] (HIVE-20659) Update commons-compress to 1.18 due to security issues
[ https://issues.apache.org/jira/browse/HIVE-20659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635056#comment-16635056 ] Bharathkrishna Guruvayoor Murali commented on HIVE-20659: - Should we be making changes on all the versions, or just in the current (4.0)? I can take this up if nobody is working on it. > Update commons-compress to 1.18 due to security issues > -- > > Key: HIVE-20659 > URL: https://issues.apache.org/jira/browse/HIVE-20659 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1, 3.0.0, 2.3.2, 3.1.0 >Reporter: Jörn Franke >Priority: Critical > > Currently most Hive version depends on commons-compress 1.9 or 1.4. Those > versions have several security issues: > [https://commons.apache.org/proper/commons-compress/security-reports.html] > I propose to upgrade all commons-compress dependencies in all Hive > (sub-)projects to at least 1.18. This will also make it easier for future > extensions to Hive (serde, udfs, etc.) that have dependencies to > commons-compress (e.g. [https://github.com/zuinnote/hadoopoffice/wiki)] to > integrate into Hive without upgrading the commons-compress library manually > in the Hive lib folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20648) LLAP: Vector group by operator should use memory per executor
[
https://issues.apache.org/jira/browse/HIVE-20648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635046#comment-16635046
]
Hive QA commented on HIVE-20648:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 15 new + 565 unchanged - 0
fixed = 580 total (was 565) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14170/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14170/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14170/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Vector group by operator should use memory per executor
> -
>
> Key: HIVE-20648
> URL: https://issues.apache.org/jira/browse/HIVE-20648
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20648.1.patch, HIVE-20648.2.patch
>
>
> HIVE-15503 treatment has to be applied for vector group by operator as well.
> Vector group by currently uses MemoryMX bean to get heap usage and heap max
> memory which will not work for LLAP. Instead it should use memory per
> executor as upper bound to make flush decision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635026#comment-16635026
]
Hive QA commented on HIVE-20563:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942017/HIVE-20563.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15008 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_expressions]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_case_when_conversion]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_expressions]
(batchId=172)
org.apache.hadoop.hive.ql.exec.vector.TestVectorizationContext.testIfConditionalExprs
(batchId=301)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14169/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14169/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14169/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942017 - PreCommit-HIVE-Build
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
>
[jira] [Commented] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634974#comment-16634974
]
Hive QA commented on HIVE-20563:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14169/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14169/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>
[jira] [Commented] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1
[
https://issues.apache.org/jira/browse/HIVE-20661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634876#comment-16634876
]
Hive QA commented on HIVE-20661:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941983/HIVE-20661.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 56 failed/errored test(s), 15007 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[buckets]
(batchId=268)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=268)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_buckets]
(batchId=268)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_buckets]
(batchId=268)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_buckets]
(batchId=268)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_insert_overwrite]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats2] (batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_3]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_partition_insert]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lock3] (batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lock4] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_partitioned]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_incompat2]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=21)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_dp]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_non_mm]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge_incompat2]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_input_counters]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_incompat2]
(batchId=191)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPart (batchId=298)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPartPartial (batchId=298)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPartPartial2 (batchId=298)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPartPartial3 (batchId=298)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportPartitionedCreate (batchId=298)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportPartitionedCreate2 (batchId=298)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNoBucketsDP (batchId=300)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testNoBucketsDP
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking7
(batchId=315)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidBootstrapReplLoadRetryAfterFailure
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesBootstrap
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesBootstrapWithOpenTxnsTimeout
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesMoveOptimizationBootStrap
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesMoveOptimizationIncremental
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithPartitionDirMissing
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testMultiDBTxn
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testOpenTxnEvent
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testTxnEventNonAcid
[jira] [Commented] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1
[
https://issues.apache.org/jira/browse/HIVE-20661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634869#comment-16634869
]
Hive QA commented on HIVE-20661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 10 new + 216 unchanged - 3
fixed = 226 total (was 219) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
17s{color} | {color:red} ql generated 2 new + 2321 unchanged - 0 fixed = 2323
total (was 2321) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Invocation of toString on java.util.Set.toArray(Object[]) in
org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Path, String,
Map, LoadTableDesc$LoadFileType, int, int, boolean, long, int, boolean,
AcidUtils$Operation, boolean) At Hive.java:in
org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Path, String,
Map, LoadTableDesc$LoadFileType, int, int, boolean, long, int, boolean,
AcidUtils$Operation, boolean) At Hive.java:[line 2543] |
| | org.apache.hadoop.hive.ql.metadata.Hive.addPartitionsToMetastore(List,
boolean, Table) concatenates strings using + in a loop At Hive.java:using + in
a loop At Hive.java:[line 2131] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14168/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14168/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14168/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14168/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Dynamic partitions loading calls add partition for every partition 1-by-1
> -
>
> Key: HIVE-20661
> URL: https://issues.apache.org/jira/browse/HIVE-20661
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20661.01.patch
>
>
> Hive.loadDynamicPartitions creates partitions using a threadpool, but the
> update of the Metastore via the MetastoreClient is done 1-by-1. This adds
> unnecessary
[jira] [Commented] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634868#comment-16634868
]
Jesus Camacho Rodriguez commented on HIVE-20652:
[~daijy], should I take a look at this one? Thanks
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
>
[jira] [Assigned] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-20652:
--
Assignee: Jesus Camacho Rodriguez
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Commented] (HIVE-20657) pre-allocate LLAP cache at init time
[
https://issues.apache.org/jira/browse/HIVE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634853#comment-16634853
]
Hive QA commented on HIVE-20657:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942003/HIVE-20657.01.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15007 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_related_col]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_context_ngrams]
(batchId=78)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14167/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14167/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14167/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942003 - PreCommit-HIVE-Build
> pre-allocate LLAP cache at init time
>
>
> Key: HIVE-20657
> URL: https://issues.apache.org/jira/browse/HIVE-20657
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20657.01.patch, HIVE-20657.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20657) pre-allocate LLAP cache at init time
[
https://issues.apache.org/jira/browse/HIVE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634836#comment-16634836
]
Hive QA commented on HIVE-20657:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} llap-server in master has 84 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
20s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 20s{color}
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} llap-server: The patch generated 1 new + 125 unchanged
- 1 fixed = 126 total (was 126) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
20s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14167/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14167/yetus/patch-mvninstall-llap-server.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14167/yetus/patch-compile-llap-server.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14167/yetus/patch-compile-llap-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14167/yetus/diff-checkstyle-llap-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14167/yetus/patch-findbugs-llap-server.txt
|
| modules | C: common llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14167/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> pre-allocate LLAP cache at init time
>
>
> Key: HIVE-20657
> URL: https://issues.apache.org/jira/browse/HIVE-20657
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20657.01.patch, HIVE-20657.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-11166) HiveHBaseTableOutputFormat can't call getFileExtension(JobConf jc, boolean isCompressed, HiveOutputFormat hiveOutputFormat)
[
https://issues.apache.org/jira/browse/HIVE-11166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634830#comment-16634830
]
Alice Fan commented on HIVE-11166:
--
Hi [~Yun Zhao], the issue still exists in current master branch (hive 4.0.0).
Do you want to complete the patch? Thanks.
> HiveHBaseTableOutputFormat can't call getFileExtension(JobConf jc, boolean
> isCompressed, HiveOutputFormat hiveOutputFormat)
> -
>
> Key: HIVE-11166
> URL: https://issues.apache.org/jira/browse/HIVE-11166
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler, Spark
>Reporter: meiyoula
>Assignee: Yun Zhao
>Priority: Major
> Attachments: HIVE-11166.2.patch, HIVE-11166.patch
>
>
> I create a hbase table with HBaseStorageHandler in JDBCServer of spark, then
> execute the *insert into* sql statement, ClassCastException occurs.
> {quote}
> Error: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in
> stage 3.0 (TID 12, vm-17): java.lang.ClassCastException:
> org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat cannot be cast to
> org.apache.hadoop.hive.ql.io.HiveOutputFormat
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.outputFormat$lzycompute(hiveWriterContainers.scala:72)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.outputFormat(hiveWriterContainers.scala:71)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.getOutputName(hiveWriterContainers.scala:91)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.initWriters(hiveWriterContainers.scala:115)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.executorSideSetup(hiveWriterContainers.scala:84)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1(InsertIntoHiveTable.scala:112)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:93)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:93)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:56)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> {quote}
> It's because the code in spark below. To hbase table, the outputFormat is
> HiveHBaseTableOutputFormat, it isn't instanceOf[HiveOutputForm
> at].
> {quote}
> @transient private lazy val
> outputFormat=conf.value.getOutputFormat.asInstanceOf[HiveOutputForm
> at[AnyRef, Writable]]
> val extension = Utilities.getFileExtension(conf.value,
> fileSinkConf.getCompressed, outputFormat)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634823#comment-16634823
]
Hive QA commented on HIVE-20665:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941992/HIVE-20665.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15007 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_handler_bulk]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz2]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[druid_address]
(batchId=98)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14166/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14166/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14166/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941992 - PreCommit-HIVE-Build
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL: https://issues.apache.org/jira/browse/HIVE-20665
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-20665.1.patch
>
>
> When parallel tasks are enabled in Hive, all of the resulting queries share
> the same Hive configuration. This is problematic as each query will modify
> the same {{HiveConf}} object with things like query ID and query text. This
> will overwrite each other and cause {{ConcurrentModificationException}}
> issues.
> {code:java|title=SQLOperation.java}
> public Object run() throws HiveSQLException {
> Hive.set(parentHive, false);
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(SessionState.getPerfLogger());
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // ...
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
> From this code it can be seen that for every thread launched, it is all
> calling {{setCurrentSessionState}}.
> {code:java|title=SessionStates.java}
> /**
>* Sets the given session state in the thread local var for sessions.
>*/
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> // SessionState is not available in runtime and Hive.get().getConf() is not
> safe to call
> private static class SessionStates {
> private SessionState state;
> private HiveConf conf;
> private void attach(SessionState state) {
> this.state = state;
> attach(state.getConf());
> }
> private void attach(HiveConf conf) {
> // -- SHALLOW COPY HERE, ALL THREADS SHARING SAME REFERENCE -- //
> this.conf = conf;
> ClassLoader classLoader = conf.getClassLoader();
> if (classLoader != null) {
> Thread.currentThread().setContextClassLoader(classLoader);
> }
> }
> }
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/7795c0a7dc59941671f8845d78b16d9e5ddc9ea3/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L540-L556]
> Ensure that all threads get their own copy of the {{HiveConf}} object to use
> and modify.
--
This message was sent by Atlassian JIRA
[jira] [Assigned] (HIVE-20608) Incorrect handling of sql command args in hive service leading to misleading error messages
[
https://issues.apache.org/jira/browse/HIVE-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Jiang reassigned HIVE-20608:
---
Assignee: Li Jiang (was: Soumabrata Chakraborty)
> Incorrect handling of sql command args in hive service leading to misleading
> error messages
> ---
>
> Key: HIVE-20608
> URL: https://issues.apache.org/jira/browse/HIVE-20608
> Project: Hive
> Issue Type: Bug
>Reporter: Soumabrata Chakraborty
>Assignee: Li Jiang
>Priority: Major
>
> *Steps to reproduce:*
> (1) Connect to HiveServer2 using JDBC driver (not via Beeline)
> (2) Execute a set command with a space before set – e.g. " set
> hive.exec.dynamic.partiton=true"
> (3) The error that is returned says:
> {code:java}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> processing statement: Cannot modify set hive.exec.dynamic.partition at
> runtime. It is not in list of params that are allowed to be modified at
> runtime
> {code}
> (4) However on removing the space before the set command - it works fine.
>
> *Analysis:*
> Looks like an issue with
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java]
>
> In the runInternal() method, the untrimmed sql statement is split by space
> (\\s) but the substring operation to get the command args is done on the
> trimmed sql statement. This causes the issue.
> {code:java}
> String command = getStatement().trim();
> String[] tokens = statement.split("\\s");
> String commandArgs = command.substring(tokens[0].length()).trim();
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634803#comment-16634803
]
Hive QA commented on HIVE-20665:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14166/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14166/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL: https://issues.apache.org/jira/browse/HIVE-20665
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-20665.1.patch
>
>
> When parallel tasks are enabled in Hive, all of the resulting queries share
> the same Hive configuration. This is problematic as each query will modify
> the same {{HiveConf}} object with things like query ID and query text. This
> will overwrite each other and cause {{ConcurrentModificationException}}
> issues.
> {code:java|title=SQLOperation.java}
> public Object run() throws HiveSQLException {
> Hive.set(parentHive, false);
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(SessionState.getPerfLogger());
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // ...
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
> From this code it can be seen that for every thread launched, it is all
>
[jira] [Commented] (HIVE-20623) Shared work: Extend sharing of map-join cache entries in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634794#comment-16634794
]
Hive QA commented on HIVE-20623:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941996/HIVE-20623.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15007 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14165/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14165/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14165/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941996 - PreCommit-HIVE-Build
> Shared work: Extend sharing of map-join cache entries in LLAP
> -
>
> Key: HIVE-20623
> URL: https://issues.apache.org/jira/browse/HIVE-20623
> Project: Hive
> Issue Type: Improvement
> Components: llap, Logical Optimizer
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20623.01.patch, HIVE-20623.02.patch,
> HIVE-20623.02.patch, HIVE-20623.02.patch, HIVE-20623.03.patch,
> HIVE-20623.03.patch, HIVE-20623.03.patch, HIVE-20623.04.patch,
> HIVE-20623.patch, hash-shared-work.json.txt, hash-shared-work.svg
>
>
> For a query like this
> {code}
> with all_sales as (
> select ss_customer_sk as customer_sk, ss_ext_list_price-ss_ext_discount_amt
> as ext_price from store_sales
> UNION ALL
> select ws_bill_customer_sk as customer_sk,
> ws_ext_list_price-ws_ext_discount_amt as ext_price from web_sales
> UNION ALL
> select cs_bill_customer_sk as customer_sk, cs_ext_sales_price -
> cs_ext_discount_amt as ext_price from catalog_sales)
> select sum(ext_price) total_price, c_customer_id from all_sales, customer
> where customer_sk = c_customer_sk
> group by c_customer_id
> order by total_price desc
> limit 100;
> {code}
> The hashtable used for all 3 joins are identical, which is loaded 3x times in
> the same LLAP instance because they are named.
> {code}
> cacheKey = "HASH_MAP_" + this.getOperatorId() + "_container";
> {code}
> in the cache.
> If those are identical in nature (i.e vectorization, hashtable type etc),
> then the duplication is just wasted CPU, memory and network - using the cache
> name for hashtables which will be identical in layout would be extremely
> useful.
> In cases where the join is pushed through a UNION, those are identical.
> This optimization can only be done without concern for accidental delays when
> the same upstream task is generating all of these hashtables, which is what
> is achieved by the shared scan optimizer already.
> In case the shared work is not present, this has potential downsides - in
> case two customer broadcasts were sourced from "Map 1" and "Map 2", the Map 1
> builder will block the other task from reading from Map 2, even though Map 2
> might have started after, but finished ahead of Map 1.
> So this specific optimization can always be considered for cases where the
> shared work unifies the operator tree and the parents of all the RS entries
> involved are same (& the RS layout is the same).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20623) Shared work: Extend sharing of map-join cache entries in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634773#comment-16634773
]
Hive QA commented on HIVE-20623:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14165/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14165/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Shared work: Extend sharing of map-join cache entries in LLAP
> -
>
> Key: HIVE-20623
> URL: https://issues.apache.org/jira/browse/HIVE-20623
> Project: Hive
> Issue Type: Improvement
> Components: llap, Logical Optimizer
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20623.01.patch, HIVE-20623.02.patch,
> HIVE-20623.02.patch, HIVE-20623.02.patch, HIVE-20623.03.patch,
> HIVE-20623.03.patch, HIVE-20623.03.patch, HIVE-20623.04.patch,
> HIVE-20623.patch, hash-shared-work.json.txt, hash-shared-work.svg
>
>
> For a query like this
> {code}
> with all_sales as (
> select ss_customer_sk as customer_sk, ss_ext_list_price-ss_ext_discount_amt
> as ext_price from store_sales
> UNION ALL
> select ws_bill_customer_sk as customer_sk,
> ws_ext_list_price-ws_ext_discount_amt as ext_price from web_sales
> UNION ALL
> select cs_bill_customer_sk as customer_sk, cs_ext_sales_price -
> cs_ext_discount_amt as ext_price from catalog_sales)
> select sum(ext_price) total_price, c_customer_id from all_sales, customer
> where customer_sk = c_customer_sk
>
[jira] [Commented] (HIVE-20662) Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
[
https://issues.apache.org/jira/browse/HIVE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634739#comment-16634739
]
Hive QA commented on HIVE-20662:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941990/HIVE-20662.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15006 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14164/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14164/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14164/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941990 - PreCommit-HIVE-Build
> Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
>
>
> Key: HIVE-20662
> URL: https://issues.apache.org/jira/browse/HIVE-20662
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Blocker
> Attachments: HIVE-20662.patch
>
>
> TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] is more than flaky.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables
[ https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20556: --- Attachment: HIVE-20556.18.patch Status: Patch Available (was: Open) > Expose an API to retrieve the TBL_ID from TBLS in the metastore tables > -- > > Key: HIVE-20556 > URL: https://issues.apache.org/jira/browse/HIVE-20556 > Project: Hive > Issue Type: New Feature > Components: Metastore, Standalone Metastore >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20556.1.patch, HIVE-20556.10.patch, > HIVE-20556.11.patch, HIVE-20556.12.patch, HIVE-20556.13.patch, > HIVE-20556.14.patch, HIVE-20556.15.patch, HIVE-20556.16.patch, > HIVE-20556.17.patch, HIVE-20556.18.patch, HIVE-20556.2.patch, > HIVE-20556.3.patch, HIVE-20556.4.patch, HIVE-20556.5.patch, > HIVE-20556.6.patch, HIVE-20556.7.patch, HIVE-20556.8.patch, HIVE-20556.9.patch > > > We have two options to do this > 1) Use the current MTable and add a field for this value > 2) Add an independent API call to the metastore that would return the TBL_ID. > Option 1 is preferable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables
[ https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20556: --- Status: Open (was: Patch Available) > Expose an API to retrieve the TBL_ID from TBLS in the metastore tables > -- > > Key: HIVE-20556 > URL: https://issues.apache.org/jira/browse/HIVE-20556 > Project: Hive > Issue Type: New Feature > Components: Metastore, Standalone Metastore >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20556.1.patch, HIVE-20556.10.patch, > HIVE-20556.11.patch, HIVE-20556.12.patch, HIVE-20556.13.patch, > HIVE-20556.14.patch, HIVE-20556.15.patch, HIVE-20556.16.patch, > HIVE-20556.17.patch, HIVE-20556.18.patch, HIVE-20556.2.patch, > HIVE-20556.3.patch, HIVE-20556.4.patch, HIVE-20556.5.patch, > HIVE-20556.6.patch, HIVE-20556.7.patch, HIVE-20556.8.patch, HIVE-20556.9.patch > > > We have two options to do this > 1) Use the current MTable and add a field for this value > 2) Add an independent API call to the metastore that would return the TBL_ID. > Option 1 is preferable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId
[ https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634728#comment-16634728 ] Prasanth Jayachandran commented on HIVE-20291: -- +1 > Allow HiveStreamingConnection to receive a WriteId > -- > > Key: HIVE-20291 > URL: https://issues.apache.org/jira/browse/HIVE-20291 > Project: Hive > Issue Type: Improvement >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20291.1.patch, HIVE-20291.2.patch, > HIVE-20291.3.patch, HIVE-20291.4.patch, HIVE-20291.5.patch, > HIVE-20291.6.patch, HIVE-20291.7.patch, HIVE-20291.8.patch > > > If the writeId is received externally it won't need to open connections to > the metastore. It won't be able to the commit in this case as well so it must > be done by the entity passing the writeId. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20667) fix bug - CREATE TABLE tabletype problem while executing addDbAndTabToOutputs
[ https://issues.apache.org/jira/browse/HIVE-20667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maolin Tu updated HIVE-20667: - Description: problem is in [ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java|https://github.com/apache/hive/pull/426/files#diff-f2040374293a91cbcc6594ee571b20e4] In the release version of Hive, outputs'TableType always be MANAGED_TABLE, no matter the value of isExt is true of false. While I using post_execute_hook, I found this problem because I cannot get correct tabletype in HiveHookContext. pull request: https://github.com/apache/hive/pull/426/files was: problem is in [ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java|https://github.com/apache/hive/pull/426/files#diff-f2040374293a91cbcc6594ee571b20e4] In the release version of Hive, outputs'TableType always be MANAGED_TABLE, no matter the value of isExt is true of false. While I using post_execute_hook, I found this problem because I cannot get correct tabletype in HiveHookContext. > fix bug - CREATE TABLE tabletype problem while executing addDbAndTabToOutputs > - > > Key: HIVE-20667 > URL: https://issues.apache.org/jira/browse/HIVE-20667 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.3.1, 3.1.0 >Reporter: Maolin Tu >Assignee: Maolin Tu >Priority: Minor > > problem is in > [ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java|https://github.com/apache/hive/pull/426/files#diff-f2040374293a91cbcc6594ee571b20e4] > In the release version of Hive, outputs'TableType always be MANAGED_TABLE, no > matter the value of isExt is true of false. While I using post_execute_hook, > I found this problem because I cannot get correct tabletype in > HiveHookContext. > > pull request: https://github.com/apache/hive/pull/426/files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20667) fix bug - CREATE TABLE tabletype problem while executing addDbAndTabToOutputs
[ https://issues.apache.org/jira/browse/HIVE-20667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maolin Tu reassigned HIVE-20667: > fix bug - CREATE TABLE tabletype problem while executing addDbAndTabToOutputs > - > > Key: HIVE-20667 > URL: https://issues.apache.org/jira/browse/HIVE-20667 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0, 2.3.1 >Reporter: Maolin Tu >Assignee: Maolin Tu >Priority: Minor > > problem is in > [ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java|https://github.com/apache/hive/pull/426/files#diff-f2040374293a91cbcc6594ee571b20e4] > In the release version of Hive, outputs'TableType always be MANAGED_TABLE, no > matter the value of isExt is true of false. While I using post_execute_hook, > I found this problem because I cannot get correct tabletype in > HiveHookContext. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20662) Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
[
https://issues.apache.org/jira/browse/HIVE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634689#comment-16634689
]
Hive QA commented on HIVE-20662:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14164/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/util U: itests/util |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14164/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
>
>
> Key: HIVE-20662
> URL: https://issues.apache.org/jira/browse/HIVE-20662
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Blocker
> Attachments: HIVE-20662.patch
>
>
> TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] is more than flaky.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20664) Potential ArrayIndexOutOfBoundsException in VectorizedOrcAcidRowBatchReader.findMinMaxKeys
[
https://issues.apache.org/jira/browse/HIVE-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634676#comment-16634676
]
Hive QA commented on HIVE-20664:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941988/HIVE-20664.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15007 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14163/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14163/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14163/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941988 - PreCommit-HIVE-Build
> Potential ArrayIndexOutOfBoundsException in
> VectorizedOrcAcidRowBatchReader.findMinMaxKeys
> --
>
> Key: HIVE-20664
> URL: https://issues.apache.org/jira/browse/HIVE-20664
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Saurabh Seth
>Assignee: Saurabh Seth
>Priority: Minor
> Attachments: HIVE-20664.patch
>
>
> [~ekoifman], could you please confirm if my understanding is correct and if
> so, review the fix?
> In the method {{VectorizedOrcAcidRowBatchReader.findMinMaxKeys}}, the code
> snippet that identifies the first and last stripe indices in the current
> split could result in an ArrayIndexOutOfBoundsException if a complete split
> is within the same stripe:
> {noformat}
> for(int i = 0; i < stripes.size(); i++) {
> StripeInformation stripe = stripes.get(i);
> long stripeEnd = stripe.getOffset() + stripe.getLength();
> if(firstStripeIndex == -1 && stripe.getOffset() >= splitStart) {
> firstStripeIndex = i;
> }
> if(lastStripeIndex == -1 && splitEnd <= stripeEnd &&
> stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset() ) {
> //the last condition is for when both splitStart and splitEnd are in
> // the same stripe
> lastStripeIndex = i;
> }
> }
> {noformat}
> Consider the example where there are 2 stripes - 0-500 and 500-1000 and
> splitStart is 600 and splitEnd is 800.
> In the first iteration of the loop, stripe.getOffset() is 0 and stripeEnd is
> 500. In this iteration, neither of the if statement conditions will be met
> and firstSripeIndex as well as lastStripeIndex remain -1.
> In the second iteration of the loop stripe.getOffset() is 500, stripeEnd is
> 1000, The first if statement condition will not be met in this case because
> stripe's offset (500) is not greater than or equal to the splitStart (600).
> However, in the second if statement, splitEnd (800) is <= stripeEnd(1000) and
> it will try to compute the last condition
> {{stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset()}}. This
> will throw an ArrayIndexOutOfBoundsException because firstStripeIndex is
> still -1.
> I'm not sure if this scenario is possible at all, hence logging this as a low
> priority issue. Perhaps block based split generation using BISplitStrategy
> could trigger this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-12254) Improve logging with yarn/hdfs
[ https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-12254: Status: Patch Available (was: In Progress) Patch-4: address comments. After the query is done, now we are resetting callerContext back to sessionId. I'm updating caller context in beforeRun() and afterRun() for HS2 and update that separately for CLI in CliDriver.java. [~pvary] Can you take a look at the new patch? > Improve logging with yarn/hdfs > -- > > Key: HIVE-12254 > URL: https://issues.apache.org/jira/browse/HIVE-12254 > Project: Hive > Issue Type: Bug > Components: Shims >Affects Versions: 1.2.1 >Reporter: Vikram Dixit K >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch, > HIVE-12254.3.patch, HIVE-12254.4.patch > > > In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both > HIVE-12249 and HDFS-9184 are required (and upgraded in hive for the HDFS > issue) before this can be resolved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20648) LLAP: Vector group by operator should use memory per executor
[ https://issues.apache.org/jira/browse/HIVE-20648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634645#comment-16634645 ] Sergey Shelukhin commented on HIVE-20648: - +1 pending tests > LLAP: Vector group by operator should use memory per executor > - > > Key: HIVE-20648 > URL: https://issues.apache.org/jira/browse/HIVE-20648 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20648.1.patch, HIVE-20648.2.patch > > > HIVE-15503 treatment has to be applied for vector group by operator as well. > Vector group by currently uses MemoryMX bean to get heap usage and heap max > memory which will not work for LLAP. Instead it should use memory per > executor as upper bound to make flush decision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-12254) Improve logging with yarn/hdfs
[ https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-12254: Attachment: HIVE-12254.4.patch > Improve logging with yarn/hdfs > -- > > Key: HIVE-12254 > URL: https://issues.apache.org/jira/browse/HIVE-12254 > Project: Hive > Issue Type: Bug > Components: Shims >Affects Versions: 1.2.1 >Reporter: Vikram Dixit K >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch, > HIVE-12254.3.patch, HIVE-12254.4.patch > > > In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both > HIVE-12249 and HDFS-9184 are required (and upgraded in hive for the HDFS > issue) before this can be resolved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-12254) Improve logging with yarn/hdfs
[ https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-12254: Status: In Progress (was: Patch Available) > Improve logging with yarn/hdfs > -- > > Key: HIVE-12254 > URL: https://issues.apache.org/jira/browse/HIVE-12254 > Project: Hive > Issue Type: Bug > Components: Shims >Affects Versions: 1.2.1 >Reporter: Vikram Dixit K >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch, > HIVE-12254.3.patch, HIVE-12254.4.patch > > > In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both > HIVE-12249 and HDFS-9184 are required (and upgraded in hive for the HDFS > issue) before this can be resolved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20664) Potential ArrayIndexOutOfBoundsException in VectorizedOrcAcidRowBatchReader.findMinMaxKeys
[
https://issues.apache.org/jira/browse/HIVE-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634631#comment-16634631
]
Hive QA commented on HIVE-20664:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} ql: The patch generated 0 new + 56 unchanged - 1
fixed = 56 total (was 57) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14163/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14163/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14163/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Potential ArrayIndexOutOfBoundsException in
> VectorizedOrcAcidRowBatchReader.findMinMaxKeys
> --
>
> Key: HIVE-20664
> URL: https://issues.apache.org/jira/browse/HIVE-20664
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Saurabh Seth
>Assignee: Saurabh Seth
>Priority: Minor
> Attachments: HIVE-20664.patch
>
>
> [~ekoifman], could you please confirm if my understanding is correct and if
> so, review the fix?
> In the method {{VectorizedOrcAcidRowBatchReader.findMinMaxKeys}}, the code
> snippet that identifies the first and last stripe indices in the current
> split could result in an ArrayIndexOutOfBoundsException if a complete split
> is within the same stripe:
> {noformat}
> for(int i = 0; i < stripes.size(); i++) {
> StripeInformation stripe = stripes.get(i);
> long stripeEnd = stripe.getOffset() + stripe.getLength();
> if(firstStripeIndex == -1 && stripe.getOffset() >= splitStart) {
> firstStripeIndex = i;
> }
> if(lastStripeIndex == -1 && splitEnd <= stripeEnd &&
> stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset() ) {
> //the last condition is for when both splitStart and splitEnd are in
> // the same stripe
> lastStripeIndex = i;
> }
>
[jira] [Commented] (HIVE-20635) VectorizedOrcAcidRowBatchReader doesn't filter delete events for original files
[
https://issues.apache.org/jira/browse/HIVE-20635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634586#comment-16634586
]
Hive QA commented on HIVE-20635:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941986/HIVE-20635.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15009 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14162/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14162/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14162/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941986 - PreCommit-HIVE-Build
> VectorizedOrcAcidRowBatchReader doesn't filter delete events for original
> files
> ---
>
> Key: HIVE-20635
> URL: https://issues.apache.org/jira/browse/HIVE-20635
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20635.patch
>
>
> this is a followup to HIVE-16812 which adds support for delete event
> filtering for splits from native acid files
> need to add the same for {{OrcSplit.isOriginal()}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20575) Fix flaky connection metric tests
[ https://issues.apache.org/jira/browse/HIVE-20575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20575: - Attachment: HIVE-20575.10.patch > Fix flaky connection metric tests > - > > Key: HIVE-20575 > URL: https://issues.apache.org/jira/browse/HIVE-20575 > Project: Hive > Issue Type: Test > Components: Hive, Test >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-20575.01.patch, HIVE-20575.02.patch, > HIVE-20575.03.patch, HIVE-20575.04.patch, HIVE-20575.05.patch, > HIVE-20575.06.patch, HIVE-20575.07.patch, HIVE-20575.08.patch, > HIVE-20575.09.patch, HIVE-20575.10.patch > > > TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics() is flaky. We need to > fix it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20224) ReplChangeManager.java Remove Logging Guards
[
https://issues.apache.org/jira/browse/HIVE-20224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Morio Ramdenbourg reassigned HIVE-20224:
Assignee: Morio Ramdenbourg
> ReplChangeManager.java Remove Logging Guards
>
>
> Key: HIVE-20224
> URL: https://issues.apache.org/jira/browse/HIVE-20224
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Morio Ramdenbourg
>Priority: Trivial
> Labels: newb, newbie, noob
>
> {code:java|title=metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java}
> if (LOG.isDebugEnabled()) {
> LOG.debug("A file with the same content of {} already exists, ignore",
> path.toString());
> }
> // >
> LOG.debug("A file with the same content of {} already exists, ignore", path);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Encoded URI: " + encodedUri);
> }
> // >
> LOG.debug("Encoded URI: {}", encodedUri);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Move " + file.toString() + " to trash");
> }
> // >
> LOG.debug("Move {} to trash", file);
> ... others
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20224) ReplChangeManager.java Remove Logging Guards
[
https://issues.apache.org/jira/browse/HIVE-20224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Morio Ramdenbourg reassigned HIVE-20224:
Assignee: (was: Morio Ramdenbourg)
> ReplChangeManager.java Remove Logging Guards
>
>
> Key: HIVE-20224
> URL: https://issues.apache.org/jira/browse/HIVE-20224
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Priority: Trivial
> Labels: newb, newbie, noob
>
> {code:java|title=metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java}
> if (LOG.isDebugEnabled()) {
> LOG.debug("A file with the same content of {} already exists, ignore",
> path.toString());
> }
> // >
> LOG.debug("A file with the same content of {} already exists, ignore", path);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Encoded URI: " + encodedUri);
> }
> // >
> LOG.debug("Encoded URI: {}", encodedUri);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Move " + file.toString() + " to trash");
> }
> // >
> LOG.debug("Move {} to trash", file);
> ... others
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20365) Fix warnings when regenerating thrift code
[ https://issues.apache.org/jira/browse/HIVE-20365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-20365: --- Resolution: Not A Problem Status: Resolved (was: Patch Available) > Fix warnings when regenerating thrift code > -- > > Key: HIVE-20365 > URL: https://issues.apache.org/jira/browse/HIVE-20365 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20365.01.patch > > > When you build thrift code you can see thrift warning like below. > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2167] > No field key specified for rqst, resulting protocol may have conflicts or > not be backwards compatible! > [exec] > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2235] > No field key specified for o2, resulting protocol may have conflicts or not > be backwards compatible! > [exec] > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2167] > No field key specified for rqst, resulting protocol may have conflicts or > not be backwards compatible! > [exec] > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2235] > No field key specified for o2, resulting protocol may have conflicts or not > be backwards compatible! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20365) Fix warnings when regenerating thrift code
[ https://issues.apache.org/jira/browse/HIVE-20365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634552#comment-16634552 ] Vihang Karajgaonkar commented on HIVE-20365: Looks like HIVE-20293 already fixed this. > Fix warnings when regenerating thrift code > -- > > Key: HIVE-20365 > URL: https://issues.apache.org/jira/browse/HIVE-20365 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20365.01.patch > > > When you build thrift code you can see thrift warning like below. > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2167] > No field key specified for rqst, resulting protocol may have conflicts or > not be backwards compatible! > [exec] > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2235] > No field key specified for o2, resulting protocol may have conflicts or not > be backwards compatible! > [exec] > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2167] > No field key specified for rqst, resulting protocol may have conflicts or > not be backwards compatible! > [exec] > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2235] > No field key specified for o2, resulting protocol may have conflicts or not > be backwards compatible! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20648) LLAP: Vector group by operator should use memory per executor
[ https://issues.apache.org/jira/browse/HIVE-20648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20648: - Attachment: HIVE-20648.2.patch > LLAP: Vector group by operator should use memory per executor > - > > Key: HIVE-20648 > URL: https://issues.apache.org/jira/browse/HIVE-20648 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20648.1.patch, HIVE-20648.2.patch > > > HIVE-15503 treatment has to be applied for vector group by operator as well. > Vector group by currently uses MemoryMX bean to get heap usage and heap max > memory which will not work for LLAP. Instead it should use memory per > executor as upper bound to make flush decision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20648) LLAP: Vector group by operator should use memory per executor
[ https://issues.apache.org/jira/browse/HIVE-20648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634550#comment-16634550 ] Prasanth Jayachandran commented on HIVE-20648: -- .2 patch uses LlapProxy.isDaemon() > LLAP: Vector group by operator should use memory per executor > - > > Key: HIVE-20648 > URL: https://issues.apache.org/jira/browse/HIVE-20648 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20648.1.patch, HIVE-20648.2.patch > > > HIVE-15503 treatment has to be applied for vector group by operator as well. > Vector group by currently uses MemoryMX bean to get heap usage and heap max > memory which will not work for LLAP. Instead it should use memory per > executor as upper bound to make flush decision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15989) Incorrect rounding in decimal data types
[ https://issues.apache.org/jira/browse/HIVE-15989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benito Kestelman updated HIVE-15989: Description: I have a numeric field in a file in my data lake and created a hive external table pointing to this field. The field value is 0. but when I fetched this record using the query it display only 0.. I tried using DECIMAL and DOUBLE data types but nothing worked.is this a bug or am I not using the exact data type for this? Thanks, NIkesh was: I have a numeric field in a file in my data lake and created a hive external table pointing to this field. The field value is 0. but when I fetched this record using the query it display only 0.. I tried using DECIMAL and DOUBLE data types but nothing worked.is this a bug or am I not using the exact data type for this? Thanks, NIkesh > Incorrect rounding in decimal data types > > > Key: HIVE-15989 > URL: https://issues.apache.org/jira/browse/HIVE-15989 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Reporter: Nikesh >Priority: Critical > Attachments: ANA_AUTO_E.csv > > > I have a numeric field in a file in my data lake and created a hive external > table pointing to this field. The field value is > 0. but when I fetched this record > using the query it display only 0.. I tried using DECIMAL and > DOUBLE data types but nothing worked.is this a bug or am I not using the > exact data type for this? > Thanks, > NIkesh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20635) VectorizedOrcAcidRowBatchReader doesn't filter delete events for original files
[
https://issues.apache.org/jira/browse/HIVE-20635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634536#comment-16634536
]
Hive QA commented on HIVE-20635:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 7 new + 68 unchanged - 0 fixed
= 75 total (was 68) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14162/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14162/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14162/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> VectorizedOrcAcidRowBatchReader doesn't filter delete events for original
> files
> ---
>
> Key: HIVE-20635
> URL: https://issues.apache.org/jira/browse/HIVE-20635
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20635.patch
>
>
> this is a followup to HIVE-16812 which adds support for delete event
> filtering for splits from native acid files
> need to add the same for {{OrcSplit.isOriginal()}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20575) Fix flaky connection metric tests
[
https://issues.apache.org/jira/browse/HIVE-20575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634504#comment-16634504
]
Hive QA commented on HIVE-20575:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941984/HIVE-20575.09.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15008 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14161/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14161/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14161/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941984 - PreCommit-HIVE-Build
> Fix flaky connection metric tests
> -
>
> Key: HIVE-20575
> URL: https://issues.apache.org/jira/browse/HIVE-20575
> Project: Hive
> Issue Type: Test
> Components: Hive, Test
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20575.01.patch, HIVE-20575.02.patch,
> HIVE-20575.03.patch, HIVE-20575.04.patch, HIVE-20575.05.patch,
> HIVE-20575.06.patch, HIVE-20575.07.patch, HIVE-20575.08.patch,
> HIVE-20575.09.patch
>
>
> TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics() is flaky. We need to
> fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20563:
Status: In Progress (was: Patch Available)
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
> cstring1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:149)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:136)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20563:
Status: Patch Available (was: In Progress)
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
> cstring1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:149)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:136)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20563:
Attachment: HIVE-20563.04.patch
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
> cstring1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:149)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:136)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Comment Edited] (HIVE-20664) Potential ArrayIndexOutOfBoundsException in VectorizedOrcAcidRowBatchReader.findMinMaxKeys
[
https://issues.apache.org/jira/browse/HIVE-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634482#comment-16634482
]
Eugene Koifman edited comment on HIVE-20664 at 10/1/18 6:58 PM:
I agree that this can produce {{ArrayIndexOutOfBounds}}.
In your example, the ORC {{Reader}} should return no rows. It should always
read whole stripes.
So I think you patch will cause firstStripeIndex = -1 and lastStripIndex = 1
did you mean {{if (firstStripeIndex == -1 && lastStripeIndex != -1 ||
firstStripeIndex > lastStripeIndex) {}}
to be {{if (firstStripeIndex == -1 && lastStripeIndex != -1 || lastStripeIndex
> firstStripeIndex) {}}
Also, I think you need to log something like
LOG.info("findMinMaxKeys(): " + keyInterval +
" stripes(" + firstStripeIndex + "," + lastStripeIndex + ")");
before returning a KeyInterval that will actually cause some Delete events to
be skipped.
was (Author: ekoifman):
I agree that this can produce {{ArrayIndexOutOfBounds}}.
In your example, the ORC {{Reader}} should return no rows. It should always
read whole stripes.
So I think you patch will cause firstStripeIndex = -1 and lastStripIndex = 1
did you mean {{if (firstStripeIndex == -1 && lastStripeIndex != -1 ||
firstStripeIndex > lastStripeIndex) {}}
to be {{if (firstStripeIndex == -1 && lastStripeIndex != -1 lastStripeIndex ||
> firstStripeIndex) {}}
Also, I think you need to log something like
LOG.info("findMinMaxKeys(): " + keyInterval +
" stripes(" + firstStripeIndex + "," + lastStripeIndex + ")");
before returning a KeyInterval that will actually cause some Delete events to
be skipped.
> Potential ArrayIndexOutOfBoundsException in
> VectorizedOrcAcidRowBatchReader.findMinMaxKeys
> --
>
> Key: HIVE-20664
> URL: https://issues.apache.org/jira/browse/HIVE-20664
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Saurabh Seth
>Assignee: Saurabh Seth
>Priority: Minor
> Attachments: HIVE-20664.patch
>
>
> [~ekoifman], could you please confirm if my understanding is correct and if
> so, review the fix?
> In the method {{VectorizedOrcAcidRowBatchReader.findMinMaxKeys}}, the code
> snippet that identifies the first and last stripe indices in the current
> split could result in an ArrayIndexOutOfBoundsException if a complete split
> is within the same stripe:
> {noformat}
> for(int i = 0; i < stripes.size(); i++) {
> StripeInformation stripe = stripes.get(i);
> long stripeEnd = stripe.getOffset() + stripe.getLength();
> if(firstStripeIndex == -1 && stripe.getOffset() >= splitStart) {
> firstStripeIndex = i;
> }
> if(lastStripeIndex == -1 && splitEnd <= stripeEnd &&
> stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset() ) {
> //the last condition is for when both splitStart and splitEnd are in
> // the same stripe
> lastStripeIndex = i;
> }
> }
> {noformat}
> Consider the example where there are 2 stripes - 0-500 and 500-1000 and
> splitStart is 600 and splitEnd is 800.
> In the first iteration of the loop, stripe.getOffset() is 0 and stripeEnd is
> 500. In this iteration, neither of the if statement conditions will be met
> and firstSripeIndex as well as lastStripeIndex remain -1.
> In the second iteration of the loop stripe.getOffset() is 500, stripeEnd is
> 1000, The first if statement condition will not be met in this case because
> stripe's offset (500) is not greater than or equal to the splitStart (600).
> However, in the second if statement, splitEnd (800) is <= stripeEnd(1000) and
> it will try to compute the last condition
> {{stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset()}}. This
> will throw an ArrayIndexOutOfBoundsException because firstStripeIndex is
> still -1.
> I'm not sure if this scenario is possible at all, hence logging this as a low
> priority issue. Perhaps block based split generation using BISplitStrategy
> could trigger this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20664) Potential ArrayIndexOutOfBoundsException in VectorizedOrcAcidRowBatchReader.findMinMaxKeys
[
https://issues.apache.org/jira/browse/HIVE-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634482#comment-16634482
]
Eugene Koifman commented on HIVE-20664:
---
I agree that this can produce {{ArrayIndexOutOfBounds}}.
In your example, the ORC {{Reader}} should return no rows. It should always
read whole stripes.
So I think you patch will cause firstStripeIndex = -1 and lastStripIndex = 1
did you mean {{if (firstStripeIndex == -1 && lastStripeIndex != -1 ||
firstStripeIndex > lastStripeIndex) {}}
to be {{if (firstStripeIndex == -1 && lastStripeIndex != -1 lastStripeIndex ||
> firstStripeIndex) {}}
Also, I think you need to log something like
LOG.info("findMinMaxKeys(): " + keyInterval +
" stripes(" + firstStripeIndex + "," + lastStripeIndex + ")");
before returning a KeyInterval that will actually cause some Delete events to
be skipped.
> Potential ArrayIndexOutOfBoundsException in
> VectorizedOrcAcidRowBatchReader.findMinMaxKeys
> --
>
> Key: HIVE-20664
> URL: https://issues.apache.org/jira/browse/HIVE-20664
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Saurabh Seth
>Assignee: Saurabh Seth
>Priority: Minor
> Attachments: HIVE-20664.patch
>
>
> [~ekoifman], could you please confirm if my understanding is correct and if
> so, review the fix?
> In the method {{VectorizedOrcAcidRowBatchReader.findMinMaxKeys}}, the code
> snippet that identifies the first and last stripe indices in the current
> split could result in an ArrayIndexOutOfBoundsException if a complete split
> is within the same stripe:
> {noformat}
> for(int i = 0; i < stripes.size(); i++) {
> StripeInformation stripe = stripes.get(i);
> long stripeEnd = stripe.getOffset() + stripe.getLength();
> if(firstStripeIndex == -1 && stripe.getOffset() >= splitStart) {
> firstStripeIndex = i;
> }
> if(lastStripeIndex == -1 && splitEnd <= stripeEnd &&
> stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset() ) {
> //the last condition is for when both splitStart and splitEnd are in
> // the same stripe
> lastStripeIndex = i;
> }
> }
> {noformat}
> Consider the example where there are 2 stripes - 0-500 and 500-1000 and
> splitStart is 600 and splitEnd is 800.
> In the first iteration of the loop, stripe.getOffset() is 0 and stripeEnd is
> 500. In this iteration, neither of the if statement conditions will be met
> and firstSripeIndex as well as lastStripeIndex remain -1.
> In the second iteration of the loop stripe.getOffset() is 500, stripeEnd is
> 1000, The first if statement condition will not be met in this case because
> stripe's offset (500) is not greater than or equal to the splitStart (600).
> However, in the second if statement, splitEnd (800) is <= stripeEnd(1000) and
> it will try to compute the last condition
> {{stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset()}}. This
> will throw an ArrayIndexOutOfBoundsException because firstStripeIndex is
> still -1.
> I'm not sure if this scenario is possible at all, hence logging this as a low
> priority issue. Perhaps block based split generation using BISplitStrategy
> could trigger this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-18945) Support "analyze table T"
[
https://issues.apache.org/jira/browse/HIVE-18945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-18945 started by Laszlo Bodor.
---
> Support "analyze table T"
> -
>
> Key: HIVE-18945
> URL: https://issues.apache.org/jira/browse/HIVE-18945
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
>
> I think it would be good to have it behave the same as
> {code}
> analyze table T compute statistics for columns
> {code}
> this could help people who not yet know the different analyze commands ; to
> run the probably most appropriate one
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1
[ https://issues.apache.org/jira/browse/HIVE-20661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20661: - Status: Patch Available (was: Open) > Dynamic partitions loading calls add partition for every partition 1-by-1 > - > > Key: HIVE-20661 > URL: https://issues.apache.org/jira/browse/HIVE-20661 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20661.01.patch > > > Hive.loadDynamicPartitions creates partitions using a threadpool, but the > update of the Metastore via the MetastoreClient is done 1-by-1. This adds > unnecessary extra calls. The partitions should be created in one batch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20575) Fix flaky connection metric tests
[
https://issues.apache.org/jira/browse/HIVE-20575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634447#comment-16634447
]
Hive QA commented on HIVE-20575:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14161/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/hive-unit U: itests/hive-unit |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14161/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix flaky connection metric tests
> -
>
> Key: HIVE-20575
> URL: https://issues.apache.org/jira/browse/HIVE-20575
> Project: Hive
> Issue Type: Test
> Components: Hive, Test
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20575.01.patch, HIVE-20575.02.patch,
> HIVE-20575.03.patch, HIVE-20575.04.patch, HIVE-20575.05.patch,
> HIVE-20575.06.patch, HIVE-20575.07.patch, HIVE-20575.08.patch,
> HIVE-20575.09.patch
>
>
> TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics() is flaky. We need to
> fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20535) Add new configuration to set the size of the global compile lock
[ https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634445#comment-16634445 ] Laszlo Bodor commented on HIVE-20535: - load_dyn_part3 failure is not related (HIVE-20662) > Add new configuration to set the size of the global compile lock > > > Key: HIVE-20535 > URL: https://issues.apache.org/jira/browse/HIVE-20535 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20535.1.patch, HIVE-20535.10.patch, > HIVE-20535.11.patch, HIVE-20535.12.patch, HIVE-20535.13.patch, > HIVE-20535.14.patch, HIVE-20535.15.patch, HIVE-20535.16.patch, > HIVE-20535.17.patch, HIVE-20535.18.patch, HIVE-20535.19.patch, > HIVE-20535.2.patch, HIVE-20535.20.patch, HIVE-20535.21.patch, > HIVE-20535.3.patch, HIVE-20535.4.patch, HIVE-20535.5.patch, > HIVE-20535.6.patch, HIVE-20535.8.patch, HIVE-20535.9.patch > > > When removing the compile lock, it is quite risky to remove it entirely. > It would be good to provide a pool size for the concurrent compilation, so > the administrator can limit the load -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20663) Enable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
[ https://issues.apache.org/jira/browse/HIVE-20663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634438#comment-16634438 ] Zoltan Haindrich commented on HIVE-20663: - this is most probably the same as HIVE-20585 the symptom occurs in case of a specific set of metastore call sequence > Enable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] > --- > > Key: HIVE-20663 > URL: https://issues.apache.org/jira/browse/HIVE-20663 > Project: Hive > Issue Type: Test >Reporter: Karen Coppage >Assignee: Zoltan Haindrich >Priority: Major > > results/clientpositive/llap/load_dyn_part3.q.out currently requires > Column stats: COMPLETE > but test outputs > Column stats: PARTIAL -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20663) Enable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
[ https://issues.apache.org/jira/browse/HIVE-20663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-20663: --- Assignee: Zoltan Haindrich > Enable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] > --- > > Key: HIVE-20663 > URL: https://issues.apache.org/jira/browse/HIVE-20663 > Project: Hive > Issue Type: Test >Reporter: Karen Coppage >Assignee: Zoltan Haindrich >Priority: Major > > results/clientpositive/llap/load_dyn_part3.q.out currently requires > Column stats: COMPLETE > but test outputs > Column stats: PARTIAL -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20662) Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
[ https://issues.apache.org/jira/browse/HIVE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634436#comment-16634436 ] Zoltan Haindrich commented on HIVE-20662: - +1 I think this is the same issue which I've seen during enabling col stat fetch (HIVE-17084) > Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] > > > Key: HIVE-20662 > URL: https://issues.apache.org/jira/browse/HIVE-20662 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Blocker > Attachments: HIVE-20662.patch > > > TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] is more than flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904
[ https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634428#comment-16634428 ] Laszlo Bodor commented on HIVE-20638: - load_dyn_part3 failure is not related, it's tracked on HIVE-20662 [~nishantbangarwa] could you check the patch please? > Upgrade version of Jetty to 9.3.25.v20180904 > > > Key: HIVE-20638 > URL: https://issues.apache.org/jira/browse/HIVE-20638 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch, > HIVE-20638.03.patch > > > Current version is 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18908) FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634421#comment-16634421 ] Matt McCline commented on HIVE-18908: - Committed to master. > FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, > HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, > HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch, > HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch, > HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch, > HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch, > HIVE-18908.09994.patch, HIVE-18908.09995.patch, HIVE-18908.09996.patch, > HIVE-18908.09997.patch, HIVE-18908.09998.patch, JOIN to MAPJOIN > Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. > Rough TPC-DS timings run on laptop: > (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play) > FULL OUTER MapJoin OFF = MergeJoin > Query 51: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 4:30 minutes > • FULL OUTER MapJoin ON: 4:37 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 2:35 minutes > • FULL OUTER MapJoin ON: 1:47 minutes > Query 97: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 2:37 minutes > • FULL OUTER MapJoin ON: 2:42 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 1:17 minutes > • FULL OUTER MapJoin ON: 0:06 minutes > FULL OUTER Join 10,000,000 rows against 323,910 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 14:56 minutes > • FULL OUTER MapJoin ON: 1:45 minutes > FULL OUTER Join 10,000,000 rows against 1,000 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 12:37 minutes > • FULL OUTER MapJoin ON: 1:38 minutes > Hopefully, someone will do large scale cluster testing. > [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than > [Sort] MergeJoin reduce-shuffle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18908) FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634420#comment-16634420
]
Matt McCline commented on HIVE-18908:
-
{noformat}
commit a37827ecd557c7f7d69f3b2ccdbf6535908b1461
Author: Matt McCline
Date: Sun Sep 16 09:04:59 2018 -0500
HIVE-18908: FULL OUTER JOIN to MapJoin (Matt McCline, reviewed by Teddy
Choi)
{noformat}
> FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: FULL OUTER MapJoin Code Changes.docx,
> HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch,
> HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch,
> HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch,
> HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch,
> HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch,
> HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch,
> HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch,
> HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch,
> HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch,
> HIVE-18908.09994.patch, HIVE-18908.09995.patch, HIVE-18908.09996.patch,
> HIVE-18908.09997.patch, HIVE-18908.09998.patch, JOIN to MAPJOIN
> Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf
>
>
> Currently, we do not support FULL OUTER JOIN in MapJoin.
> Rough TPC-DS timings run on laptop:
> (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play)
> FULL OUTER MapJoin OFF = MergeJoin
> Query 51:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 4:30 minutes
> • FULL OUTER MapJoin ON: 4:37 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 2:35 minutes
> • FULL OUTER MapJoin ON: 1:47 minutes
> Query 97:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 2:37 minutes
> • FULL OUTER MapJoin ON: 2:42 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 1:17 minutes
> • FULL OUTER MapJoin ON: 0:06 minutes
> FULL OUTER Join 10,000,000 rows against 323,910 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 14:56 minutes
> • FULL OUTER MapJoin ON: 1:45 minutes
> FULL OUTER Join 10,000,000 rows against 1,000 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 12:37 minutes
> • FULL OUTER MapJoin ON: 1:38 minutes
> Hopefully, someone will do large scale cluster testing.
> [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than
> [Sort] MergeJoin reduce-shuffle.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Resolution: Fixed Status: Resolved (was: Patch Available) > FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, > HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, > HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch, > HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch, > HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch, > HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch, > HIVE-18908.09994.patch, HIVE-18908.09995.patch, HIVE-18908.09996.patch, > HIVE-18908.09997.patch, HIVE-18908.09998.patch, JOIN to MAPJOIN > Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. > Rough TPC-DS timings run on laptop: > (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play) > FULL OUTER MapJoin OFF = MergeJoin > Query 51: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 4:30 minutes > • FULL OUTER MapJoin ON: 4:37 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 2:35 minutes > • FULL OUTER MapJoin ON: 1:47 minutes > Query 97: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 2:37 minutes > • FULL OUTER MapJoin ON: 2:42 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 1:17 minutes > • FULL OUTER MapJoin ON: 0:06 minutes > FULL OUTER Join 10,000,000 rows against 323,910 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 14:56 minutes > • FULL OUTER MapJoin ON: 1:45 minutes > FULL OUTER Join 10,000,000 rows against 1,000 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 12:37 minutes > • FULL OUTER MapJoin ON: 1:38 minutes > Hopefully, someone will do large scale cluster testing. > [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than > [Sort] MergeJoin reduce-shuffle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[
https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634414#comment-16634414
]
Hive QA commented on HIVE-20647:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941973/HIVE-20647.5.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15007 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=168)
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsBinary.testOpenConnectionMetrics
(batchId=256)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14160/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14160/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14160/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941973 - PreCommit-HIVE-Build
> HadoopVer was ignored in QTestUtil
> --
>
> Key: HIVE-20647
> URL: https://issues.apache.org/jira/browse/HIVE-20647
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: denys kuzmenko
>Assignee: denys kuzmenko
>Priority: Major
> Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch,
> HIVE-20647.3.patch, HIVE-20647.4.patch, HIVE-20647.5.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[
https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634354#comment-16634354
]
Hive QA commented on HIVE-20647:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/util: The patch generated 3 new + 91 unchanged
- 16 fixed = 94 total (was 107) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
54s{color} | {color:red} itests/util generated 1 new + 49 unchanged - 3 fixed =
50 total (was 52) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:itests/util |
| | setup isn't initialized in
org.apache.hadoop.hive.accumulo.AccumuloQTestUtil.initConf() when invoked from
constructor for superclass At AccumuloQTestUtil.java:when invoked from
constructor for superclass At AccumuloQTestUtil.java:[line 52] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14160/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14160/yetus/diff-checkstyle-itests_util.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14160/yetus/new-findbugs-itests_util.html
|
| modules | C: itests/util itests/hive-unit U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14160/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HadoopVer was ignored in QTestUtil
> --
>
> Key: HIVE-20647
> URL: https://issues.apache.org/jira/browse/HIVE-20647
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: denys kuzmenko
>Assignee: denys kuzmenko
>Priority: Major
> Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch,
> HIVE-20647.3.patch, HIVE-20647.4.patch, HIVE-20647.5.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20657) pre-allocate LLAP cache at init time
[ https://issues.apache.org/jira/browse/HIVE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634352#comment-16634352 ] Sergey Shelukhin commented on HIVE-20657: - Updated for SSD cache based on a comment from [~t3rmin4t0r] > pre-allocate LLAP cache at init time > > > Key: HIVE-20657 > URL: https://issues.apache.org/jira/browse/HIVE-20657 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20657.01.patch, HIVE-20657.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20657) pre-allocate LLAP cache at init time
[ https://issues.apache.org/jira/browse/HIVE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20657: Attachment: HIVE-20657.01.patch > pre-allocate LLAP cache at init time > > > Key: HIVE-20657 > URL: https://issues.apache.org/jira/browse/HIVE-20657 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20657.01.patch, HIVE-20657.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20648) LLAP: Vector group by operator should use memory per executor
[ https://issues.apache.org/jira/browse/HIVE-20648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634338#comment-16634338 ] Sergey Shelukhin commented on HIVE-20648: - LlapProxy.isDaemon() should be used for LLAP detection > LLAP: Vector group by operator should use memory per executor > - > > Key: HIVE-20648 > URL: https://issues.apache.org/jira/browse/HIVE-20648 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20648.1.patch > > > HIVE-15503 treatment has to be applied for vector group by operator as well. > Vector group by currently uses MemoryMX bean to get heap usage and heap max > memory which will not work for LLAP. Instead it should use memory per > executor as upper bound to make flush decision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12254) Improve logging with yarn/hdfs
[ https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634334#comment-16634334 ] Aihua Xu commented on HIVE-12254: - [~pvary] Thanks for reviewing the patch. * Such CallerContext is a thread local variable defined in Hadoop, since different queries are compiled or executed in different threads, different context will be carried for different queries. * In the previous version, we are only setting the context around query compilation or execution (only certain operations) that might cause to print incorrect info if we don't reset. In this patch, I'm setting the context whenever there is a new session or new query and such info should be carried for the whole lifetime. But you are right that after we are done with the queries, there are some actions belonging to the session rather than the queries (the ones in closeSession). I will investigate that. * You are right. I should remove such ":". I will correct it. > Improve logging with yarn/hdfs > -- > > Key: HIVE-12254 > URL: https://issues.apache.org/jira/browse/HIVE-12254 > Project: Hive > Issue Type: Bug > Components: Shims >Affects Versions: 1.2.1 >Reporter: Vikram Dixit K >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch, > HIVE-12254.3.patch > > > In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both > HIVE-12249 and HDFS-9184 are required (and upgraded in hive for the HDFS > issue) before this can be resolved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20535) Add new configuration to set the size of the global compile lock
[
https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634314#comment-16634314
]
Hive QA commented on HIVE-20535:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941960/HIVE-20535.21.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15007 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
org.apache.hive.service.server.TestInformationSchemaWithPrivilege.test
(batchId=250)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14159/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14159/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14159/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941960 - PreCommit-HIVE-Build
> Add new configuration to set the size of the global compile lock
>
>
> Key: HIVE-20535
> URL: https://issues.apache.org/jira/browse/HIVE-20535
> Project: Hive
> Issue Type: Task
> Components: HiveServer2
>Reporter: denys kuzmenko
>Assignee: denys kuzmenko
>Priority: Major
> Attachments: HIVE-20535.1.patch, HIVE-20535.10.patch,
> HIVE-20535.11.patch, HIVE-20535.12.patch, HIVE-20535.13.patch,
> HIVE-20535.14.patch, HIVE-20535.15.patch, HIVE-20535.16.patch,
> HIVE-20535.17.patch, HIVE-20535.18.patch, HIVE-20535.19.patch,
> HIVE-20535.2.patch, HIVE-20535.20.patch, HIVE-20535.21.patch,
> HIVE-20535.3.patch, HIVE-20535.4.patch, HIVE-20535.5.patch,
> HIVE-20535.6.patch, HIVE-20535.8.patch, HIVE-20535.9.patch
>
>
> When removing the compile lock, it is quite risky to remove it entirely.
> It would be good to provide a pool size for the concurrent compilation, so
> the administrator can limit the load
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20666) HiveServer2 Interactive LLAP reconnect to already running Yarn app
[
https://issues.apache.org/jira/browse/HIVE-20666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-20666:
---
Description:
Improve HiveServer2 Interactive LLAP to reconnect to already running hive llap
yarn app.
Currently HiveServer2 Interactive startup may fail with the following error if
it cannot get enough containers on the queue:
{code:java}
WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
state RUNNING is attained.
2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
Desired Instance : '4'
2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
2018-10-01 16:26:55,625 - Stopping LLAP
2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
True, 'user': 'hive', 'stderr': -1}{code}
I could meanwhile see 5 containers for a previous hive llap invocation in the
yarn scheduler page and this is the only HiveServer2 Interactive instance, so
it appears it wasn't reconnecting and making use of the running llap app.
was:
Improve HiveServer2 Interactive LLAP to reconnect to already running hive llap
yarn app.
Currently HiveServer2 Interactive startup may fail with the following error if
it cannot get enough containers on the queue:
{code:java}
WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
state RUNNING is attained.
2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
Desired Instance : '4'
2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
2018-10-01 16:26:55,625 - Stopping LLAP
2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
True, 'user': 'hive', 'stderr': -1}{code}
> HiveServer2 Interactive LLAP reconnect to already running Yarn app
> --
>
> Key: HIVE-20666
> URL: https://issues.apache.org/jira/browse/HIVE-20666
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, llap
>Affects Versions: 1.2.1
>Reporter: Hari Sekhon
>Priority: Major
>
> Improve HiveServer2 Interactive LLAP to reconnect to already running hive
> llap yarn app.
> Currently HiveServer2 Interactive startup may fail with the following error
> if it cannot get enough containers on the queue:
> {code:java}
> WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
> state RUNNING is attained.
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
> Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
> 2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
> Desired Instance : '4'
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
> 2018-10-01 16:26:55,625 - Stopping LLAP
> 2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
> True, 'user': 'hive', 'stderr': -1}{code}
> I could meanwhile see 5 containers for a previous hive llap invocation in the
> yarn scheduler page and this is the only HiveServer2 Interactive instance, so
> it appears it wasn't reconnecting and making use of the running llap app.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20623) Shared work: Extend sharing of map-join cache entries in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20623:
---
Attachment: HIVE-20623.04.patch
> Shared work: Extend sharing of map-join cache entries in LLAP
> -
>
> Key: HIVE-20623
> URL: https://issues.apache.org/jira/browse/HIVE-20623
> Project: Hive
> Issue Type: Improvement
> Components: llap, Logical Optimizer
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20623.01.patch, HIVE-20623.02.patch,
> HIVE-20623.02.patch, HIVE-20623.02.patch, HIVE-20623.03.patch,
> HIVE-20623.03.patch, HIVE-20623.03.patch, HIVE-20623.04.patch,
> HIVE-20623.patch, hash-shared-work.json.txt, hash-shared-work.svg
>
>
> For a query like this
> {code}
> with all_sales as (
> select ss_customer_sk as customer_sk, ss_ext_list_price-ss_ext_discount_amt
> as ext_price from store_sales
> UNION ALL
> select ws_bill_customer_sk as customer_sk,
> ws_ext_list_price-ws_ext_discount_amt as ext_price from web_sales
> UNION ALL
> select cs_bill_customer_sk as customer_sk, cs_ext_sales_price -
> cs_ext_discount_amt as ext_price from catalog_sales)
> select sum(ext_price) total_price, c_customer_id from all_sales, customer
> where customer_sk = c_customer_sk
> group by c_customer_id
> order by total_price desc
> limit 100;
> {code}
> The hashtable used for all 3 joins are identical, which is loaded 3x times in
> the same LLAP instance because they are named.
> {code}
> cacheKey = "HASH_MAP_" + this.getOperatorId() + "_container";
> {code}
> in the cache.
> If those are identical in nature (i.e vectorization, hashtable type etc),
> then the duplication is just wasted CPU, memory and network - using the cache
> name for hashtables which will be identical in layout would be extremely
> useful.
> In cases where the join is pushed through a UNION, those are identical.
> This optimization can only be done without concern for accidental delays when
> the same upstream task is generating all of these hashtables, which is what
> is achieved by the shared scan optimizer already.
> In case the shared work is not present, this has potential downsides - in
> case two customer broadcasts were sourced from "Map 1" and "Map 2", the Map 1
> builder will block the other task from reading from Map 2, even though Map 2
> might have started after, but finished ahead of Map 1.
> So this specific optimization can always be considered for cases where the
> shared work unifies the operator tree and the parents of all the RS entries
> involved are same (& the RS layout is the same).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20665:
---
Description:
When parallel tasks are enabled in Hive, all of the resulting queries share the
same Hive configuration. This is problematic as each query will modify the
same {{HiveConf}} object with things like query ID and query text. This will
overwrite each other and cause {{ConcurrentModificationException}} issues.
{code:java|title=SQLOperation.java}
public Object run() throws HiveSQLException {
Hive.set(parentHive, false);
// TODO: can this result in cross-thread reuse of session state?
SessionState.setCurrentSessionState(parentSessionState);
PerfLogger.setPerfLogger(SessionState.getPerfLogger());
LogUtils.registerLoggingContext(queryState.getConf());
try {
if (asyncPrepare) {
prepare(queryState);
}
runQuery();
} catch (HiveSQLException e) {
// ...
{code}
[Code
Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
>From this code it can be seen that for every thread launched, it is all
>calling {{setCurrentSessionState}}.
{code:java|title=SessionStates.java}
/**
* Sets the given session state in the thread local var for sessions.
*/
public static void setCurrentSessionState(SessionState startSs) {
tss.get().attach(startSs);
}
// SessionState is not available in runtime and Hive.get().getConf() is not
safe to call
private static class SessionStates {
private SessionState state;
private HiveConf conf;
private void attach(SessionState state) {
this.state = state;
attach(state.getConf());
}
private void attach(HiveConf conf) {
// -- SHALLOW COPY HERE, ALL THREADS SHARING SAME REFERENCE -- //
this.conf = conf;
ClassLoader classLoader = conf.getClassLoader();
if (classLoader != null) {
Thread.currentThread().setContextClassLoader(classLoader);
}
}
}
{code}
[Code
Here|https://github.com/apache/hive/blob/7795c0a7dc59941671f8845d78b16d9e5ddc9ea3/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L540-L556]
Ensure that all threads get their own copy of the {{HiveConf}} object to use
and modify.
was:
When parallel tasks are enabled in Hive, all of the resulting queries share the
same Hive configuration. This is problematic as each query will modify the
same {{HiveConf}} object with things like query ID and query text. This will
overwrite each other and cause {{ConcurrentModificationException}} issues.
{code:java|title=SQLOperation.java}
public Object run() throws HiveSQLException {
Hive.set(parentHive, false);
// TODO: can this result in cross-thread reuse of session state?
SessionState.setCurrentSessionState(parentSessionState);
PerfLogger.setPerfLogger(SessionState.getPerfLogger());
LogUtils.registerLoggingContext(queryState.getConf());
try {
if (asyncPrepare) {
prepare(queryState);
}
runQuery();
} catch (HiveSQLException e) {
// ...
{code}
[Code
Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
>From this code we can see that for every thread launched, they are all calling
>{{setCurrentSessionState}}.
{code:java|title=SessionStates.java}
/**
* Sets the given session state in the thread local var for sessions.
*/
public static void setCurrentSessionState(SessionState startSs) {
tss.get().attach(startSs);
}
// SessionState is not available in runtime and Hive.get().getConf() is not
safe to call
private static class SessionStates {
private SessionState state;
private HiveConf conf;
private void attach(SessionState state) {
this.state = state;
attach(state.getConf());
}
private void attach(HiveConf conf) {
// -- SHALLOW COPY HERE, ALL THREADS SHARING SAME REFERENCE -- //
this.conf = conf;
ClassLoader classLoader = conf.getClassLoader();
if (classLoader != null) {
Thread.currentThread().setContextClassLoader(classLoader);
}
}
}
{code}
[Code
Here|https://github.com/apache/hive/blob/7795c0a7dc59941671f8845d78b16d9e5ddc9ea3/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L540-L556]
Ensure that all threads get their own copy of the {{HiveConf}} object to use
and modify.
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL:
[jira] [Updated] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20665:
---
Attachment: HIVE-20665.1.patch
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL: https://issues.apache.org/jira/browse/HIVE-20665
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-20665.1.patch
>
>
> When parallel tasks are enabled in Hive, all of the resulting queries share
> the same Hive configuration. This is problematic as each query will modify
> the same {{HiveConf}} object with things like query ID and query text. This
> will overwrite each other and cause {{ConcurrentModificationException}}
> issues.
> {code:java|title=SQLOperation.java}
> public Object run() throws HiveSQLException {
> Hive.set(parentHive, false);
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(SessionState.getPerfLogger());
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // ...
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
> From this code we can see that for every thread launched, they are all
> calling {{setCurrentSessionState}}.
> {code:java|title=SessionStates.java}
> /**
>* Sets the given session state in the thread local var for sessions.
>*/
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> // SessionState is not available in runtime and Hive.get().getConf() is not
> safe to call
> private static class SessionStates {
> private SessionState state;
> private HiveConf conf;
> private void attach(SessionState state) {
> this.state = state;
> attach(state.getConf());
> }
> private void attach(HiveConf conf) {
> // -- SHALLOW COPY HERE, ALL THREADS SHARING SAME REFERENCE -- //
> this.conf = conf;
> ClassLoader classLoader = conf.getClassLoader();
> if (classLoader != null) {
> Thread.currentThread().setContextClassLoader(classLoader);
> }
> }
> }
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/7795c0a7dc59941671f8845d78b16d9e5ddc9ea3/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L540-L556]
> Ensure that all threads get their own copy of the {{HiveConf}} object to use
> and modify.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20665:
---
Status: Patch Available (was: Open)
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL: https://issues.apache.org/jira/browse/HIVE-20665
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 2.3.2, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-20665.1.patch
>
>
> When parallel tasks are enabled in Hive, all of the resulting queries share
> the same Hive configuration. This is problematic as each query will modify
> the same {{HiveConf}} object with things like query ID and query text. This
> will overwrite each other and cause {{ConcurrentModificationException}}
> issues.
> {code:java|title=SQLOperation.java}
> public Object run() throws HiveSQLException {
> Hive.set(parentHive, false);
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(SessionState.getPerfLogger());
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // ...
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
> From this code we can see that for every thread launched, they are all
> calling {{setCurrentSessionState}}.
> {code:java|title=SessionStates.java}
> /**
>* Sets the given session state in the thread local var for sessions.
>*/
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> // SessionState is not available in runtime and Hive.get().getConf() is not
> safe to call
> private static class SessionStates {
> private SessionState state;
> private HiveConf conf;
> private void attach(SessionState state) {
> this.state = state;
> attach(state.getConf());
> }
> private void attach(HiveConf conf) {
> // -- SHALLOW COPY HERE, ALL THREADS SHARING SAME REFERENCE -- //
> this.conf = conf;
> ClassLoader classLoader = conf.getClassLoader();
> if (classLoader != null) {
> Thread.currentThread().setContextClassLoader(classLoader);
> }
> }
> }
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/7795c0a7dc59941671f8845d78b16d9e5ddc9ea3/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L540-L556]
> Ensure that all threads get their own copy of the {{HiveConf}} object to use
> and modify.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-20665:
--
Assignee: BELUGA BEHR
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL: https://issues.apache.org/jira/browse/HIVE-20665
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
>
> When parallel tasks are enabled in Hive, all of the resulting queries share
> the same Hive configuration. This is problematic as each query will modify
> the same {{HiveConf}} object with things like query ID and query text. This
> will overwrite each other and cause {{ConcurrentModificationException}}
> issues.
> {code:java|title=SQLOperation.java}
> public Object run() throws HiveSQLException {
> Hive.set(parentHive, false);
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(SessionState.getPerfLogger());
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // ...
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
> From this code we can see that for every thread launched, they are all
> calling {{setCurrentSessionState}}.
> {code:java|title=SessionStates.java}
> /**
>* Sets the given session state in the thread local var for sessions.
>*/
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> // SessionState is not available in runtime and Hive.get().getConf() is not
> safe to call
> private static class SessionStates {
> private SessionState state;
> private HiveConf conf;
> private void attach(SessionState state) {
> this.state = state;
> attach(state.getConf());
> }
> private void attach(HiveConf conf) {
> // -- SHALLOW COPY HERE, ALL THREADS SHARING SAME REFERENCE -- //
> this.conf = conf;
> ClassLoader classLoader = conf.getClassLoader();
> if (classLoader != null) {
> Thread.currentThread().setContextClassLoader(classLoader);
> }
> }
> }
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/7795c0a7dc59941671f8845d78b16d9e5ddc9ea3/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L540-L556]
> Ensure that all threads get their own copy of the {{HiveConf}} object to use
> and modify.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20665:
---
Description:
When parallel tasks are enabled in Hive, all of the resulting queries share the
same Hive configuration. This is problematic as each query will modify the
same {{HiveConf}} object with things like query ID and query text. This will
overwrite each other and cause {{ConcurrentModificationException}} issues.
{code:java|title=SQLOperation.java}
public Object run() throws HiveSQLException {
Hive.set(parentHive, false);
// TODO: can this result in cross-thread reuse of session state?
SessionState.setCurrentSessionState(parentSessionState);
PerfLogger.setPerfLogger(SessionState.getPerfLogger());
LogUtils.registerLoggingContext(queryState.getConf());
try {
if (asyncPrepare) {
prepare(queryState);
}
runQuery();
} catch (HiveSQLException e) {
// ...
{code}
[Code
Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
>From this code we can see that for every thread launched, they are all calling
>{{setCurrentSessionState}}.
{code:java|title=SessionStates.java}
/**
* Sets the given session state in the thread local var for sessions.
*/
public static void setCurrentSessionState(SessionState startSs) {
tss.get().attach(startSs);
}
// SessionState is not available in runtime and Hive.get().getConf() is not
safe to call
private static class SessionStates {
private SessionState state;
private HiveConf conf;
private void attach(SessionState state) {
this.state = state;
attach(state.getConf());
}
private void attach(HiveConf conf) {
// -- SHALLOW COPY HERE, ALL THREADS SHARING SAME REFERENCE -- //
this.conf = conf;
ClassLoader classLoader = conf.getClassLoader();
if (classLoader != null) {
Thread.currentThread().setContextClassLoader(classLoader);
}
}
}
{code}
[Code
Here|https://github.com/apache/hive/blob/7795c0a7dc59941671f8845d78b16d9e5ddc9ea3/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L540-L556]
Ensure that all threads get their own copy of the {{HiveConf}} object to use
and modify.
was:
When parallel tasks are enabled in Hive, all of the resulting queries share the
same Hive configuration. This is problematic as each query will modify the
same {{HiveConf}} object with things like query ID and query text. This will
overwrite each other and cause {{ConcurrentModificationException}} issues.
> Hive Parallel Tasks - Hive Configuration ConcurrentModificationException
>
>
> Key: HIVE-20665
> URL: https://issues.apache.org/jira/browse/HIVE-20665
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.2, 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Priority: Major
>
> When parallel tasks are enabled in Hive, all of the resulting queries share
> the same Hive configuration. This is problematic as each query will modify
> the same {{HiveConf}} object with things like query ID and query text. This
> will overwrite each other and cause {{ConcurrentModificationException}}
> issues.
> {code:java|title=SQLOperation.java}
> public Object run() throws HiveSQLException {
> Hive.set(parentHive, false);
> // TODO: can this result in cross-thread reuse of session state?
> SessionState.setCurrentSessionState(parentSessionState);
> PerfLogger.setPerfLogger(SessionState.getPerfLogger());
> LogUtils.registerLoggingContext(queryState.getConf());
> try {
> if (asyncPrepare) {
> prepare(queryState);
> }
> runQuery();
> } catch (HiveSQLException e) {
> // ...
> {code}
> [Code
> Here|https://github.com/apache/hive/blob/6e27a5315a44c55ef3b178e7212c9068de322d01/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L308-L319]
> From this code we can see that for every thread launched, they are all
> calling {{setCurrentSessionState}}.
> {code:java|title=SessionStates.java}
> /**
>* Sets the given session state in the thread local var for sessions.
>*/
> public static void setCurrentSessionState(SessionState startSs) {
> tss.get().attach(startSs);
> }
> // SessionState is not available in runtime and Hive.get().getConf() is not
> safe to call
> private static class SessionStates {
> private SessionState state;
> private HiveConf conf;
> private void attach(SessionState state) {
> this.state =
[jira] [Commented] (HIVE-20535) Add new configuration to set the size of the global compile lock
[
https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634258#comment-16634258
]
Hive QA commented on HIVE-20535:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 3 new + 142 unchanged - 6
fixed = 145 total (was 148) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14159/dev-support/hive-personality.sh
|
| git revision | master / c05733e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14159/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14159/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add new configuration to set the size of the global compile lock
>
>
> Key: HIVE-20535
> URL: https://issues.apache.org/jira/browse/HIVE-20535
> Project: Hive
> Issue Type: Task
> Components: HiveServer2
>Reporter: denys kuzmenko
>Assignee: denys kuzmenko
>Priority: Major
> Attachments: HIVE-20535.1.patch, HIVE-20535.10.patch,
> HIVE-20535.11.patch, HIVE-20535.12.patch, HIVE-20535.13.patch,
> HIVE-20535.14.patch, HIVE-20535.15.patch, HIVE-20535.16.patch,
> HIVE-20535.17.patch, HIVE-20535.18.patch, HIVE-20535.19.patch,
> HIVE-20535.2.patch, HIVE-20535.20.patch, HIVE-20535.21.patch,
> HIVE-20535.3.patch, HIVE-20535.4.patch, HIVE-20535.5.patch,
> HIVE-20535.6.patch, HIVE-20535.8.patch, HIVE-20535.9.patch
>
>
> When removing the compile lock, it is quite risky to remove it entirely.
> It would be good to provide a pool size for the concurrent compilation, so
> the administrator can limit the load
--
This message was
[jira] [Commented] (HIVE-20662) Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
[ https://issues.apache.org/jira/browse/HIVE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634245#comment-16634245 ] Karen Coppage commented on HIVE-20662: -- This test succeeded locally at the commits of HIVE-20636 and HIVE-20637, so they are not the obvious culprits. More digging is needed. In the meantime, [~pvary] or [~asherman], could you help by committing? > Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] > > > Key: HIVE-20662 > URL: https://issues.apache.org/jira/browse/HIVE-20662 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Blocker > Attachments: HIVE-20662.patch > > > TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] is more than flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20662) Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
[ https://issues.apache.org/jira/browse/HIVE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-20662: - Attachment: HIVE-20662.patch Status: Patch Available (was: Open) > Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] > > > Key: HIVE-20662 > URL: https://issues.apache.org/jira/browse/HIVE-20662 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Blocker > Attachments: HIVE-20662.patch > > > TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] is more than flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634241#comment-16634241 ] Daniel Dai commented on HIVE-20646: --- LGTM, +1. > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-20662) Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
[ https://issues.apache.org/jira/browse/HIVE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-20662: - Comment: was deleted (was: Re-enable) > Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] > > > Key: HIVE-20662 > URL: https://issues.apache.org/jira/browse/HIVE-20662 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Blocker > > TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] is more than flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20662) Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
[ https://issues.apache.org/jira/browse/HIVE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634227#comment-16634227 ] Karen Coppage commented on HIVE-20662: -- Re-enable > Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] > > > Key: HIVE-20662 > URL: https://issues.apache.org/jira/browse/HIVE-20662 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Blocker > > TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] is more than flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20664) Potential ArrayIndexOutOfBoundsException in VectorizedOrcAcidRowBatchReader.findMinMaxKeys
[
https://issues.apache.org/jira/browse/HIVE-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saurabh Seth updated HIVE-20664:
Attachment: HIVE-20664.patch
Status: Patch Available (was: Open)
Removed the condition that could cause the exception and added a check after
the loop to handle the case when a split is completely within an orc stripe.
> Potential ArrayIndexOutOfBoundsException in
> VectorizedOrcAcidRowBatchReader.findMinMaxKeys
> --
>
> Key: HIVE-20664
> URL: https://issues.apache.org/jira/browse/HIVE-20664
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Saurabh Seth
>Assignee: Saurabh Seth
>Priority: Minor
> Attachments: HIVE-20664.patch
>
>
> [~ekoifman], could you please confirm if my understanding is correct and if
> so, review the fix?
> In the method {{VectorizedOrcAcidRowBatchReader.findMinMaxKeys}}, the code
> snippet that identifies the first and last stripe indices in the current
> split could result in an ArrayIndexOutOfBoundsException if a complete split
> is within the same stripe:
> {noformat}
> for(int i = 0; i < stripes.size(); i++) {
> StripeInformation stripe = stripes.get(i);
> long stripeEnd = stripe.getOffset() + stripe.getLength();
> if(firstStripeIndex == -1 && stripe.getOffset() >= splitStart) {
> firstStripeIndex = i;
> }
> if(lastStripeIndex == -1 && splitEnd <= stripeEnd &&
> stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset() ) {
> //the last condition is for when both splitStart and splitEnd are in
> // the same stripe
> lastStripeIndex = i;
> }
> }
> {noformat}
> Consider the example where there are 2 stripes - 0-500 and 500-1000 and
> splitStart is 600 and splitEnd is 800.
> In the first iteration of the loop, stripe.getOffset() is 0 and stripeEnd is
> 500. In this iteration, neither of the if statement conditions will be met
> and firstSripeIndex as well as lastStripeIndex remain -1.
> In the second iteration of the loop stripe.getOffset() is 500, stripeEnd is
> 1000, The first if statement condition will not be met in this case because
> stripe's offset (500) is not greater than or equal to the splitStart (600).
> However, in the second if statement, splitEnd (800) is <= stripeEnd(1000) and
> it will try to compute the last condition
> {{stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset()}}. This
> will throw an ArrayIndexOutOfBoundsException because firstStripeIndex is
> still -1.
> I'm not sure if this scenario is possible at all, hence logging this as a low
> priority issue. Perhaps block based split generation using BISplitStrategy
> could trigger this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20664) Potential ArrayIndexOutOfBoundsException in VectorizedOrcAcidRowBatchReader.findMinMaxKeys
[
https://issues.apache.org/jira/browse/HIVE-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saurabh Seth reassigned HIVE-20664:
---
> Potential ArrayIndexOutOfBoundsException in
> VectorizedOrcAcidRowBatchReader.findMinMaxKeys
> --
>
> Key: HIVE-20664
> URL: https://issues.apache.org/jira/browse/HIVE-20664
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Saurabh Seth
>Assignee: Saurabh Seth
>Priority: Minor
>
> [~ekoifman], could you please confirm if my understanding is correct and if
> so, review the fix?
> In the method {{VectorizedOrcAcidRowBatchReader.findMinMaxKeys}}, the code
> snippet that identifies the first and last stripe indices in the current
> split could result in an ArrayIndexOutOfBoundsException if a complete split
> is within the same stripe:
> {noformat}
> for(int i = 0; i < stripes.size(); i++) {
> StripeInformation stripe = stripes.get(i);
> long stripeEnd = stripe.getOffset() + stripe.getLength();
> if(firstStripeIndex == -1 && stripe.getOffset() >= splitStart) {
> firstStripeIndex = i;
> }
> if(lastStripeIndex == -1 && splitEnd <= stripeEnd &&
> stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset() ) {
> //the last condition is for when both splitStart and splitEnd are in
> // the same stripe
> lastStripeIndex = i;
> }
> }
> {noformat}
> Consider the example where there are 2 stripes - 0-500 and 500-1000 and
> splitStart is 600 and splitEnd is 800.
> In the first iteration of the loop, stripe.getOffset() is 0 and stripeEnd is
> 500. In this iteration, neither of the if statement conditions will be met
> and firstSripeIndex as well as lastStripeIndex remain -1.
> In the second iteration of the loop stripe.getOffset() is 500, stripeEnd is
> 1000, The first if statement condition will not be met in this case because
> stripe's offset (500) is not greater than or equal to the splitStart (600).
> However, in the second if statement, splitEnd (800) is <= stripeEnd(1000) and
> it will try to compute the last condition
> {{stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset()}}. This
> will throw an ArrayIndexOutOfBoundsException because firstStripeIndex is
> still -1.
> I'm not sure if this scenario is possible at all, hence logging this as a low
> priority issue. Perhaps block based split generation using BISplitStrategy
> could trigger this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20635) VectorizedOrcAcidRowBatchReader doesn't filter delete events for original files
[
https://issues.apache.org/jira/browse/HIVE-20635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saurabh Seth updated HIVE-20635:
Attachment: HIVE-20635.patch
Status: Patch Available (was: Open)
Added delete event filtering for original file splits in
{{VectorizedOrcAcidRowBatchReader.findMinMaxKeys}}.
> VectorizedOrcAcidRowBatchReader doesn't filter delete events for original
> files
> ---
>
> Key: HIVE-20635
> URL: https://issues.apache.org/jira/browse/HIVE-20635
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20635.patch
>
>
> this is a followup to HIVE-16812 which adds support for delete event
> filtering for splits from native acid files
> need to add the same for {{OrcSplit.isOriginal()}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-10296) Cast exception observed when hive runs a multi join query on metastore (postgres), since postgres pushes the filter into the join, and ignores the condition before applyi
[
https://issues.apache.org/jira/browse/HIVE-10296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-10296:
---
Resolution: Fixed
Fix Version/s: 3.2.0
4.0.0
Status: Resolved (was: Patch Available)
I committed this to master.
Thanks [~karthik.manamcheri] for your contribution, and [~sershe] for you
review.
> Cast exception observed when hive runs a multi join query on metastore
> (postgres), since postgres pushes the filter into the join, and ignores the
> condition before applying cast
> -
>
> Key: HIVE-10296
> URL: https://issues.apache.org/jira/browse/HIVE-10296
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1
>Reporter: Yash Datta
>Assignee: Karthik Manamcheri
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-10296.1.patch
>
>
> Try to drop a partition from hive:
> ALTER TABLE f___edr_bin_source___900_sub_id DROP IF EXISTS PARTITION (
> exporttimestamp=1427824800, timestamp=1427824800)
> This triggers a query on the metastore like this :
> "select "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? inner join
> "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 inner join
> "PARTITION_KEY_VALS" "FILTER1" on "FILTER1"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER1"."INTEGER_IDX" = 1 where ( (((case when
> "TBLS"."TBL_NAME" = ? and "DBS"."NAME" = ? then cast("FILTER0"."PART_KEY_VAL"
> as decimal(21,0)) else null end) = ?) and ((case when "TBLS"."TBL_NAME" = ?
> and "DBS"."NAME" = ? then cast("FILTER1"."PART_KEY_VAL" as decimal(21,0))
> else null end) = ?)) )"
> In some cases, when the internal tables in postgres (metastore) have some
> amount of data, the query plan pushes the condition down into the join.
> Now because of DERBY-6358 , case when clause is used before the cast, but in
> this case , cast is evaluated before condition being evaluated. So in case we
> have different tables partitioned on string and integer columns, cast
> exception is observed!
> 15/04/06 08:41:20 ERROR metastore.ObjectStore: Direct SQL failed, falling
> back to ORM
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? inner join
> "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 inner join
> "PARTITION_KEY_VALS" "FILTER1" on "FILTER1"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER1"."INTEGER_IDX" = 1 where ( (((case when
> "TBLS"."TBL_NAME" = ? and "DBS"."NAME" = ? then cast("FILTER0"."PART_KEY_VAL"
> as decimal(21,0)) else null end) = ?) and ((case when "TBLS"."TBL_NAME" = ?
> and "DBS"."NAME" = ? then cast("FILTER1"."PART_KEY_VAL" as decimal(21,0))
> else null end) = ?)) )".
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>
> at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:300)
>
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:211)
>
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getSqlResult(ObjectStore.java:1915)
>
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getSqlResult(ObjectStore.java:1909)
>
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2208)
>
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:1909)
>
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExpr(ObjectStore.java:1882)
>
> org.postgresql.util.PSQLException: ERROR: invalid input syntax for type
> numeric: "__DEFAULT_BINSRC__"
> 15/04/06 08:41:20 INFO metastore.ObjectStore: JDO filter pushdown cannot be
> used: Filtering is supported only on partition keys of type string
> 15/04/06 08:41:20 ERROR metastore.ObjectStore:
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
>
> at
[jira] [Commented] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[
https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634201#comment-16634201
]
Hive QA commented on HIVE-20646:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941953/HIVE-20646.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15007 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14158/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14158/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14158/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941953 - PreCommit-HIVE-Build
> Partition filter condition is not pushed down to metastore query if it has IS
> NOT NULL.
> ---
>
> Key: HIVE-20646
> URL: https://issues.apache.org/jira/browse/HIVE-20646
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: Metastore, Performance, pull-request-available
> Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch
>
>
> If the partition filter condition has "is not null" then the filter query
> isn't getting pushed to the SQL query in RDMBS.
> This slows down metastore api calls for getting list of partitions with
> filter condition.
> This condition gets added by optimizer in many cases so this is affecting
> many queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20662) Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
[ https://issues.apache.org/jira/browse/HIVE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-20662: > Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] > > > Key: HIVE-20662 > URL: https://issues.apache.org/jira/browse/HIVE-20662 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Blocker > > TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] is more than flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20627) Concurrent async queries intermittently fails with LockException and cause memory leak.
[
https://issues.apache.org/jira/browse/HIVE-20627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634189#comment-16634189
]
Sankar Hariappan commented on HIVE-20627:
-
[~pvary], Thanks for your attention and it is really a good question!
{quote}
* Do you know what will happen with Hive objects used by the async threads if
the master threads get another asynchronous query which requires
HiveMetaStoreClient reinitialization, like {{set
hive.metastore.client.socket.timeout=500ms}}?
{quote}
AFAIK, the master thread initializes the *sessionHive* object in
*HiveSessionImpl* class when we open a new session for a client connection and
by default all queries from this connection shares the same sessionHive object.
If the master thread executes a *synchronous* query, it closes the sessionHive
object (referred via thread local hiveDb) if {{Hive.isCompatible}} returns
false and sets new Hive object in thread local HiveDb but doesn't change the
sessionHive object in the session. Whereas, *asynchronous* query execution via
async threads never closes the sessionHive object and it just creates a new one
if needed and sets it as their thread local hiveDb.
So, the same problem can happen in the case where an *asynchronous* query is
being executed by async threads refers to sessionHive object and the master
thread receives a *synchronous* query that closes the same sessionHive object.
However, I'm not sure if Hive allows to run both *synchronous* and
*asynchronous* query from same session.
Also, each query execution overwrites the thread local hiveDb object to
sessionHive object which potentially leaks a metastore connection if the
previous synchronous query execution re-created the Hive object.
*Possible fix:*
* Master thread should never close the sessionHive object even for
*synchronous* execution.
* Each query execution will have to close the thread local Hive object of
master thread if it is different from sessionHive object (to avoid memory leak)
before overwriting it.
Please take a look at my explanation and let me know if my understanding is
correct.
cc [~daijy], [~thejas]
> Concurrent async queries intermittently fails with LockException and cause
> memory leak.
> ---
>
> Key: HIVE-20627
> URL: https://issues.apache.org/jira/browse/HIVE-20627
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Transactions
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20627.01.patch
>
>
> When multiple async queries are executed from same session, it leads to
> multiple async query execution DAGs share the same Hive object which is set
> by caller for all threads. In case of loading dynamic partitions, it creates
> MoveTask which re-creates the Hive object and closes the shared Hive object
> which causes metastore connection issues for other async execution thread who
> still access it. This is also seen if ReplDumpTask and ReplLoadTask are part
> of the DAG.
> *Call Stack:*
> {code:java}
> 2018-09-16T04:38:04,280 ERROR [load-dynamic-partitions-7]: metadata.Hive
> (Hive.java:call(2436)) - Exception when loading partition with parameters
> partPath=hdfs://mycluster/warehouse/tablespace/managed/hive/tbl_3bcvvdubni/.hive-staging_hive_2018-09-16_04-35-50_708_7776079613819042057-1147/-ext-1/age=55,
> table=tbl_3bcvvdubni, partSpec={age=55}, loadFileType=KEEP_EXISTING,
> listBucketingLevel=0, isAcid=true, hasFollowingStatsTask=true
> org.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating with the
> metastore
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidWriteIds(DbTxnManager.java:714)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableValidWriteIdListWithTxnList(AcidUtils.java:1791)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableSnapshot(AcidUtils.java:1756)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableSnapshot(AcidUtils.java:1714)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1976)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2415)
> [hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2406)
> [hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
> at
>
[jira] [Commented] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[
https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634179#comment-16634179
]
Hive QA commented on HIVE-20646:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
40s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
11s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
11s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 52 unchanged - 1
fixed = 52 total (was 53) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14158/dev-support/hive-personality.sh
|
| git revision | master / 2ad65f7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common ql
standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14158/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Partition filter condition is not pushed down to metastore query if it has IS
> NOT NULL.
> ---
>
> Key: HIVE-20646
> URL: https://issues.apache.org/jira/browse/HIVE-20646
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: Metastore,
[jira] [Updated] (HIVE-20575) Fix flaky connection metric tests
[ https://issues.apache.org/jira/browse/HIVE-20575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20575: - Attachment: HIVE-20575.09.patch > Fix flaky connection metric tests > - > > Key: HIVE-20575 > URL: https://issues.apache.org/jira/browse/HIVE-20575 > Project: Hive > Issue Type: Test > Components: Hive, Test >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-20575.01.patch, HIVE-20575.02.patch, > HIVE-20575.03.patch, HIVE-20575.04.patch, HIVE-20575.05.patch, > HIVE-20575.06.patch, HIVE-20575.07.patch, HIVE-20575.08.patch, > HIVE-20575.09.patch > > > TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics() is flaky. We need to > fix it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1
[ https://issues.apache.org/jira/browse/HIVE-20661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20661: - Attachment: HIVE-20661.01.patch > Dynamic partitions loading calls add partition for every partition 1-by-1 > - > > Key: HIVE-20661 > URL: https://issues.apache.org/jira/browse/HIVE-20661 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20661.01.patch > > > Hive.loadDynamicPartitions creates partitions using a threadpool, but the > update of the Metastore via the MetastoreClient is done 1-by-1. This adds > unnecessary extra calls. The partitions should be created in one batch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20544) TOpenSessionReq logs password and username
[
https://issues.apache.org/jira/browse/HIVE-20544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634092#comment-16634092
]
Hive QA commented on HIVE-20544:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941949/HIVE-20544.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15008 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14157/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14157/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14157/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941949 - PreCommit-HIVE-Build
> TOpenSessionReq logs password and username
> --
>
> Key: HIVE-20544
> URL: https://issues.apache.org/jira/browse/HIVE-20544
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: beginner, patch, security
> Attachments: HIVE-20544.1.patch, HIVE-20544.2.patch,
> HIVE-20544.3.patch, HIVE-20544.3.patch, HIVE-20544.4.patch,
> HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.4.patch,
> HIVE-20544.4.patch, HIVE-20544.patch, non-solution.patch,
> working-solution.patch
>
>
> In
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq,
> if client protocol is unset, validate() and toString() prints both username
> and password to logs.
> Logging a password is a security risk. We should hide the ***.
> =Edit= (no longer relevant, see comments)
> This issue is tricky since it is caused in a fully generated class. I've been
> playing around and have found one working solution, butI'd truly appreciate
> ideas for a more elegant solution or input.
> The problem:
> TCLIService.thrift is the template for generating all classes in
> service-rpc. Struct TOpenSessionReq is OpenSession()'s one parameter and is
> defined thus:
> {noformat}
> struct TOpenSessionReq {
> 1: required TProtocolVersion client_protocol =
> TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V10
> 2: optional string username
> 3: optional string password
> 4: optional map configuration
> }
> {noformat}
> In the generated class TOpenSessionReq.java, client_protocol is checked by a
> validate() method, which is called quite a few times; if client_protocol is
> not set, it throws a TProtocolException, passing along a toString(). This
> toString() gets the names and values of all fields, including username and
> password.
> Working solution:
> * Create a separate struct containing only the username and password, and
> pass it to OpenSession() as a second parameter. Since all fields in the new
> struct are "optional", the generated validate() is empty – toString() is
> never used. This involves changing core classes and breaks the "Each function
> should take exactly one parameter" coding convention (detailed at
> service-rpc/if/TCLIService.thrift:27).
> See working-solution.patch.
> What doesn't work:
> * Making client_protocol optional instead of required. Apparently this will
> break everything.
> * Overwriting toString() – TOpenSessionReq is a struct.
> * Creating two Thrift structs, one struct for required (TRequiredReq) and
> one for optional (TOptionalReq) fields, and nesting them in struct
> TOpenSessionReq. This doesn't work because validate() in TOpenSessionReq can
> call TOptionalReq.toString(), which prints the password to logs. This will
> happen if TRequiredReq.client_protocol isn't set.
> See non-solution.patch
> * Asking Thrift devs to change their code. I wrote them an email but have no
> expectations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1
[ https://issues.apache.org/jira/browse/HIVE-20661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-20661: > Dynamic partitions loading calls add partition for every partition 1-by-1 > - > > Key: HIVE-20661 > URL: https://issues.apache.org/jira/browse/HIVE-20661 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > > Hive.loadDynamicPartitions creates partitions using a threadpool, but the > update of the Metastore via the MetastoreClient is done 1-by-1. This adds > unnecessary extra calls. The partitions should be created in one batch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12254) Improve logging with yarn/hdfs
[
https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634066#comment-16634066
]
Peter Vary commented on HIVE-12254:
---
[~aihuaxu]: 3 quick questions:
* What happens when multiple queries are running / compiling in parallel? They
might be a mix up between the context for queries, especially if async
compilation is turned on {{hive.server2.async.exec.async.compile}}. Or this is
handled some way by the Shims themselves?
* In the previous version of the patch [~vikram.dixit] tried to reset the
context to the original value after the query is finished the run. I think it
was useful to reset it to the sessionId after every query is finished. Do we
need this? Or there is no hdfs/yarn commands when we do not running a query?
* In {{Hadoop23Shims.setHadoopSessionContext}} there is ":" in the name, in
{{Hadoop23Shims.setHadoopQueryContext}} it is "_". This might be a mistake.
Thanks,
Peter
> Improve logging with yarn/hdfs
> --
>
> Key: HIVE-12254
> URL: https://issues.apache.org/jira/browse/HIVE-12254
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 1.2.1
>Reporter: Vikram Dixit K
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch,
> HIVE-12254.3.patch
>
>
> In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both
> HIVE-12249 and HDFS-9184 are required (and upgraded in hive for the HDFS
> issue) before this can be resolved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20619) Include MultiDelimitSerDe in HIveServer2 By Default
[
https://issues.apache.org/jira/browse/HIVE-20619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-20619:
-
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Fix has been pushed to master. Thank you for your contribution [~afan]
> Include MultiDelimitSerDe in HIveServer2 By Default
> ---
>
> Key: HIVE-20619
> URL: https://issues.apache.org/jira/browse/HIVE-20619
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Serializers/Deserializers
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20619.2.patch
>
>
> In [HIVE-20020], the hive-contrib JAR file was removed from the HiveServer2
> classpath. With this change, the {{MultiDelimitSerDe}} is no longer
> included. This is fine, because {{MultiDelimitSerDe}} was a pain in that
> environment anyway. It was available to HiveServer2, and therefore would
> work with a limited set of queries (select * from table limit 1) but any
> other query on that table which launched a MapReduce project would fail
> because the hive-contrib JAR file was not sent out with the rest of the Hive
> JARs for MapReduce jobs.
> Please bring {{MultiDelimitSerDe}} back into the fold so that it's available
> to users out of the box without having to install the hive-contrib JAR into
> the HiveServer2 auxiliary directory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20544) TOpenSessionReq logs password and username
[
https://issues.apache.org/jira/browse/HIVE-20544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634022#comment-16634022
]
Hive QA commented on HIVE-20544:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14157/dev-support/hive-personality.sh
|
| git revision | master / 4570807 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14157/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: service-rpc itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14157/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TOpenSessionReq logs password and username
> --
>
> Key: HIVE-20544
> URL: https://issues.apache.org/jira/browse/HIVE-20544
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: beginner, patch, security
> Attachments: HIVE-20544.1.patch, HIVE-20544.2.patch,
> HIVE-20544.3.patch, HIVE-20544.3.patch, HIVE-20544.4.patch,
> HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.4.patch,
> HIVE-20544.4.patch, HIVE-20544.patch, non-solution.patch,
> working-solution.patch
>
>
> In
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq,
> if client protocol is unset, validate() and toString() prints both username
> and password to logs.
> Logging a password is a security risk. We should hide the ***.
> =Edit= (no longer relevant, see
[jira] [Commented] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[
https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633974#comment-16633974
]
Hive QA commented on HIVE-20545:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941946/HIVE-20545.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15006 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=195)
[druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14156/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14156/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14156/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941946 - PreCommit-HIVE-Build
> Ability to exclude potentially large parameters in HMS Notifications
>
>
> Key: HIVE-20545
> URL: https://issues.apache.org/jira/browse/HIVE-20545
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch,
> HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch,
> HIVE-20545.6.patch
>
>
> Clients can add large-sized parameters in Table/Partition objects. So we need
> to enable adding regex patterns through HiveConf to match parameters to be
> filtered from table and partition objects before serialization in HMS
> notifications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20627) Concurrent async queries intermittently fails with LockException and cause memory leak.
[
https://issues.apache.org/jira/browse/HIVE-20627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633957#comment-16633957
]
Peter Vary commented on HIVE-20627:
---
[~sankarh]: Very nice description! Thanks for it and for the patch too!
Reading it I think there could be a problem the other way around too:
* Do you know what will happen with Hive objects used by the async threads if
the master threads get another asynchronous query which requires
HiveMetaStoreClient reinitialization, like {{set
hive.metastore.client.socket.timeout=500ms}}?
My guess is in this case the {{Hive.isCompatible}} returns false and the
original Hive instance is closed and a new one is created on the master thread.
This might cause the same problem for the async threads.
Sadly I did not have time to test this out, but probably you already have
better understanding of this part of the code, and can confirm or refute my
theory.
Thanks,
Peter
> Concurrent async queries intermittently fails with LockException and cause
> memory leak.
> ---
>
> Key: HIVE-20627
> URL: https://issues.apache.org/jira/browse/HIVE-20627
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Transactions
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20627.01.patch
>
>
> When multiple async queries are executed from same session, it leads to
> multiple async query execution DAGs share the same Hive object which is set
> by caller for all threads. In case of loading dynamic partitions, it creates
> MoveTask which re-creates the Hive object and closes the shared Hive object
> which causes metastore connection issues for other async execution thread who
> still access it. This is also seen if ReplDumpTask and ReplLoadTask are part
> of the DAG.
> *Call Stack:*
> {code:java}
> 2018-09-16T04:38:04,280 ERROR [load-dynamic-partitions-7]: metadata.Hive
> (Hive.java:call(2436)) - Exception when loading partition with parameters
> partPath=hdfs://mycluster/warehouse/tablespace/managed/hive/tbl_3bcvvdubni/.hive-staging_hive_2018-09-16_04-35-50_708_7776079613819042057-1147/-ext-1/age=55,
> table=tbl_3bcvvdubni, partSpec={age=55}, loadFileType=KEEP_EXISTING,
> listBucketingLevel=0, isAcid=true, hasFollowingStatsTask=true
> org.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating with the
> metastore
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidWriteIds(DbTxnManager.java:714)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableValidWriteIdListWithTxnList(AcidUtils.java:1791)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableSnapshot(AcidUtils.java:1756)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getTableSnapshot(AcidUtils.java:1714)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1976)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2415)
> [hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:2406)
> [hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_171]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_171]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
> Caused by: org.apache.thrift.protocol.TProtocolException: Required field
> 'validTxnList' is unset!
> Struct:GetValidWriteIdsRequest(fullTableNames:[default.tbl_3bcvvdubni],
> validTxnList:null)
> at
> org.apache.hadoop.hive.metastore.api.GetValidWriteIdsRequest.validate(GetValidWriteIdsRequest.java:396)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args.validate(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args$get_valid_write_ids_argsStandardScheme.write(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.0.1.0-184.jar:3.1.0.3.0.1.0-184]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args$get_valid_write_ids_argsStandardScheme.write(ThriftHiveMetastore.java)
>
[jira] [Updated] (HIVE-20647) HadoopVer was ignored in QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] denys kuzmenko updated HIVE-20647: -- Attachment: HIVE-20647.5.patch > HadoopVer was ignored in QTestUtil > -- > > Key: HIVE-20647 > URL: https://issues.apache.org/jira/browse/HIVE-20647 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch, > HIVE-20647.3.patch, HIVE-20647.4.patch, HIVE-20647.5.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[
https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633934#comment-16633934
]
Hive QA commented on HIVE-20545:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
35s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14156/dev-support/hive-personality.sh
|
| git revision | master / 4570807 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14156/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Ability to exclude potentially large parameters in HMS Notifications
>
>
> Key: HIVE-20545
> URL: https://issues.apache.org/jira/browse/HIVE-20545
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch,
> HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch,
> HIVE-20545.6.patch
>
>
> Clients can add large-sized parameters in Table/Partition objects. So we need
> to enable adding regex patterns through HiveConf to match parameters to be
> filtered from table and partition objects before serialization in HMS
> notifications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904
[
https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633907#comment-16633907
]
Hive QA commented on HIVE-20638:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941908/HIVE-20638.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15007 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=160)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14155/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14155/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14155/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12941908 - PreCommit-HIVE-Build
> Upgrade version of Jetty to 9.3.25.v20180904
>
>
> Key: HIVE-20638
> URL: https://issues.apache.org/jira/browse/HIVE-20638
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch,
> HIVE-20638.03.patch
>
>
> Current version is 9.3.20.v20170531
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633893#comment-16633893 ] Zoltan Haindrich commented on HIVE-20647: - Thank you [~dkuzmenko] for taking care of this! I think that hadoopver is not used anymore...and since to my best knowledge even hive3 does not support even hadoop 2.x I feel that this is not needed anymore... according to https://hive.apache.org/downloads.html it seems to me that the last release line which was supporting multiple hadoop versions was branch-1 > HadoopVer was ignored in QTestUtil > -- > > Key: HIVE-20647 > URL: https://issues.apache.org/jira/browse/HIVE-20647 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch, > HIVE-20647.3.patch, HIVE-20647.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633877#comment-16633877 ] Peter Vary commented on HIVE-20647: --- I think it is a good to have a little cleanup in QTestUtil. [~kgyrtkirk]: Could you please verify that this will not break anything for your extra tests? (Like setting a class final or moving methods to private) +1 from my side pending tests > HadoopVer was ignored in QTestUtil > -- > > Key: HIVE-20647 > URL: https://issues.apache.org/jira/browse/HIVE-20647 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch, > HIVE-20647.3.patch, HIVE-20647.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904
[
https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633876#comment-16633876
]
Hive QA commented on HIVE-20638:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
39s{color} | {color:red} root in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
44s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 44s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14155/dev-support/hive-personality.sh
|
| git revision | master / 4570807 |
| Default Java | 1.8.0_111 |
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14155/yetus/branch-compile-root.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14155/yetus/patch-compile-root.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14155/yetus/patch-compile-root.txt
|
| modules | C: serde . hbase-handler hcatalog/webhcat/svr U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14155/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade version of Jetty to 9.3.25.v20180904
>
>
> Key: HIVE-20638
> URL: https://issues.apache.org/jira/browse/HIVE-20638
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch,
> HIVE-20638.03.patch
>
>
> Current version is 9.3.20.v20170531
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20613) CachedStore: Add more UT coverage (outside of .q files)
[ https://issues.apache.org/jira/browse/HIVE-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-20613: Attachment: HIVE-20615.1.patch > CachedStore: Add more UT coverage (outside of .q files) > --- > > Key: HIVE-20613 > URL: https://issues.apache.org/jira/browse/HIVE-20613 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-20615.1.patch > > > 1. Add tests which will use the background thread for updating the cached > data (database, table, partition, table stats, partition stats) > 2. Add more tests for existing APIs: stats aggregation, listing partitions > when partial specs are provided, testing the storage descriptor > caching/deduplication (specially when tables/ptns are dropped/added), table > col stats, partition col stats > 3. Test 1. in a multithreaded scenario > 4. Test whitelist/blacklist > 5. Test prewarm memory limit estimation code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
