[jira] [Commented] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[
https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352147#comment-16352147
]
Hive QA commented on HIVE-18467:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} standalone-metastore: The patch generated 6 new + 13
unchanged - 4 fixed = 19 total (was 17) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 5 new + 515 unchanged - 3
fixed = 520 total (was 518) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hive-unit: The patch generated 16 new + 12
unchanged - 3 fixed = 28 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0a328f0 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9016/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9016/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9016/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9016/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> support whole warehouse dump / load + create/drop database events
> -
>
> Key: HIVE-18467
> URL: https://issues.apache.org/jira/browse/HIVE-18467
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch,
> HIVE-18467.2.patch, HIVE-18467.5.patch
>
>
> A complete hive warehouse might be required to replicate to a DR site for
> certain use cases and rather than allowing only a database name in the REPL
> DUMP commands,
[jira] [Updated] (HIVE-18553) VectorizedParquetReader fails after adding a new column to table
[
https://issues.apache.org/jira/browse/HIVE-18553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu updated HIVE-18553:
Attachment: HIVE-18553.4.patch
> VectorizedParquetReader fails after adding a new column to table
>
>
> Key: HIVE-18553
> URL: https://issues.apache.org/jira/browse/HIVE-18553
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
>Priority: Major
> Attachments: HIVE-18553.2.patch, HIVE-18553.3.patch,
> HIVE-18553.4.patch, HIVE-18553.patch
>
>
> VectorizedParquetReader throws an exception when trying to reading from a
> parquet table on which new columns are added. Steps to reproduce below:
> {code}
> 0: jdbc:hive2://localhost:1/default> desc test_p;
> +---++--+
> | col_name | data_type | comment |
> +---++--+
> | t1| tinyint| |
> | t2| tinyint| |
> | i1| int| |
> | i2| int| |
> +---++--+
> 0: jdbc:hive2://localhost:1/default> set hive.fetch.task.conversion=none;
> 0: jdbc:hive2://localhost:1/default> set
> hive.vectorized.execution.enabled=true;
> 0: jdbc:hive2://localhost:1/default> alter table test_p add columns (ts
> timestamp);
> 0: jdbc:hive2://localhost:1/default> select * from test_p;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)
> {code}
> Following exception is seen in the logs
> {code}
> Caused by: java.lang.IllegalArgumentException: [ts] BINARY is not in the
> store: [[i1] INT32, [i2] INT32, [t1] INT32, [t2] INT32] 3
> at
> org.apache.parquet.hadoop.ColumnChunkPageReadStore.getPageReader(ColumnChunkPageReadStore.java:160)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:479)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:432)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:393)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:345)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:88)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:167)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:52)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:229)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:142)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
> ~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
> ~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
> ~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
> ~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(Local
[jira] [Commented] (HIVE-18523) Fix summary row in case there are no inputs
[ https://issues.apache.org/jira/browse/HIVE-18523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352156#comment-16352156 ] Zoltan Haindrich commented on HIVE-18523: - I've rerun all test which have failed first in this build; and all of them passed [~ashutoshc] Could you please take a look? cc: [~prasanth_j] > Fix summary row in case there are no inputs > --- > > Key: HIVE-18523 > URL: https://issues.apache.org/jira/browse/HIVE-18523 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18523.01.patch > > > HIVE-17617 added a fix to handle empty groups correctly; however it might be > possible that the summary row is returned multiple times - in case the > mappers are "seem" to be optimized away... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18622) Vectorization: IF statement, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline reassigned HIVE-18622:
---
> Vectorization: IF statement, Comparisons, and more do not handle NULLs
> correctly
>
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
>
>
> Many vector expression classes are missing guards around setting noNulls
> among other things.
> {code:java}
> // Carefully update noNulls...
> if (outputColVector.noNulls) {
> outputColVector.noNulls = inputColVector.noNulls;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF statement, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-18622:
Attachment: HIVE-18622.01.patch
> Vectorization: IF statement, Comparisons, and more do not handle NULLs
> correctly
>
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.01.patch
>
>
>
> Many vector expression classes are missing guards around setting noNulls
> among other things.
> {code:java}
> // Carefully update noNulls...
> if (outputColVector.noNulls) {
> outputColVector.noNulls = inputColVector.noNulls;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF statement, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-18622:
Status: Patch Available (was: Open)
Work in progress. There are more vector expressions to fix.
> Vectorization: IF statement, Comparisons, and more do not handle NULLs
> correctly
>
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.01.patch
>
>
>
> Many vector expression classes are missing guards around setting noNulls
> among other things.
> {code:java}
> // Carefully update noNulls...
> if (outputColVector.noNulls) {
> outputColVector.noNulls = inputColVector.noNulls;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-18622:
Summary: Vectorization: IF Statements, Comparisons, and more do not handle
NULLs correctly (was: Vectorization: IF statement, Comparisons, and more do
not handle NULLs correctly)
> Vectorization: IF Statements, Comparisons, and more do not handle NULLs
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.01.patch
>
>
>
> Many vector expression classes are missing guards around setting noNulls
> among other things.
> {code:java}
> // Carefully update noNulls...
> if (outputColVector.noNulls) {
> outputColVector.noNulls = inputColVector.noNulls;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[
https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352174#comment-16352174
]
Hive QA commented on HIVE-18467:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909188/HIVE-18467.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 77 failed/errored test(s), 12973 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_00_nonpart_empty]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_01_nonpart]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_02_00_part_empty]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_02_part]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_03_nonpart_over_compat]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_04_all_part]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_04_evolved_parts]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_05_some_part]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_06_one_part]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_07_all_part_over_nonoverlap]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_08_nonpart_rename]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_09_part_spec_nonoverlap]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_10_external_managed]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_11_managed_external]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_12_external_location]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_13_managed_location]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_14_managed_location_over_existing]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_15_external_part]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_16_part_external]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_17_part_managed]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_18_part_external]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_19_00_part_external_location]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_19_part_external_location]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_20_part_managed_location]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_22_import_exist_authsuccess]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_23_import_part_authsuccess]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_24_import_nonexist_authsuccess]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_hidden_files]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_01_nonpart_over_loaded]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_02_all_part_over_overlap]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_03_nonpart_noncompat_colschema]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_04_nonpart_noncompat_colnumbe
[jira] [Commented] (HIVE-18545) Add UDF to parse complex types from json
[ https://issues.apache.org/jira/browse/HIVE-18545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352179#comment-16352179 ] Zoltan Haindrich commented on HIVE-18545: - test failures are not related [~ashutoshc] Could you please take a look? > Add UDF to parse complex types from json > > > Key: HIVE-18545 > URL: https://issues.apache.org/jira/browse/HIVE-18545 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18545.02.patch, HIVE-18545.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18553) VectorizedParquetReader fails after adding a new column to table
[
https://issues.apache.org/jira/browse/HIVE-18553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352196#comment-16352196
]
Hive QA commented on HIVE-18553:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} ql: The patch generated 0 new + 126 unchanged - 11
fixed = 126 total (was 137) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0a328f0 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9017/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> VectorizedParquetReader fails after adding a new column to table
>
>
> Key: HIVE-18553
> URL: https://issues.apache.org/jira/browse/HIVE-18553
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
>Priority: Major
> Attachments: HIVE-18553.2.patch, HIVE-18553.3.patch,
> HIVE-18553.4.patch, HIVE-18553.patch
>
>
> VectorizedParquetReader throws an exception when trying to reading from a
> parquet table on which new columns are added. Steps to reproduce below:
> {code}
> 0: jdbc:hive2://localhost:1/default> desc test_p;
> +---++--+
> | col_name | data_type | comment |
> +---++--+
> | t1| tinyint| |
> | t2| tinyint| |
> | i1| int| |
> | i2| int| |
> +---++--+
> 0: jdbc:hive2://localhost:1/default> set hive.fetch.task.conversion=none;
> 0: jdbc:hive2://localhost:1/default> set
> hive.vectorized.execution.enabled=true;
> 0: jdbc:hive2://localhost:1/default> alter table test_p add columns (ts
> timestamp);
> 0: jdbc:hive2://localhost:1/default> select * from test_p;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)
> {code}
> Following exception is seen in the logs
> {code}
> Caused by: java.lang.IllegalArgumentException: [ts] BINARY is not in the
> store: [[i1] INT32, [i2] INT32, [t1] INT32, [t2] INT32] 3
> at
> org.apache.parquet.hadoop.ColumnChunkPageReadSto
[jira] [Assigned] (HIVE-18623) Hive throws an exception "Renames across Mount points not supported" when running in a federated cluster
[ https://issues.apache.org/jira/browse/HIVE-18623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yangfang reassigned HIVE-18623: --- > Hive throws an exception "Renames across Mount points not supported" when > running in a federated cluster > > > Key: HIVE-18623 > URL: https://issues.apache.org/jira/browse/HIVE-18623 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.3.2, 2.3.1, 2.3.0, 2.2.0 > Environment: hadoop 2.7.5, HDFS Federation enabled > hive 3.0.0 >Reporter: yangfang >Assignee: yangfang >Priority: Major > > > I run a sql query in in a federated cluster and I have two namespaces: > nameservice and nameservice1. I set > hive.exec.stagingdir=/nameservice1/hive_tmp in hive-site.xml and my data > tables are located in the directory of nameservice, then I got the exception > as below: > hive> create external table test_par6(id int,name string) partitioned by(p > int); > OK > Time taken: 1.527 seconds > hive> insert into table test_par6 partition(p = 1) values(1,'Jack'); > Moving data to directory > viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-1 > Loading data to table default.test_par6 partition (p=1) > Failed with exception java.io.IOException: Renames across Mount points not > supported > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Renames across > Mount points not supported > MapReduce Jobs Launched: > Stage-Stage-1: Map: 1 Cumulative CPU: 2.08 sec HDFS Read: 3930 HDFS Write: 7 > SUCCESS > Total MapReduce CPU Time Spent: 2 seconds 80 msec -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18623) Hive throws an exception "Renames across Mount points not supported" when running in a federated cluster
[ https://issues.apache.org/jira/browse/HIVE-18623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yangfang updated HIVE-18623: Attachment: HIVE-18623.1.patch > Hive throws an exception "Renames across Mount points not supported" when > running in a federated cluster > > > Key: HIVE-18623 > URL: https://issues.apache.org/jira/browse/HIVE-18623 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.2.0, 2.3.0, 2.3.1, 2.3.2 > Environment: hadoop 2.7.5, HDFS Federation enabled > hive 3.0.0 >Reporter: yangfang >Assignee: yangfang >Priority: Major > Attachments: HIVE-18623.1.patch > > > > I run a sql query in in a federated cluster and I have two namespaces: > nameservice and nameservice1. I set > hive.exec.stagingdir=/nameservice1/hive_tmp in hive-site.xml and my data > tables are located in the directory of nameservice, then I got the exception > as below: > hive> create external table test_par6(id int,name string) partitioned by(p > int); > OK > Time taken: 1.527 seconds > hive> insert into table test_par6 partition(p = 1) values(1,'Jack'); > Moving data to directory > viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-1 > Loading data to table default.test_par6 partition (p=1) > Failed with exception java.io.IOException: Renames across Mount points not > supported > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Renames across > Mount points not supported > MapReduce Jobs Launched: > Stage-Stage-1: Map: 1 Cumulative CPU: 2.08 sec HDFS Read: 3930 HDFS Write: 7 > SUCCESS > Total MapReduce CPU Time Spent: 2 seconds 80 msec -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18553) VectorizedParquetReader fails after adding a new column to table
[
https://issues.apache.org/jira/browse/HIVE-18553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352225#comment-16352225
]
Hive QA commented on HIVE-18553:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909194/HIVE-18553.4.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 12971 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9017/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9017/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9017/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909194 - PreCommit-HIVE-Build
> VectorizedParquetReader fails after adding a new column to table
>
>
> Key: HIVE-18553
> URL: https://issues.apache.org/jira/browse/HIVE-18553
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
>Priority: Major
> Attachments: HIVE-18553.2.patch, HIVE-18553.3.patch,
> HIVE-18553.4.patch, HIVE-18553.patch
>
>
> VectorizedParquetReader throws an exception when trying to reading from a
> parquet table on which new columns are added. Steps to reproduce below:
> {code}
> 0: jdbc:hive2://localhost:1/default> desc test_p;
> +---++--+
> | col_name | data_type | comment |
> +---++--+
> | t1| tinyint| |
> | t2| tinyint| |
> | i1| int| |
> | i2| int| |
> +---++--+
> 0: jdbc:hive2://localhost:1/default> set hive.fetch.task.conversion=none;
> 0: jdbc:hive2://localhost:1/default> set
> hive.vectorized.execution.enabled=true;
> 0: jdbc:hive2://localhost:1/default> alter table test_p add columns (ts
> timestamp);
> 0: jdbc:hive2://localhost:1/default> select * fr
[jira] [Updated] (HIVE-18623) Hive throws an exception "Renames across Mount points not supported" when running in a federated cluster
[
https://issues.apache.org/jira/browse/HIVE-18623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yangfang updated HIVE-18623:
Status: Patch Available (was: Open)
Hive should move the execution result from the source path to the distination
path at the end of the execution, In my test, the source path is
viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-1
and the distination path is viewfs://nsX/nameservice/hive/test_par6/p=1/ ,
Moving files from source path to distination path are acrossing mount points.
HDFS use rename to move files , but it does not suport renames across mount
points, so it throws an exception.
The exception comes from the following code:
private static Path mvFile(HiveConf conf, FileSystem sourceFs, Path
sourcePath, FileSystem destFs, Path destDirPath,
boolean isSrcLocal, boolean isOverwrite, boolean isRenameAllowed) throws
IOException {
for (int counter = 1; destFs.exists(destFilePath); counter++) {
if (isOverwrite) {
destFs.delete(destFilePath, false);
break;
}
destFilePath = new Path(destDirPath, name + (Utilities.COPY_KEYWORD + counter)
+ (!type.isEmpty() ? "." + type : ""));
}
if (isRenameAllowed) {
destFs.rename(sourcePath, destFilePath);
}else if (isSrcLocal) {
destFs.copyFromLocalFile(sourcePath, destFilePath);
} else {
FileUtils.copy(sourceFs, sourcePath, destFs, destFilePath,
true, // delete source
false, // overwrite destination
conf);
}
}
I think we should make isRenameAllowed false because across mount points can
not use rename, we should use copy In this case, so I have changed needToCopy()
method , if I found source path and destination path are acrossing mount point
, needToCopy() return false.Now isRenameAllowed is false , Hive will use
FileUtils.copy() method for moving files.
> Hive throws an exception "Renames across Mount points not supported" when
> running in a federated cluster
>
>
> Key: HIVE-18623
> URL: https://issues.apache.org/jira/browse/HIVE-18623
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.3.2, 2.3.1, 2.3.0, 2.2.0
> Environment: hadoop 2.7.5, HDFS Federation enabled
> hive 3.0.0
>Reporter: yangfang
>Assignee: yangfang
>Priority: Major
> Attachments: HIVE-18623.1.patch
>
>
>
> I run a sql query in in a federated cluster and I have two namespaces:
> nameservice and nameservice1. I set
> hive.exec.stagingdir=/nameservice1/hive_tmp in hive-site.xml and my data
> tables are located in the directory of nameservice, then I got the exception
> as below:
> hive> create external table test_par6(id int,name string) partitioned by(p
> int);
> OK
> Time taken: 1.527 seconds
> hive> insert into table test_par6 partition(p = 1) values(1,'Jack');
> Moving data to directory
> viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-1
> Loading data to table default.test_par6 partition (p=1)
> Failed with exception java.io.IOException: Renames across Mount points not
> supported
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Renames across
> Mount points not supported
> MapReduce Jobs Launched:
> Stage-Stage-1: Map: 1 Cumulative CPU: 2.08 sec HDFS Read: 3930 HDFS Write: 7
> SUCCESS
> Total MapReduce CPU Time Spent: 2 seconds 80 msec
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352230#comment-16352230
]
Hive QA commented on HIVE-18622:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909198/HIVE-18622.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9018/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9018/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9018/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-02-05 10:43:24.717
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9018/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-02-05 10:43:24.720
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 0a328f0 HIVE-18546: Remove unnecessary code introduced in
HIVE-14498 (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 0a328f0 HIVE-18546: Remove unnecessary code introduced in
HIVE-14498 (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-02-05 10:43:28.218
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9018/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:51: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:67: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:129: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:1081: space before tab in indent.
setNodeIssue("Not vectorizedVertexNum = 3");
/data/hiveptest/working/scratch/build.patch:1082: space before tab in indent.
return false;
warning: squelched 6 whitespace errors
warning: 11 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: protoc version: 250, detected platform: linux/amd64
protoc-jar: executing: [/tmp/protoc6090506525793380960.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logger (DataNucleus.General).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer : Classpath
>> /usr/share/maven/boot/plexus-classworlds-2.x.jar
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDatabase
ENHANCED (Persistable) : org
[jira] [Comment Edited] (HIVE-18623) Hive throws an exception "Renames across Mount points not supported" when running in a federated cluster
[
https://issues.apache.org/jira/browse/HIVE-18623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352229#comment-16352229
]
yangfang edited comment on HIVE-18623 at 2/5/18 10:47 AM:
--
Hive should move the execution result from the source path to the distination
path at the end of the execution, In my test, the source path is
viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-1
and the distination path is viewfs://nsX/nameservice/hive/test_par6/p=1/ ,
Moving files from source path to distination path are acrossing mount points.
HDFS use rename to move files , but it does not support renames across mount
points, so it throws an exception.
The exception comes from the following code:
private static Path mvFile(HiveConf conf, FileSystem sourceFs, Path
sourcePath, FileSystem destFs, Path destDirPath,
boolean isSrcLocal, boolean isOverwrite, boolean isRenameAllowed) throws
IOException {
for (int counter = 1; destFs.exists(destFilePath); counter++) {
if (isOverwrite)
{ destFs.delete(destFilePath, false); break; }
destFilePath = new Path(destDirPath, name + (Utilities.COPY_KEYWORD + counter)
+ (!type.isEmpty() ? "." + type : ""));
}
if (isRenameAllowed)
{ destFs.rename(sourcePath, destFilePath); }
else if (isSrcLocal)
{ destFs.copyFromLocalFile(sourcePath, destFilePath); }
else
{ FileUtils.copy(sourceFs, sourcePath, destFs, destFilePath, true, // delete
source false, // overwrite destination conf); }
}
I think we should make isRenameAllowed false because across mount points can
not use rename, we should use copy In this case, so I have changed needToCopy()
method , if I found source path and destination path are acrossing mount point
, needToCopy() return false.Now isRenameAllowed is false , Hive will use
FileUtils.copy() method for moving files.
was (Author: yangfang):
Hive should move the execution result from the source path to the distination
path at the end of the execution, In my test, the source path is
viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-1
and the distination path is viewfs://nsX/nameservice/hive/test_par6/p=1/ ,
Moving files from source path to distination path are acrossing mount points.
HDFS use rename to move files , but it does not suport renames across mount
points, so it throws an exception.
The exception comes from the following code:
private static Path mvFile(HiveConf conf, FileSystem sourceFs, Path
sourcePath, FileSystem destFs, Path destDirPath,
boolean isSrcLocal, boolean isOverwrite, boolean isRenameAllowed) throws
IOException {
for (int counter = 1; destFs.exists(destFilePath); counter++) {
if (isOverwrite) {
destFs.delete(destFilePath, false);
break;
}
destFilePath = new Path(destDirPath, name + (Utilities.COPY_KEYWORD + counter)
+ (!type.isEmpty() ? "." + type : ""));
}
if (isRenameAllowed) {
destFs.rename(sourcePath, destFilePath);
}else if (isSrcLocal) {
destFs.copyFromLocalFile(sourcePath, destFilePath);
} else {
FileUtils.copy(sourceFs, sourcePath, destFs, destFilePath,
true, // delete source
false, // overwrite destination
conf);
}
}
I think we should make isRenameAllowed false because across mount points can
not use rename, we should use copy In this case, so I have changed needToCopy()
method , if I found source path and destination path are acrossing mount point
, needToCopy() return false.Now isRenameAllowed is false , Hive will use
FileUtils.copy() method for moving files.
> Hive throws an exception "Renames across Mount points not supported" when
> running in a federated cluster
>
>
> Key: HIVE-18623
> URL: https://issues.apache.org/jira/browse/HIVE-18623
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0, 2.3.0, 2.3.1, 2.3.2
> Environment: hadoop 2.7.5, HDFS Federation enabled
> hive 3.0.0
>Reporter: yangfang
>Assignee: yangfang
>Priority: Major
> Attachments: HIVE-18623.1.patch
>
>
>
> I run a sql query in in a federated cluster and I have two namespaces:
> nameservice and nameservice1. I set
> hive.exec.stagingdir=/nameservice1/hive_tmp in hive-site.xml and my data
> tables are located in the directory of nameservice, then I got the exception
> as below:
> hive> create external table test_pa
[jira] [Commented] (HIVE-18623) Hive throws an exception "Renames across Mount points not supported" when running in a federated cluster
[
https://issues.apache.org/jira/browse/HIVE-18623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352257#comment-16352257
]
Hive QA commented on HIVE-18623:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} common: The patch generated 4 new + 28 unchanged - 0
fixed = 32 total (was 28) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0a328f0 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9019/yetus/diff-checkstyle-common.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9019/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive throws an exception "Renames across Mount points not supported" when
> running in a federated cluster
>
>
> Key: HIVE-18623
> URL: https://issues.apache.org/jira/browse/HIVE-18623
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0, 2.3.0, 2.3.1, 2.3.2
> Environment: hadoop 2.7.5, HDFS Federation enabled
> hive 3.0.0
>Reporter: yangfang
>Assignee: yangfang
>Priority: Major
> Attachments: HIVE-18623.1.patch
>
>
>
> I run a sql query in in a federated cluster and I have two namespaces:
> nameservice and nameservice1. I set
> hive.exec.stagingdir=/nameservice1/hive_tmp in hive-site.xml and my data
> tables are located in the directory of nameservice, then I got the exception
> as below:
> hive> create external table test_par6(id int,name string) partitioned by(p
> int);
> OK
> Time taken: 1.527 seconds
> hive> insert into table test_par6 partition(p = 1) values(1,'Jack');
> Moving data to directory
> viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-1
> Loading data to table default.test_par6 partition (p=1)
> Failed with exception java.io.IOException: Renames across Mount points not
> supported
> FAIL
[jira] [Commented] (HIVE-18623) Hive throws an exception "Renames across Mount points not supported" when running in a federated cluster
[
https://issues.apache.org/jira/browse/HIVE-18623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352297#comment-16352297
]
Hive QA commented on HIVE-18623:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909208/HIVE-18623.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 31 failed/errored test(s), 12970 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[insert_overwrite_directory2]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[root_dir_external_table]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[schemeAuthority2]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[schemeAuthority]
(batchId=180)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[index_bitmap3]
(batchId=93)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[root_dir_external_table]
(batchId=92)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.TestHiveMetaTool.testListFSRoot (batchId=225)
org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet
(batchId=215)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9019/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9019/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9019/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 31 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909208 - PreCommit-HIVE-Build
> Hive throws an exception "Renames across Mount points not supported" when
> running in a federated cluster
>
>
> Key: HIVE-18623
> URL: https://issues.apache.org/jira/browse/HIVE-18623
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0, 2.3.0, 2.3.1, 2.3.2
> Environment: hadoop 2.7.5, HDFS Federation enabled
> hive 3.0.0
>Reporter: yangfang
>Assignee: yangfang
>Priority: Major
> Attachments: HIVE-18623.1.patch
>
>
>
> I run a sql query in in a federated cluster and I have two namespaces:
> nameservice and nameservice1. I set
> hive.exec.stagingdir=/nameservice1/hive_tmp in hive-site.xml and my data
> tables are located in the directory of nameservice, the
[jira] [Updated] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18238:
Attachment: HIVE-18238.04.patch
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch, HIVE-18238.04.patch, HIVE-18238.04wip01.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17991) Remove CommandNeedRetryException
[
https://issues.apache.org/jira/browse/HIVE-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17991:
Attachment: HIVE-17991.02.patch
> Remove CommandNeedRetryException
>
>
> Key: HIVE-17991
> URL: https://issues.apache.org/jira/browse/HIVE-17991
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-17991.01.patch, HIVE-17991.02.patch
>
>
> * only really used in clidriver (via an infinite loop)
> [here|https://github.com/apache/hive/blob/master/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L307]
> * only 1 thing depends on the value of trycount; I think we can remove it
> [GlobalLimitOptimizer|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GlobalLimitOptimizer.java#L78]
> * there are a few loops which have pre-set trycount to 1
> [example|https://github.com/apache/hive/blob/master/hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java#L497]
> * other catches are exception repackagings
> I think it would be better to remove it.
> for more info:
> {code}
> git grep -C10 'catch.*CommandNeedRetryException'
> git grep -i gettrycount
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352372#comment-16352372
]
Adam Szita commented on HIVE-15995:
---
[~michal.ferlinski], [~pxiong] any update on this please? Do I understand
correctly that this patch is ready to go in master branch?
> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Michal Ferlinski
>Assignee: Michal Ferlinski
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can
> alter the 'avro.schema.url' property to update table schema to the next
> version. Updating properties however doesn't affect column list stored in
> metastore DB so the table is not in the newest version when returned from
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution.
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
> PARTITIONED BY (dt string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION
> '/tmp/schema-evolution/user_test1'
> TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' =
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18422) Vectorized input format should not be used when vectorized input format is excluded and row.serde is enabled
[
https://issues.apache.org/jira/browse/HIVE-18422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352378#comment-16352378
]
Zoltan Haindrich commented on HIVE-18422:
-
[~vihangk1] it seems to me that the q.out comitted with this patch have not
matched the actual "master" q.out style - there are some minor things which
make it fail :)
https://builds.apache.org/job/PreCommit-HIVE-Build/9019/testReport/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_vectorization_input_format_excludes_/
I think the diffs are all right...just need an addendum commit...
{code}
-Dtest=TestMiniLlapLocalCliDriver#testCliDriver[vectorization_input_format_excludes]
-DskipSparkTests
-pl itests/qtest
{code}
> Vectorized input format should not be used when vectorized input format is
> excluded and row.serde is enabled
>
>
> Key: HIVE-18422
> URL: https://issues.apache.org/jira/browse/HIVE-18422
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-18422.01.patch, HIVE-18422.02.patch
>
>
> HIVE-17534 introduced a config which gives a capability to exclude certain
> inputformat from vectorized execution without affecting other input formats.
> If an input format is excluded and row.serde is enabled at the same time,
> vectorizer still sets the {{useVectorizedInputFormat}} to true which causes
> Vectorized readers to be used in row.serde mode.
> In order to reproduce:
> {noformat}
> set hive.fetch.task.conversion=none;
> set hive.vectorized.use.row.serde.deserialize=true;
> set hive.vectorized.use.vector.serde.deserialize=true;
> set hive.vectorized.execution.enabled=true;
> set hive.vectorized.execution.reduce.enabled=true;
> set hive.vectorized.row.serde.inputformat.excludes=;
> -- SORT_QUERY_RESULTS
> -- exclude MapredParquetInputFormat from vectorization, this should cause
> mapwork vectorization to be disabled
> set
> hive.vectorized.input.format.excludes=org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat;
> set hive.vectorized.use.vectorized.input.format=true;
> create table orcTbl (t1 tinyint, t2 tinyint)
> stored as orc;
> insert into orcTbl values (54, 9), (-104, 25), (-112, 24);
> explain vectorization select t1, t2, (t1+t2) from orcTbl where (t1+t2) > 10;
> select t1, t2, (t1+t2) from orcTbl where (t1+t2) > 10;
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16805) Utilities isEmptyPath Logging Too Chatty and Uses Bad Format
[
https://issues.apache.org/jira/browse/HIVE-16805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16805:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the patch [~belugabehr]!
> Utilities isEmptyPath Logging Too Chatty and Uses Bad Format
>
>
> Key: HIVE-16805
> URL: https://issues.apache.org/jira/browse/HIVE-16805
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 2.1.1, 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16805.1.patch, HIVE-16805.2.patch,
> HIVE-16805.3.patch
>
>
> {code:title=org.apache.hadoop.hive.ql.exec.Utilities}
> public static boolean isEmptyPath(JobConf job, Path dirPath, Context ctx)
> throws Exception {
> if (ctx != null) {
> ContentSummary cs = ctx.getCS(dirPath);
> if (cs != null) {
> LOG.info("Content Summary " + dirPath + "length: " + cs.getLength() +
> " num files: "
> + cs.getFileCount() + " num directories: " +
> cs.getDirectoryCount());
> return (cs.getLength() == 0 && cs.getFileCount() == 0 &&
> cs.getDirectoryCount() <= 1);
> } else {
> LOG.info("Content Summary not cached for " + dirPath);
> }
> }
> return isEmptyPath(job, dirPath);
> }
> {code}
> # This is too chatty. Logging for caching is not necessary for INFO logging.
> Move to DEBUG
> # Use Log4J's {} format
> # Make the messages more symmetrical - The success message doesn't include
> the word "cache" in it at all
> # Missing a space between the path and "length: " so the path is concatenated
> with the string "length:"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352411#comment-16352411
]
Hive QA commented on HIVE-18238:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
51s{color} | {color:red} ql: The patch generated 12 new + 1869 unchanged - 9
fixed = 1881 total (was 1878) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} cli: The patch generated 1 new + 31 unchanged - 1
fixed = 32 total (was 32) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hcatalog/core: The patch generated 1 new + 33
unchanged - 3 fixed = 34 total (was 36) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} hcatalog/hcatalog-pig-adapter: The patch generated 0
new + 18 unchanged - 1 fixed = 18 total (was 19) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0a328f0 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9020/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9020/yetus/diff-checkstyle-cli.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9020/yetus/diff-checkstyle-hcatalog_core.txt
|
| modules | C: ql cli hcatalog/core hcatalog/hcatalog-pig-adapter
itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9020/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Z
[jira] [Updated] (HIVE-18373) Make it easier to search for column name in a table
[
https://issues.apache.org/jira/browse/HIVE-18373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhudeep Petwal updated HIVE-18373:
Attachment: HIVE-18373.1.patch
> Make it easier to search for column name in a table
> ---
>
> Key: HIVE-18373
> URL: https://issues.apache.org/jira/browse/HIVE-18373
> Project: Hive
> Issue Type: New Feature
>Reporter: Siddhant Saraf
>Assignee: Madhudeep Petwal
>Priority: Minor
> Attachments: HIVE-18373.1.patch
>
>
> Within a database, to filter for tables with the string 'abc' in its name, I
> can use something like:
> {code:java}
> hive> use my_database;
> hive> show tables '*abc*';
> {code}
> It would be great if I can do something similar to search within the list of
> columns in a table.
> I have a table with around 3200 columns. Searching for the column of interest
> is an onerous task after doing a {{describe}} on it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-18622:
Attachment: HIVE-18622.02.patch
> Vectorization: IF Statements, Comparisons, and more do not handle NULLs
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.01.patch, HIVE-18622.02.patch
>
>
>
> Many vector expression classes are missing guards around setting noNulls
> among other things.
> {code:java}
> // Carefully update noNulls...
> if (outputColVector.noNulls) {
> outputColVector.noNulls = inputColVector.noNulls;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-18622:
Status: In Progress (was: Patch Available)
> Vectorization: IF Statements, Comparisons, and more do not handle NULLs
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.01.patch, HIVE-18622.02.patch
>
>
>
> Many vector expression classes are missing guards around setting noNulls
> among other things.
> {code:java}
> // Carefully update noNulls...
> if (outputColVector.noNulls) {
> outputColVector.noNulls = inputColVector.noNulls;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18373) Make it easier to search for column name in a table
[
https://issues.apache.org/jira/browse/HIVE-18373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhudeep Petwal updated HIVE-18373:
Status: Patch Available (was: Open)
> Make it easier to search for column name in a table
> ---
>
> Key: HIVE-18373
> URL: https://issues.apache.org/jira/browse/HIVE-18373
> Project: Hive
> Issue Type: New Feature
>Reporter: Siddhant Saraf
>Assignee: Madhudeep Petwal
>Priority: Minor
> Attachments: HIVE-18373.1.patch
>
>
> Within a database, to filter for tables with the string 'abc' in its name, I
> can use something like:
> {code:java}
> hive> use my_database;
> hive> show tables '*abc*';
> {code}
> It would be great if I can do something similar to search within the list of
> columns in a table.
> I have a table with around 3200 columns. Searching for the column of interest
> is an onerous task after doing a {{describe}} on it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-18622:
Status: Patch Available (was: In Progress)
> Vectorization: IF Statements, Comparisons, and more do not handle NULLs
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.01.patch, HIVE-18622.02.patch
>
>
>
> Many vector expression classes are missing guards around setting noNulls
> among other things.
> {code:java}
> // Carefully update noNulls...
> if (outputColVector.noNulls) {
> outputColVector.noNulls = inputColVector.noNulls;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18373) Make it easier to search for column name in a table
[
https://issues.apache.org/jira/browse/HIVE-18373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352452#comment-16352452
]
Madhudeep Petwal commented on HIVE-18373:
-
Patch submitted. Who will review this [~thejas] ?
> Make it easier to search for column name in a table
> ---
>
> Key: HIVE-18373
> URL: https://issues.apache.org/jira/browse/HIVE-18373
> Project: Hive
> Issue Type: New Feature
>Reporter: Siddhant Saraf
>Assignee: Madhudeep Petwal
>Priority: Minor
> Attachments: HIVE-18373.1.patch
>
>
> Within a database, to filter for tables with the string 'abc' in its name, I
> can use something like:
> {code:java}
> hive> use my_database;
> hive> show tables '*abc*';
> {code}
> It would be great if I can do something similar to search within the list of
> columns in a table.
> I have a table with around 3200 columns. Searching for the column of interest
> is an onerous task after doing a {{describe}} on it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18578) Some class has missed the ASF header
[ https://issues.apache.org/jira/browse/HIVE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18578: Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you [~txhsj] for fixing this! > Some class has missed the ASF header > > > Key: HIVE-18578 > URL: https://issues.apache.org/jira/browse/HIVE-18578 > Project: Hive > Issue Type: Bug >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-18578.1.patch > > > Some class has missed the ASF header -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18612) Build subprocesses under Yetus in Ptest use 1.7 jre instead of 1.8
[
https://issues.apache.org/jira/browse/HIVE-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352478#comment-16352478
]
Peter Vary commented on HIVE-18612:
---
+1
> Build subprocesses under Yetus in Ptest use 1.7 jre instead of 1.8
> --
>
> Key: HIVE-18612
> URL: https://issues.apache.org/jira/browse/HIVE-18612
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-18612.0.patch
>
>
> As per this jira comment made by Yetus maven plugins that want to use
> {{java}} executable are seeing a 1.7 java binary. In this particular case
> Yetus sets JAVA_HOME to a 1.8 JDK installation, and thus maven uses that, but
> any subsequent {{java}} executes will use the JRE which they see on PATH.
> This should be fixed by adding the proper java/bin (that of JAVA_HOME
> setting) to PATH.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352481#comment-16352481
]
Hive QA commented on HIVE-18238:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909219/HIVE-18238.04.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 12971 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hadoop.hive.ql.lockmgr.TestDummyTxnManager.testSingleReadTable
(batchId=295)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase
(batchId=242)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9020/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9020/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9020/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 29 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909219 - PreCommit-HIVE-Build
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch, HIVE-18238.04.patch, HIVE-18238.04wip01.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnaly
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352490#comment-16352490
]
Hive QA commented on HIVE-15995:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854469/HIVE-15995.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9021/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9021/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9021/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-02-05 14:49:05.454
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9021/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-02-05 14:49:05.486
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
0a328f0..a128c0e master -> origin/master
+ git reset --hard HEAD
HEAD is now at 0a328f0 HIVE-18546: Remove unnecessary code introduced in
HIVE-14498 (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at a128c0e HIVE-18578: Some class has missed the ASF header (Saijin
Huang via Zoltan Haindrich)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-02-05 14:49:15.271
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9021/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g: does not
exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java:
does not exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g:1145
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g'
cleanly.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java:121
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java'
cleanly.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java:229
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java'
cleanly.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g:1145
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g'
cleanly.
error: patch failed:
ql/src/java/org/apa
[jira] [Commented] (HIVE-18599) Transactions: Fix CTAS on Micromanaged tables
[
https://issues.apache.org/jira/browse/HIVE-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352492#comment-16352492
]
Zoltan Haindrich commented on HIVE-18599:
-
[~steveyeom2017] mm_cttas was failing on master; because there was no q.out for
the standard cli version ; and the llap version have missed some posthook
change...
I've added an addendum to your patch.
> Transactions: Fix CTAS on Micromanaged tables
> -
>
> Key: HIVE-18599
> URL: https://issues.apache.org/jira/browse/HIVE-18599
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18599.01.patch, HIVE-18599.02.patch,
> HIVE-18599.03.patch, HIVE-18599.04.patch
>
>
> CTTAS on temporary micromanaged table does not write data.
> I.e., "SELECT * FROM ctas0_mm;" does not return any rows from the below
> script:
>
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set tez.grouping.min-size=1;
> set tez.grouping.max-size=2;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
> drop table intermediate;
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from
> src where key >= 100 order by key asc limit 2;
>
> drop table ctas0_mm;
> explain create temporary table ctas0_mm tblproperties
> ("transactional"="true", "transactional_properties"="insert_only") as select
> * from intermediate;
> create temporary table ctas0_mm tblproperties ("transactional"="true",
> "transactional_properties"="insert_only") as select * from intermediate;
>
> select * from ctas0_mm;
> drop table ctas0_mm;
> drop table intermediate;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18567) ObjectStore.getPartitionNamesNoTxn doesn't handle max param properly
[
https://issues.apache.org/jira/browse/HIVE-18567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352508#comment-16352508
]
Adam Szita commented on HIVE-18567:
---
[~spena], [~sershe] can you please let me know what you think?
Is {{checkLimitNumberOfPartitionsByFilter}} missing here on purpose, or am I
right to think that it is needed here for consistency with other methods?
> ObjectStore.getPartitionNamesNoTxn doesn't handle max param properly
>
>
> Key: HIVE-18567
> URL: https://issues.apache.org/jira/browse/HIVE-18567
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-18567.0.patch
>
>
> As per [this HMS API test
> case|https://github.com/apache/hive/commit/fa0a8d27d4149cc5cc2dbb49d8eb6b03f46bc279#diff-25c67d898000b53e623a6df9221aad5dR1044]
> listing partition names doesn't check tha max param against
> MetaStoreConf.LIMIT_PARTITION_REQUEST (as other methods do by
> checkLimitNumberOfPartitionsByFilter), and also behaves differently on max=0
> setting compared to other methods.
> We should bring this into consistency.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17991) Remove CommandNeedRetryException
[
https://issues.apache.org/jira/browse/HIVE-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352535#comment-16352535
]
Hive QA commented on HIVE-17991:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 2 new + 328 unchanged - 11
fixed = 330 total (was 339) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch service passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} cli: The patch generated 4 new + 40 unchanged - 5
fixed = 44 total (was 45) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} hcatalog/core: The patch generated 2 new + 55
unchanged - 8 fixed = 57 total (was 63) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hcatalog/hcatalog-pig-adapter: The patch generated 1
new + 251 unchanged - 15 fixed = 252 total (was 266) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch server-extensions passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch java-client passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} hcatalog/streaming: The patch generated 2 new + 342
unchanged - 20 fixed = 344 total (was 362) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 792
unchanged - 6 fixed = 792 total (was 798) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 3a67730 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9022/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.
[jira] [Commented] (HIVE-18567) ObjectStore.getPartitionNamesNoTxn doesn't handle max param properly
[ https://issues.apache.org/jira/browse/HIVE-18567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352577#comment-16352577 ] Sergio Peña commented on HIVE-18567: [~szita] There is a class 'PartitionIterable' which may fetch partition in batches to avoid fetching a large number of partitions from HMS. This iterable class depends on the full list of partition names that exist on the HMS server. If a max limit is applied to partition names, then this iterable class won't make sense. Also, fetching partition names is less memory costly than a full partition information. Perhaps this is the reason the partition max limit does not apply there. > ObjectStore.getPartitionNamesNoTxn doesn't handle max param properly > > > Key: HIVE-18567 > URL: https://issues.apache.org/jira/browse/HIVE-18567 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-18567.0.patch > > > As per [this HMS API test > case|https://github.com/apache/hive/commit/fa0a8d27d4149cc5cc2dbb49d8eb6b03f46bc279#diff-25c67d898000b53e623a6df9221aad5dR1044] > listing partition names doesn't check tha max param against > MetaStoreConf.LIMIT_PARTITION_REQUEST (as other methods do by > checkLimitNumberOfPartitionsByFilter), and also behaves differently on max=0 > setting compared to other methods. > We should bring this into consistency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17991) Remove CommandNeedRetryException
[
https://issues.apache.org/jira/browse/HIVE-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352583#comment-16352583
]
Hive QA commented on HIVE-17991:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909220/HIVE-17991.02.patch
{color:green}SUCCESS:{color} +1 due to 26 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 12970 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_filter]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=250)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9022/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9022/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9022/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909220 - PreCommit-HIVE-Build
> Remove CommandNeedRetryException
>
>
> Key: HIVE-17991
> URL: https://issues.apache.org/jira/browse/HIVE-17991
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-17991.01.patch, HIVE-17991.02.patch
>
>
> * only really used in clidriver (via an infinite loop)
> [here|https://github.com/apache/hive/blob/master/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L307]
> * only 1 thing depends on the value of trycount; I think we can remove it
> [GlobalLimitOptimizer|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GlobalLimitOptimizer.java#L78]
> * there are a few loops which have pre-set trycount to 1
> [example|https://github.com/apache/hive/blob/master/hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java#L497]
> * other catches are exception repackagings
> I think it would be better to remove it.
> for more info:
> {code}
> git grep -C10 'catch.*CommandNeedRetryException'
> git grep -i gettrycount
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18575: -- Component/s: Transactions > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18575.patch > > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18051) qfiles: dataset support
[
https://issues.apache.org/jira/browse/HIVE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352641#comment-16352641
]
Hive QA commented on HIVE-18051:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
44s{color} | {color:red} root: The patch generated 9 new + 170 unchanged - 9
fixed = 179 total (was 179) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/util: The patch generated 9 new + 165 unchanged
- 9 fixed = 174 total (was 174) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 3a67730 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9023/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9023/yetus/diff-checkstyle-itests_util.txt
|
| modules | C: ql . itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9023/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> qfiles: dataset support
> ---
>
> Key: HIVE-18051
> URL: https://issues.apache.org/jira/browse/HIVE-18051
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18051.01.patch, HIVE-18051.02.patch,
> HIVE-18051.03.patch, HIVE-18051.04.patch, HIVE-18051.05.patch,
> HIVE-18051.06.patch, HIVE-18051.07.patch, HIVE-18051.08.patch
>
>
> it would be great to have some kind of test dataset support; currently there
> is the {{q_test_init.sql}} which is quite large; and I'm often override it
> with an invalid string; because I write independent qtests most of the time -
> and the load of {{src}} and other tables are just a waste of time for me ;
> not to mention that the loading of those tables may also trigger breakpoints
> - which is a bit annoying.
> Most of the tests are "only" using the {{src}} table and possibly 2 others;
> however the main init script contains a bunch of
[jira] [Commented] (HIVE-18051) qfiles: dataset support
[
https://issues.apache.org/jira/browse/HIVE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352664#comment-16352664
]
Hive QA commented on HIVE-18051:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908952/HIVE-18051.08.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 223 failed/errored test(s), 12976 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_blobstore_to_blobstore]
(batchId=251)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[testdataset] (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[testdataset_2]
(batchId=16)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query10]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query11]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query12]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query13]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query15]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query17]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query18]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query19]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query20]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query21]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query22]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query23]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query24]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query25]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query26]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query27]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query28]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query29]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query2]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query30]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query31]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query32]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query33]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query34]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query35]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query36]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query37]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query38]
(batchId=250)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=250)
o
[jira] [Updated] (HIVE-18624) Parsing time is very high in certain complex expressions
[
https://issues.apache.org/jira/browse/HIVE-18624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amruth S updated HIVE-18624:
Description:
Explain of the same query takes
0.1 to 3 seconds in hive 2.1.0 &
10-15 min in hive 2.3.2 & latest master
Sample expression below
{code:java}
EXPLAIN
SELECT DISTINCT
IF(lower('a') <= lower('a')
,'a'
,IF(('a' IS NULL AND from_unixtime(UNIX_TIMESTAMP()) <= 'a')
,'a'
,IF(if('a' = 'a', TRUE, FALSE) = 1
,'a'
,IF(('a' = 1 and lower('a') NOT IN ('a', 'a')
and lower(if('a' = 'a','a','a')) <= lower('a'))
OR ('a' like 'a' OR 'a' like 'a')
OR 'a' in ('a','a')
,'a'
,IF(if(lower('a') in ('a', 'a') and 'a'='a', TRUE, FALSE) = 1
,'a'
,IF('a'='a' and unix_timestamp(if('a' = 'a',cast('a' as
string),coalesce('a',cast('a' as string),from_unixtime(unix_timestamp() <=
unix_timestamp(concat_ws('a',cast(lower('a') as string),'00:00:00')) + 9*3600
,'a'
,If(lower('a') <= lower('a')
and if(lower('a') in ('a', 'a') and 'a'<>'a', TRUE, FALSE) <> 1
,'a'
,IF('a'=1 AND 'a'=1
,'a'
,IF('a' = 1 and COALESCE(cast('a' as int),0) = 0
,'a'
,IF('a' = 'a'
,'a'
,If('a' = 'a' AND
lower('a')>lower(if(lower('a')<1830,'a',cast(date_add('a',1) as timestamp)))
,'a'
,IF('a' = 1
,IF('a' in ('a', 'a') and ((unix_timestamp('a')-unix_timestamp('a')) / 60) >
30 and 'a' = 1
,'a', 'a')
,IF(if('a' = 'a', FALSE, TRUE ) = 1 AND 'a' IS NULL
,'a'
,IF('a' = 1 and 'a'>0
, 'a'
,IF('a' = 1 AND 'a' ='a'
,'a'
,IF('a' is not null and 'a' is not null and 'a' > 'a'
,'a'
,IF('a' = 1
,'a'
,IF('a' = 'a'
,'a'
,If('a' = 1
,'a'
,IF('a' = 1
,'a'
,IF('a' = 1
,'a'
,IF('a' ='a' and 'a' ='a' and cast(unix_timestamp('a') as int) + 93600 <
cast(unix_timestamp() as int)
,'a'
,IF('a' = 'a'
,'a'
,IF('a' = 'a' and 'a' in ('a','a','a')
,'a'
,IF('a' = 'a'
,'a','a'))
)))
AS test_comp_exp
{code}
Taking a look at [^thread_dump] shows a very large function stack getting
created.
was:
Explain of the same query takes
0.1 to 3 seconds in hive 2.1.0 &
10-15 min in hive 2.3.2
Sample expression below
{code:java}
EXPLAIN
SELECT DISTINCT
IF(lower('a') <= lower('a')
,'a'
,IF(('a' IS NULL AND from_unixtime(UNIX_TIMESTAMP()) <= 'a')
,'a'
,IF(if('a' = 'a', TRUE, FALSE) = 1
,'a'
,IF(('a' = 1 and lower('a') NOT IN ('a', 'a')
and lower(if('a' = 'a','a','a')) <= lower('a'))
OR ('a' like 'a' OR 'a' like 'a')
OR 'a' in ('a','a')
,'a'
,IF(if(lower('a') in ('a', 'a') and 'a'='a', TRUE, FALSE) = 1
,'a'
,IF('a'='a' and unix_timestamp(if('a' = 'a',cast('a' as
string),coalesce('a',cast('a' as string),from_unixtime(unix_timestamp() <=
unix_timestamp(concat_ws('a',cast(lower('a') as string),'00:00:00')) + 9*3600
,'a'
,If(lower('a') <= lower('a')
and if(lower('a') in ('a', 'a') and 'a'<>'a', TRUE, FALSE) <> 1
,'a'
,IF('a'=1 AND 'a'=1
,'a'
,IF('a' = 1 and COALESCE(cast('a' as int),0) = 0
,'a'
,IF('a' = 'a'
,'a'
,If('a' = 'a' AND
lower('a')>lower(if(lower('a')<1830,'a',cast(date_add('a',1) as timestamp)))
,'a'
,IF('a' = 1
,IF('a' in ('a', 'a') and ((unix_timestamp('a')-unix_timestamp('a')) / 60) >
30 and 'a' = 1
,'a', 'a')
,IF(if('a' = 'a', FALSE, TRUE ) = 1 AND 'a' IS NULL
,'a'
,IF('a' = 1 and 'a'>0
, 'a'
,IF('a' = 1 AND 'a' ='a'
,'a'
,IF('a' is not null and 'a' is not null and 'a' > 'a'
,'a'
,IF('a' = 1
,'a'
,IF('a' = 'a'
,'a'
,If('a' = 1
,'a'
,IF('a' = 1
,'a'
,IF('a' = 1
,'a'
,IF('a' ='a' and 'a' ='a' and cast(unix_timestamp('a') as int) + 93600 <
cast(unix_timestamp() as int)
,'a'
,IF('a' = 'a'
,'a'
,IF('a' = 'a' and 'a' in ('a','a','a')
,'a'
,IF('a' = 'a'
,'a','a'))
)))
AS test_comp_exp
{code}
Taking a look at [^thread_dump] shows a very large function stack getting
created.
> Parsing time is very high in certain complex expressions
>
>
> Key: HIVE-18624
> URL: https://issues.apache.org/jira/browse/HIVE-18624
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.2
>Reporter: Amruth S
>Priority: Major
> Attachments: thread_dump
>
>
> Explain of the same query takes
> 0.1 to 3 seconds in hive 2.1.0 &
> 10-15 min in hive 2.3.2 & latest master
> Sample expression below
> {code:java}
> EXPLAIN
> SELECT DISTINCT
> IF(lower('a') <= lower('a')
> ,'a'
> ,IF(('a' IS NULL AND from_unixtime(UNIX_TIMESTAMP()) <= 'a')
> ,'a'
> ,IF(if('a' = 'a', TRUE, FALSE) = 1
> ,'a'
> ,IF(('a' = 1 and lower('a') NOT IN ('a', 'a')
>and lower(if('a' = 'a','a','a')) <= lower('a'))
> OR ('a' like 'a' OR 'a' like 'a')
> OR 'a' in ('a','a')
> ,'a'
> ,IF(if(lower('a')
[jira] [Commented] (HIVE-18511) Fix generated checkstyle errors
[
https://issues.apache.org/jira/browse/HIVE-18511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352680#comment-16352680
]
Hive QA commented on HIVE-18511:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} standalone-metastore: The patch generated 0 new + 6
unchanged - 19 fixed = 6 total (was 25) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 3a67730 |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9024/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix generated checkstyle errors
> ---
>
> Key: HIVE-18511
> URL: https://issues.apache.org/jira/browse/HIVE-18511
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-18511.2.patch, HIVE-18511.patch
>
>
> HIVE-18510 identified, that checkstyle was not running for test sources.
> After running checkstyle several errors are identified
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18580) Create tests to cover exchange partitions
[
https://issues.apache.org/jira/browse/HIVE-18580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-18580:
-
Attachment: HIVE-18580.1.patch
> Create tests to cover exchange partitions
> -
>
> Key: HIVE-18580
> URL: https://issues.apache.org/jira/browse/HIVE-18580
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-18580.1.patch
>
>
> The following methods of IMetaStoreClient are covered in this Jira:
> {code:java}
> - int Partition exchange_partition(Map, String, String,
> String, String)
> - List Partition exchange_partition(Map, String,
> String, String, String){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18580) Create tests to cover exchange partitions
[
https://issues.apache.org/jira/browse/HIVE-18580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-18580:
-
Status: Patch Available (was: Open)
> Create tests to cover exchange partitions
> -
>
> Key: HIVE-18580
> URL: https://issues.apache.org/jira/browse/HIVE-18580
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-18580.1.patch
>
>
> The following methods of IMetaStoreClient are covered in this Jira:
> {code:java}
> - int Partition exchange_partition(Map, String, String,
> String, String)
> - List Partition exchange_partition(Map, String,
> String, String, String){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18613) Extend JsonSerDe to support BINARY type
[
https://issues.apache.org/jira/browse/HIVE-18613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-18613:
---
Status: Patch Available (was: In Progress)
> Extend JsonSerDe to support BINARY type
> ---
>
> Key: HIVE-18613
> URL: https://issues.apache.org/jira/browse/HIVE-18613
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> It seems only type not supported currently, from the code:
> {code}
> case BINARY:
> throw new IOException("JsonSerDe does not support BINARY type");
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18613) Extend JsonSerDe to support BINARY type
[
https://issues.apache.org/jira/browse/HIVE-18613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-18613:
---
Attachment: HIVE-18613.patch
> Extend JsonSerDe to support BINARY type
> ---
>
> Key: HIVE-18613
> URL: https://issues.apache.org/jira/browse/HIVE-18613
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18613.patch
>
>
> It seems only type not supported currently, from the code:
> {code}
> case BINARY:
> throw new IOException("JsonSerDe does not support BINARY type");
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352725#comment-16352725
]
Vihang Karajgaonkar commented on HIVE-18421:
Copy pasting the test result from precommit job since its not posting it to the
Jira for some reason.
[Test
Result|https://builds.apache.org/job/PreCommit-HIVE-Build/9014/testReport/]
(312 failures / +285)
Most of these test failures are due to explain plan differences since the
{{hive.vectorized.use.checked.expressions}} is set to true by default in the
patch. We can either update all these q.out files or change the default value
of this config to false.
The tests TestColumnScalarOperationVectorExpressionEvaluation are failing
because they do not set the OutputTypeInfo in the test case. This will also get
resolved if we set the outputTypeInfo appropriately in the test template. I
think I will change the patch so that
{{OverflowUtils.accountForOverflow[Double|Long]}} will not throw an exception
if the outputTypeInfo is set to null since based on the tests, it doesn't look
like its always going to be true for all the expression. [~mmccline] do you
know if this understanding is correct?
I will update the patch which should fix most of these test failures.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18599) Transactions: Fix CTAS on Micromanaged tables
[
https://issues.apache.org/jira/browse/HIVE-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352732#comment-16352732
]
Steve Yeom commented on HIVE-18599:
---
Wow .. Thank you very much for your help, Zoltan.
Steve.
> Transactions: Fix CTAS on Micromanaged tables
> -
>
> Key: HIVE-18599
> URL: https://issues.apache.org/jira/browse/HIVE-18599
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18599.01.patch, HIVE-18599.02.patch,
> HIVE-18599.03.patch, HIVE-18599.04.patch
>
>
> CTTAS on temporary micromanaged table does not write data.
> I.e., "SELECT * FROM ctas0_mm;" does not return any rows from the below
> script:
>
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set tez.grouping.min-size=1;
> set tez.grouping.max-size=2;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
> drop table intermediate;
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from
> src where key >= 100 order by key asc limit 2;
>
> drop table ctas0_mm;
> explain create temporary table ctas0_mm tblproperties
> ("transactional"="true", "transactional_properties"="insert_only") as select
> * from intermediate;
> create temporary table ctas0_mm tblproperties ("transactional"="true",
> "transactional_properties"="insert_only") as select * from intermediate;
>
> select * from ctas0_mm;
> drop table ctas0_mm;
> drop table intermediate;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18588) Add 'checkin' profile that runs slower tests in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352745#comment-16352745 ] Alan Gates commented on HIVE-18588: --- With the groups I don't think we need the profile. With patch2 'mvn test' runs the unit tests, and doing 'mvn test -Dtest.groups=org.apache.hadoop.hive.annotation.MetastoreTest' runs them all. It seems easier than having groups and profiles. We should put comments in the DEV-README on how to run the various tests. > Add 'checkin' profile that runs slower tests in standalone-metastore > > > Key: HIVE-18588 > URL: https://issues.apache.org/jira/browse/HIVE-18588 > Project: Hive > Issue Type: Test > Components: Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-18588.2.patch, HIVE-18588.patch > > > Runtime for unit tests in standalone-metastore are now exceeding 25 minutes. > Ideally unit tests should finish within 2-3 minutes so users will run them > frequently. To solve this I propose to carve off many of the slower tests to > run in a new 'checkin' profile. This profile should be run before checkin > and by the ptest infrastructure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18128) Setting AcidUtils.setTransactionalTableScan in HiveInputFormat causes downstream errors
[ https://issues.apache.org/jira/browse/HIVE-18128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352746#comment-16352746 ] Eugene Koifman commented on HIVE-18128: --- fixed in HIVE-18575? > Setting AcidUtils.setTransactionalTableScan in HiveInputFormat causes > downstream errors > --- > > Key: HIVE-18128 > URL: https://issues.apache.org/jira/browse/HIVE-18128 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Priority: Major > Attachments: HIVE-18128.WIP.patch > > > This should really be set in addSplitsForGroup(). See attached patch for > details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18511) Fix generated checkstyle errors
[
https://issues.apache.org/jira/browse/HIVE-18511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352748#comment-16352748
]
Hive QA commented on HIVE-18511:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908308/HIVE-18511.2.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 12970 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9024/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9024/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9024/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908308 - PreCommit-HIVE-Build
> Fix generated checkstyle errors
> ---
>
> Key: HIVE-18511
> URL: https://issues.apache.org/jira/browse/HIVE-18511
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-18511.2.patch, HIVE-18511.patch
>
>
> HIVE-18510 identified, that checkstyle was not running for test sources.
> After running checkstyle several errors are identified
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul
[ https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-18347: - Attachment: HIVE-18347.5.patch > Allow dynamic lookup of Hive Metastores via Consul > -- > > Key: HIVE-18347 > URL: https://issues.apache.org/jira/browse/HIVE-18347 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch, > HIVE-18347.3.patch, HIVE-18347.4.patch, HIVE-18347.5.patch > > > In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos > as dynamic services for scalability and flexibility. > In this architecture, we would like to allow HiveServer2 to dynamically load > balance between Metastores (which may be scaled up and down or to different > nodes) for different requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul
[ https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352752#comment-16352752 ] Szehon Ho commented on HIVE-18347: -- Thanks a lot. Also I guess HIVE-18449 conflicts with this, so uploading another try to rebase it. > Allow dynamic lookup of Hive Metastores via Consul > -- > > Key: HIVE-18347 > URL: https://issues.apache.org/jira/browse/HIVE-18347 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch, > HIVE-18347.3.patch, HIVE-18347.4.patch, HIVE-18347.5.patch > > > In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos > as dynamic services for scalability and flexibility. > In this architecture, we would like to allow HiveServer2 to dynamically load > balance between Metastores (which may be scaled up and down or to different > nodes) for different requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18596) Synchronize value of hive.spark.client.connect.timeout across unit tests
[
https://issues.apache.org/jira/browse/HIVE-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18596:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Peter for the review.
> Synchronize value of hive.spark.client.connect.timeout across unit tests
>
>
> Key: HIVE-18596
> URL: https://issues.apache.org/jira/browse/HIVE-18596
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18596.1.patch
>
>
> {{hive.spark.client.connect.timeout}} is set to 30 seconds for
> {{TestMiniSparkOnYarnCliDriver}} but it left at the default value for all
> other tests. We should use the same value (30 seconds) for all other tests.
> We have seen flaky tests due to failure to establish the remote within the
> allotted timeout. This could be due to the fact that we run our tests in the
> cloud so maybe occasional network delays are more common.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352768#comment-16352768
]
Ashutosh Chauhan commented on HIVE-16605:
-
Are failures related ?
+1 otherwise.
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16605.1.patch, HIVE-16605.2.patch,
> HIVE-16605.3.patch, HIVE-16605.4.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352769#comment-16352769 ] Eugene Koifman commented on HIVE-18575: --- I left some comments on RB. Mostly, I think the change in TableScanDesc() doesn't do the right thing. Also, the test for full acid is sometimes called isAcidTable() and sometimes isFullAcidTable() - either is fine but I think it's confusing to have both. > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18575.patch > > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-18625:
-
Assignee: Andrew Sherman
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18373) Make it easier to search for column name in a table
[
https://issues.apache.org/jira/browse/HIVE-18373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352776#comment-16352776
]
Hive QA commented on HIVE-18373:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 26 new + 411 unchanged - 1
fixed = 437 total (was 412) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / e0eee6b |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9025/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9025/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9025/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make it easier to search for column name in a table
> ---
>
> Key: HIVE-18373
> URL: https://issues.apache.org/jira/browse/HIVE-18373
> Project: Hive
> Issue Type: New Feature
>Reporter: Siddhant Saraf
>Assignee: Madhudeep Petwal
>Priority: Minor
> Attachments: HIVE-18373.1.patch
>
>
> Within a database, to filter for tables with the string 'abc' in its name, I
> can use something like:
> {code:java}
> hive> use my_database;
> hive> show tables '*abc*';
> {code}
> It would be great if I can do something similar to search within the list of
> columns in a table.
> I have a table with around 3200 columns. Searching for the column of interest
> is an onerous task after doing a {{describe}} on it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17199) TestHiveMetaStoreStatsMerge: testStatsMerge
[ https://issues.apache.org/jira/browse/HIVE-17199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352780#comment-16352780 ] Janaki Lahorani commented on HIVE-17199: Thanks [~pgolash]. In that case, this can be closed. > TestHiveMetaStoreStatsMerge: testStatsMerge > --- > > Key: HIVE-17199 > URL: https://issues.apache.org/jira/browse/HIVE-17199 > Project: Hive > Issue Type: Sub-task >Reporter: Janaki Lahorani >Assignee: PRASHANT GOLASH >Priority: Major > > Stack: > org.apache.thrift.transport.TTransportException > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_aggr_stats_for(ThriftHiveMetastore.java:3724) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_aggr_stats_for(ThriftHiveMetastore.java:3711) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setPartitionColumnStatistics(HiveMetaStoreClient.java:1704) > at > org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge(TestHiveMetaStoreStatsMerge.java:187) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[ https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned HIVE-18626: - > Repl load "with" clause does not pass config to tasks > - > > Key: HIVE-18626 > URL: https://issues.apache.org/jira/browse/HIVE-18626 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > > The "with" clause in repl load suppose to pass custom hive config entries to > replication. However, the config is only effective in > BootstrapEventsIterator, but not the generated tasks (such as MoveTask, > DDLTask). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18588) Add 'checkin' profile that runs slower tests in standalone-metastore
[
https://issues.apache.org/jira/browse/HIVE-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352795#comment-16352795
]
Peter Vary commented on HIVE-18588:
---
Got it. That's good. Nit: Shall we remove the other commented out profile as
well?
The documentation is definitely needed. The tests will run only if annotated
with the {{MetastoreTest}}. I do not know about the others, but for me it would
take some time to find out what the problem is :)
+1 for the patch
> Add 'checkin' profile that runs slower tests in standalone-metastore
>
>
> Key: HIVE-18588
> URL: https://issues.apache.org/jira/browse/HIVE-18588
> Project: Hive
> Issue Type: Test
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-18588.2.patch, HIVE-18588.patch
>
>
> Runtime for unit tests in standalone-metastore are now exceeding 25 minutes.
> Ideally unit tests should finish within 2-3 minutes so users will run them
> frequently. To solve this I propose to carve off many of the slower tests to
> run in a new 'checkin' profile. This profile should be run before checkin
> and by the ptest infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18511) Fix generated checkstyle errors
[ https://issues.apache.org/jira/browse/HIVE-18511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352802#comment-16352802 ] Peter Vary commented on HIVE-18511: --- [~ychena]: Could you review this please? Thanks, Peter > Fix generated checkstyle errors > --- > > Key: HIVE-18511 > URL: https://issues.apache.org/jira/browse/HIVE-18511 > Project: Hive > Issue Type: Sub-task >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-18511.2.patch, HIVE-18511.patch > > > HIVE-18510 identified, that checkstyle was not running for test sources. > After running checkstyle several errors are identified -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18511) Fix generated checkstyle errors
[ https://issues.apache.org/jira/browse/HIVE-18511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352818#comment-16352818 ] Yongzhi Chen commented on HIVE-18511: - The checkstyle all passed and the change looks fine. +1 > Fix generated checkstyle errors > --- > > Key: HIVE-18511 > URL: https://issues.apache.org/jira/browse/HIVE-18511 > Project: Hive > Issue Type: Sub-task >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-18511.2.patch, HIVE-18511.patch > > > HIVE-18510 identified, that checkstyle was not running for test sources. > After running checkstyle several errors are identified -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-18622:
Attachment: (was: HIVE-18622.02.patch)
> Vectorization: IF Statements, Comparisons, and more do not handle NULLs
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.01.patch, HIVE-18622.02.patch
>
>
>
> Many vector expression classes are missing guards around setting noNulls
> among other things.
> {code:java}
> // Carefully update noNulls...
> if (outputColVector.noNulls) {
> outputColVector.noNulls = inputColVector.noNulls;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-18622:
Attachment: HIVE-18622.02.patch
> Vectorization: IF Statements, Comparisons, and more do not handle NULLs
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.01.patch, HIVE-18622.02.patch
>
>
>
> Many vector expression classes are missing guards around setting noNulls
> among other things.
> {code:java}
> // Carefully update noNulls...
> if (outputColVector.noNulls) {
> outputColVector.noNulls = inputColVector.noNulls;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18373) Make it easier to search for column name in a table
[
https://issues.apache.org/jira/browse/HIVE-18373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352843#comment-16352843
]
Hive QA commented on HIVE-18373:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909231/HIVE-18373.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 12971 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_columns2]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9025/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9025/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9025/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909231 - PreCommit-HIVE-Build
> Make it easier to search for column name in a table
> ---
>
> Key: HIVE-18373
> URL: https://issues.apache.org/jira/browse/HIVE-18373
> Project: Hive
> Issue Type: New Feature
>Reporter: Siddhant Saraf
>Assignee: Madhudeep Petwal
>Priority: Minor
> Attachments: HIVE-18373.1.patch
>
>
> Within a database, to filter for tables with the string 'abc' in its name, I
> can use something like:
> {code:java}
> hive> use my_database;
> hive> show tables '*abc*';
> {code}
> It would be great if I can do something similar to search within the list of
> columns in a table.
> I have a table with around 3200 columns. Searching for the column of interest
> is an onerous task after doing a {{describe}} on it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-17199) TestHiveMetaStoreStatsMerge: testStatsMerge
[ https://issues.apache.org/jira/browse/HIVE-17199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRASHANT GOLASH resolved HIVE-17199. Resolution: Won't Fix Test case was deleted as part of HIVE-17286 > TestHiveMetaStoreStatsMerge: testStatsMerge > --- > > Key: HIVE-17199 > URL: https://issues.apache.org/jira/browse/HIVE-17199 > Project: Hive > Issue Type: Sub-task >Reporter: Janaki Lahorani >Assignee: PRASHANT GOLASH >Priority: Major > > Stack: > org.apache.thrift.transport.TTransportException > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_aggr_stats_for(ThriftHiveMetastore.java:3724) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_aggr_stats_for(ThriftHiveMetastore.java:3711) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setPartitionColumnStatistics(HiveMetaStoreClient.java:1704) > at > org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge(TestHiveMetaStoreStatsMerge.java:187) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18618) Table stats on Micro-managed table becomes incorrect after INSERT OVERWRITE
[ https://issues.apache.org/jira/browse/HIVE-18618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352856#comment-16352856 ] Sergey Shelukhin commented on HIVE-18618: - There's a fix in HIVE-18571 > Table stats on Micro-managed table becomes incorrect after INSERT OVERWRITE > --- > > Key: HIVE-18618 > URL: https://issues.apache.org/jira/browse/HIVE-18618 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > > Micromanage table INSERT OVERWRITE now does not clean existing delta > directories > during the operation. Instead compaction cleaner deletes obsolete directories. > The current system does not compute correctly when it calculates table > statistics > since it does not ignore obsolete delta directories. > A fix is already available in the patch of HIVE-18052. It would be valuable > to separate > the code changes into this jira ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352863#comment-16352863
]
Eugene Koifman commented on HIVE-16605:
---
merge_constraint_notnull.q sets hive.merge.cardinality.check=false
is that intentional? if so why?
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16605.1.patch, HIVE-16605.2.patch,
> HIVE-16605.3.patch, HIVE-16605.4.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18571) stats issues for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352865#comment-16352865 ] Steve Yeom commented on HIVE-18571: --- Hi [~sershe] Do you have test cases for this jira? Please let me know all of them if you have so that I can check with my jira HIVE-18618. > stats issues for MM tables > -- > > Key: HIVE-18571 > URL: https://issues.apache.org/jira/browse/HIVE-18571 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18571.01.patch, HIVE-18571.patch > > > There are multiple stats aggregation issues with MM tables. > Some simple stats are double counted and some stats (simple stats) are > invalid for ACID table dirs altogether. > I have a patch almost ready, need to fix some more stuff and clean up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18571) stats issues for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352876#comment-16352876 ] Sergey Shelukhin commented on HIVE-18571: - The original test case if dynpart_sort_optimization with MM-by-default patch, that has stats differences due to both of these issues (rows are incorrect due to incorrect accumulation, and size and number of files due to counting obsolete directories). > stats issues for MM tables > -- > > Key: HIVE-18571 > URL: https://issues.apache.org/jira/browse/HIVE-18571 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18571.01.patch, HIVE-18571.patch > > > There are multiple stats aggregation issues with MM tables. > Some simple stats are double counted and some stats (simple stats) are > invalid for ACID table dirs altogether. > I have a patch almost ready, need to fix some more stuff and clean up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18618) Table stats on Micro-managed table becomes incorrect after INSERT OVERWRITE
[ https://issues.apache.org/jira/browse/HIVE-18618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352878#comment-16352878 ] Steve Yeom commented on HIVE-18618: --- I think I should have committed to the master the fix for this issue so that there would be no conflict. Found that the fix is included at HIVE-18052.15.patch, which is created on Jan 4, 2018 > Table stats on Micro-managed table becomes incorrect after INSERT OVERWRITE > --- > > Key: HIVE-18618 > URL: https://issues.apache.org/jira/browse/HIVE-18618 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > > Micromanage table INSERT OVERWRITE now does not clean existing delta > directories > during the operation. Instead compaction cleaner deletes obsolete directories. > The current system does not compute correctly when it calculates table > statistics > since it does not ignore obsolete delta directories. > A fix is already available in the patch of HIVE-18052. It would be valuable > to separate > the code changes into this jira ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352880#comment-16352880
]
Hive QA commented on HIVE-18622:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} storage-api: The patch generated 1 new + 128 unchanged
- 0 fixed = 129 total (was 128) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 3 new + 757 unchanged - 0
fixed = 760 total (was 757) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / e0eee6b |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9026/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9026/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9026/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9026/yetus/whitespace-tabs.txt
|
| modules | C: storage-api ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9026/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: IF Statements, Comparisons, and more do not handle NULLs
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.01.patch, HIVE-18622.02.patch
>
>
>
> Many vector expression classes are missing guards around setting noNulls
> among other things.
> {code:java}
> // Carefully update noNulls
[jira] [Updated] (HIVE-18601) Support Power platform by updating protoc-jar-maven-plugin version
[
https://issues.apache.org/jira/browse/HIVE-18601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-18601:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Patch committed. Thanks Pravin.
> Support Power platform by updating protoc-jar-maven-plugin version
> --
>
> Key: HIVE-18601
> URL: https://issues.apache.org/jira/browse/HIVE-18601
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.0.0
> Environment: # uname -a
> Linux pts00607-vm16 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC
> 2016 ppc64le ppc64le ppc64le GNU/Linux
> # # cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=16.04
> DISTRIB_CODENAME=xenial
> DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
>Reporter: Pravin Dsilva
>Assignee: Pravin Dsilva
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18601.patch
>
>
> Below is error is seen while building standalone-metastore project
> {code:java}
> [[1;34mINFO[m] [1m--- [0;32mprotoc-jar-maven-plugin:3.0.0-a3:run[m
> [1m(default)[m @ [36mhive-standalone-metastore[0;1m ---[m
> [[1;34mINFO[m] Protoc version: 2.5.0
> [[1;34mINFO[m] Input directories:
> [[1;34mINFO[m]
> /var/lib/jenkins/workspace/hive/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore
> [[1;34mINFO[m] Output targets:
> [[1;34mINFO[m] java:
> /var/lib/jenkins/workspace/hive/standalone-metastore/target/generated-sources
> (add: none, clean: false)
> [[1;34mINFO[m]
> /var/lib/jenkins/workspace/hive/standalone-metastore/target/generated-sources
> does not exist. Creating...
> [[1;34mINFO[m] Processing (java): metastore.proto
> protoc-jar: protoc version: 250, detected platform: linux/ppc64le
> protoc-jar: executing: [/tmp/protoc184130581004216.exe,
> -I/var/lib/jenkins/workspace/hive/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
>
> --java_out=/var/lib/jenkins/workspace/hive/standalone-metastore/target/generated-sources,
>
> /var/lib/jenkins/workspace/hive/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
> /tmp/protoc184130581004216.exe: 1: /tmp/protoc184130581004216.exe:
> ELF: not found
> /tmp/protoc184130581004216.exe: 1: /tmp/protoc184130581004216.exe:
> Cȁ: not found
> /tmp/protoc184130581004216.exe: 2: /tmp/protoc184130581004216.exe: �:
> not found
> /tmp/protoc184130581004216.exe: 3: /tmp/protoc184130581004216.exe:
> ��_�c���jnP���R���?��Y@�9� � Ch��߳yIk��: not found
> /tmp/protoc184130581004216.exe: 2: /tmp/protoc184130581004216.exe:
> Syntax error: Unterminated quoted string{code}
>
> The protoc-jar-maven-plugin version used is 3.0.0-a3 whereas Power (ppc64le)
> support was added in 3.5.1.1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18597) LLAP: Packaging for log4j NDC has two log4j jars it can pick
[
https://issues.apache.org/jira/browse/HIVE-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352900#comment-16352900
]
Sergey Shelukhin commented on HIVE-18597:
-
+1
> LLAP: Packaging for log4j NDC has two log4j jars it can pick
>
>
> Key: HIVE-18597
> URL: https://issues.apache.org/jira/browse/HIVE-18597
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18597.1.patch
>
>
> The log4j 1 and 2.x jars have NDC classes with identical package names.
> {code}
> // log4j-1.2-API needed for NDC
> org.apache.log4j.NDC.class,
> {code}
> Queries fail with
> {code}
> java.lang.ClassCastException: org.apache.log4j.NDC$DiagnosticContext cannot
> be cast to java.lang.String
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.setMDCFromNDC(TaskRunnerCallable.java:312)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {code}
> If the service driver packages the log4j1.x jar instead of the 2.x API shim
> jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Attachment: (was: HIVE-18614.patch) > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Attachment: HIVE-18614.patch > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Status: Open (was: Patch Available) > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Status: Patch Available (was: Open) > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352914#comment-16352914 ] Miklos Gergely commented on HIVE-18614: --- I've put back the quotes. Actually I've removed them because I tried to execute the file first from beeline shell, and it caused a problem there. Turned out that the HiveSchemaTool first fixes the quotes if the underlying database is mysql, so it's fine as it is without modification. See the attached patch. cc [~sershe], > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-18421:
---
Attachment: HIVE-18421.03.patch
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352918#comment-16352918
]
Vihang Karajgaonkar commented on HIVE-18421:
latest patch should fix most of the tests. Also, made changes to keep yetus
happy.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15353) Metastore throws NPE if StorageDescriptor.cols is null
[ https://issues.apache.org/jira/browse/HIVE-15353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-15353: -- Status: Open (was: Patch Available) [~erwaman], some files have moved around due to recent metastore changes, so this patch no longer applies. Can you please rebase and repost? Thanks. > Metastore throws NPE if StorageDescriptor.cols is null > -- > > Key: HIVE-15353 > URL: https://issues.apache.org/jira/browse/HIVE-15353 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0, 1.1.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Attachments: HIVE-15353.1.patch, HIVE-15353.2.patch, > HIVE-15353.3.patch, HIVE-15353.4.patch > > > When using the HiveMetaStoreClient API directly to talk to the metastore, you > get NullPointerExceptions when StorageDescriptor.cols is null in the > Table/Partition object in the following calls: > * create_table > * alter_table > * alter_partition > Calling add_partition with StorageDescriptor.cols set to null causes null to > be stored in the metastore database and subsequent calls to alter_partition > for that partition to fail with an NPE. > Null checks should be added to eliminate the NPEs in the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18580) Create tests to cover exchange partitions
[
https://issues.apache.org/jira/browse/HIVE-18580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352958#comment-16352958
]
Hive QA commented on HIVE-18580:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} standalone-metastore: The patch generated 30 new + 0
unchanged - 0 fixed = 30 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fa4af2f |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9027/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9027/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create tests to cover exchange partitions
> -
>
> Key: HIVE-18580
> URL: https://issues.apache.org/jira/browse/HIVE-18580
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-18580.1.patch
>
>
> The following methods of IMetaStoreClient are covered in this Jira:
> {code:java}
> - int Partition exchange_partition(Map, String, String,
> String, String)
> - List Partition exchange_partition(Map, String,
> String, String, String){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18616) work around HADOOP-15171 p2
[ https://issues.apache.org/jira/browse/HIVE-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18616: Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > work around HADOOP-15171 p2 > --- > > Key: HIVE-18616 > URL: https://issues.apache.org/jira/browse/HIVE-18616 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18616.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18395) Using stats for aggregate query on Acid/MM is off even with "hive.compute.query.using.stats" is true.
[ https://issues.apache.org/jira/browse/HIVE-18395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Yeom reassigned HIVE-18395: - Assignee: Steve Yeom > Using stats for aggregate query on Acid/MM is off even with > "hive.compute.query.using.stats" is true. > - > > Key: HIVE-18395 > URL: https://issues.apache.org/jira/browse/HIVE-18395 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18597) LLAP: Always package the log4j2 API for org.apache.log4j
[
https://issues.apache.org/jira/browse/HIVE-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18597:
---
Summary: LLAP: Always package the log4j2 API for org.apache.log4j (was:
LLAP: Packaging for log4j NDC has two log4j jars it can pick)
> LLAP: Always package the log4j2 API for org.apache.log4j
>
>
> Key: HIVE-18597
> URL: https://issues.apache.org/jira/browse/HIVE-18597
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18597.1.patch
>
>
> The log4j 1 and 2.x jars have NDC classes with identical package names.
> {code}
> // log4j-1.2-API needed for NDC
> org.apache.log4j.NDC.class,
> {code}
> Queries fail with
> {code}
> java.lang.ClassCastException: org.apache.log4j.NDC$DiagnosticContext cannot
> be cast to java.lang.String
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.setMDCFromNDC(TaskRunnerCallable.java:312)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {code}
> If the service driver packages the log4j1.x jar instead of the 2.x API shim
> jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18597) LLAP: Always package the log4j2 API jar for org.apache.log4j
[
https://issues.apache.org/jira/browse/HIVE-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18597:
---
Summary: LLAP: Always package the log4j2 API jar for org.apache.log4j
(was: LLAP: Always package the log4j2 API for org.apache.log4j)
> LLAP: Always package the log4j2 API jar for org.apache.log4j
>
>
> Key: HIVE-18597
> URL: https://issues.apache.org/jira/browse/HIVE-18597
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18597.1.patch
>
>
> The log4j 1 and 2.x jars have NDC classes with identical package names.
> {code}
> // log4j-1.2-API needed for NDC
> org.apache.log4j.NDC.class,
> {code}
> Queries fail with
> {code}
> java.lang.ClassCastException: org.apache.log4j.NDC$DiagnosticContext cannot
> be cast to java.lang.String
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.setMDCFromNDC(TaskRunnerCallable.java:312)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {code}
> If the service driver packages the log4j1.x jar instead of the 2.x API shim
> jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18597) LLAP: Always package the log4j2 API jar for org.apache.log4j
[
https://issues.apache.org/jira/browse/HIVE-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18597:
---
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed master, thanks [~sershe] and [~prasanth_j]
> LLAP: Always package the log4j2 API jar for org.apache.log4j
>
>
> Key: HIVE-18597
> URL: https://issues.apache.org/jira/browse/HIVE-18597
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18597.1.patch
>
>
> The log4j 1 and 2.x jars have NDC classes with identical package names.
> {code}
> // log4j-1.2-API needed for NDC
> org.apache.log4j.NDC.class,
> {code}
> Queries fail with
> {code}
> java.lang.ClassCastException: org.apache.log4j.NDC$DiagnosticContext cannot
> be cast to java.lang.String
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.setMDCFromNDC(TaskRunnerCallable.java:312)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {code}
> If the service driver packages the log4j1.x jar instead of the 2.x API shim
> jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353024#comment-16353024
]
Vineet Garg commented on HIVE-16605:
[~ashutoshc] Failures are unrelated. [~ekoifman] This wasn't intentional. I
will remove this and upload the patch.
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16605.1.patch, HIVE-16605.2.patch,
> HIVE-16605.3.patch, HIVE-16605.4.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18580) Create tests to cover exchange partitions
[
https://issues.apache.org/jira/browse/HIVE-18580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353025#comment-16353025
]
Hive QA commented on HIVE-18580:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909247/HIVE-18580.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13112 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=241)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_div0]
(batchId=69)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
(batchId=182)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=222)
org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet
(batchId=215)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=281)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=281)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=283)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=257)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=235)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9027/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9027/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9027/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909247 - PreCommit-HIVE-Build
> Create tests to cover exchange partitions
> -
>
> Key: HIVE-18580
> URL: https://issues.apache.org/jira/browse/HIVE-18580
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-18580.1.patch
>
>
> The following methods of IMetaStoreClient are covered in this Jira:
> {code:java}
> - int Partition exchange_partition(Map, String, String,
> String, String)
> - List Partition exchange_partition(Map, String,
> String, String, String){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18613) Extend JsonSerDe to support BINARY type
[
https://issues.apache.org/jira/browse/HIVE-18613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353047#comment-16353047
]
Hive QA commented on HIVE-18613:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hcatalog/core: The patch generated 2 new + 39
unchanged - 1 fixed = 41 total (was 40) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 3e85169 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9028/yetus/diff-checkstyle-hcatalog_core.txt
|
| modules | C: hcatalog/core U: hcatalog/core |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9028/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Extend JsonSerDe to support BINARY type
> ---
>
> Key: HIVE-18613
> URL: https://issues.apache.org/jira/browse/HIVE-18613
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18613.patch
>
>
> It seems only type not supported currently, from the code:
> {code}
> case BINARY:
> throw new IOException("JsonSerDe does not support BINARY type");
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18281) HiveServer2 HA for LLAP and Workload Manager
[ https://issues.apache.org/jira/browse/HIVE-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18281: - Attachment: HIVE-18281.2.patch > HiveServer2 HA for LLAP and Workload Manager > > > Key: HIVE-18281 > URL: https://issues.apache.org/jira/browse/HIVE-18281 > Project: Hive > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18281.1.patch, HIVE-18281.2.patch, > HIVE-18281.WIP.patch, HSI-HA.pdf > > > When running HS2 with LLAP and Workload Manager, HS2 becomes single point of > failure as some of the states for workload management and scheduling are > maintained in-memory. > The proposal is to support Active/Passive mode of high availability in which, > all HS2 and tez AMs registers with ZooKeeper and a leader have to be chosen > which will maintain stateful information. Clients using service discovery > will always connect to the leader for submitting queries. There will also be > some responsibilities for the leader, failover handling, tez session > reconnect etc. Will upload some more detailed information in a separate doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18281) HiveServer2 HA for LLAP and Workload Manager
[
https://issues.apache.org/jira/browse/HIVE-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353050#comment-16353050
]
Prasanth Jayachandran commented on HIVE-18281:
--
Added /peers endpoint which outputs the following
{code}
{
"hiveServer2Instances": [
{
"host": "localhost",
"isLeader": true,
"transportMode": "binary",
"httpEndpoint": "",
"properties": {
"hive.server2.instance.uri": "localhost:49816",
"hive.server2.authentication": "NONE",
"hive.server2.transport.mode": "binary",
"hive.server2.thrift.sasl.qop": "auth",
"hive.server2.thrift.bind.host": "localhost",
"hive.server2.thrift.port": "49816",
"hive.server2.use.SSL": "false",
"registry.unique.id": "46db281b-6182-4264-9e9d-6c9f99ed3441"
},
"workerIdentity": "46db281b-6182-4264-9e9d-6c9f99ed3441",
"port": 49816
},
{
"host": "localhost",
"isLeader": false,
"transportMode": "binary",
"httpEndpoint": "",
"properties": {
"hive.server2.instance.uri": "localhost:49820",
"hive.server2.authentication": "NONE",
"hive.server2.transport.mode": "binary",
"hive.server2.thrift.sasl.qop": "auth",
"hive.server2.thrift.bind.host": "localhost",
"hive.server2.thrift.port": "49820",
"hive.server2.use.SSL": "false",
"registry.unique.id": "46db281b-6182-4264-9e9d-6c9f99ed3441"
},
"workerIdentity": "46db281b-6182-4264-9e9d-6c9f99ed3441",
"port": 49820
}
]
}
{code}
> HiveServer2 HA for LLAP and Workload Manager
>
>
> Key: HIVE-18281
> URL: https://issues.apache.org/jira/browse/HIVE-18281
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18281.1.patch, HIVE-18281.2.patch,
> HIVE-18281.WIP.patch, HSI-HA.pdf
>
>
> When running HS2 with LLAP and Workload Manager, HS2 becomes single point of
> failure as some of the states for workload management and scheduling are
> maintained in-memory.
> The proposal is to support Active/Passive mode of high availability in which,
> all HS2 and tez AMs registers with ZooKeeper and a leader have to be chosen
> which will maintain stateful information. Clients using service discovery
> will always connect to the leader for submitting queries. There will also be
> some responsibilities for the leader, failover handling, tez session
> reconnect etc. Will upload some more detailed information in a separate doc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18281) HiveServer2 HA for LLAP and Workload Manager
[ https://issues.apache.org/jira/browse/HIVE-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353066#comment-16353066 ] Prasanth Jayachandran commented on HIVE-18281: -- Added RB link > HiveServer2 HA for LLAP and Workload Manager > > > Key: HIVE-18281 > URL: https://issues.apache.org/jira/browse/HIVE-18281 > Project: Hive > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18281.1.patch, HIVE-18281.2.patch, > HIVE-18281.WIP.patch, HSI-HA.pdf > > > When running HS2 with LLAP and Workload Manager, HS2 becomes single point of > failure as some of the states for workload management and scheduling are > maintained in-memory. > The proposal is to support Active/Passive mode of high availability in which, > all HS2 and tez AMs registers with ZooKeeper and a leader have to be chosen > which will maintain stateful information. Clients using service discovery > will always connect to the leader for submitting queries. There will also be > some responsibilities for the leader, failover handling, tez session > reconnect etc. Will upload some more detailed information in a separate doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18586) Upgrade Derby to 10.14.1.0
[ https://issues.apache.org/jira/browse/HIVE-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-18586: --- Attachment: HIVE-18586.3.patch > Upgrade Derby to 10.14.1.0 > -- > > Key: HIVE-18586 > URL: https://issues.apache.org/jira/browse/HIVE-18586 > Project: Hive > Issue Type: Improvement >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-18586.1.patch, HIVE-18586.2.patch, > HIVE-18586.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries
[
https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-17735:
---
Assignee: Aihua Xu
> ObjectStore.addNotificationEvent is leaking queries
> ---
>
> Key: HIVE-17735
> URL: https://issues.apache.org/jira/browse/HIVE-17735
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Assignee: Aihua Xu
>Priority: Major
>
> In ObjectStore.addNotificationEvent():
> {code}
> Query objectQuery = pm.newQuery(MNotificationNextId.class);
> Collection ids = (Collection)
> objectQuery.execute();
> {code}
> The query is never closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries
[
https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-17735:
Attachment: HIVE-17735.1.patch
> ObjectStore.addNotificationEvent is leaking queries
> ---
>
> Key: HIVE-17735
> URL: https://issues.apache.org/jira/browse/HIVE-17735
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-17735.1.patch
>
>
> In ObjectStore.addNotificationEvent():
> {code}
> Query objectQuery = pm.newQuery(MNotificationNextId.class);
> Collection ids = (Collection)
> objectQuery.execute();
> {code}
> The query is never closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
