[jira] [Updated] (HIVE-22588) Flush the remaining rows for the rest of the grouping sets when switching the vector groupby mode
[ https://issues.apache.org/jira/browse/HIVE-22588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22588: Status: Open (was: Patch Available) > Flush the remaining rows for the rest of the grouping sets when switching the > vector groupby mode > - > > Key: HIVE-22588 > URL: https://issues.apache.org/jira/browse/HIVE-22588 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22588.1.patch, HIVE-22588.2.patch > > > Flush the remaining rows for the rest of the grouping sets when switching the > vector groupby mode -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22588) Flush the remaining rows for the rest of the grouping sets when switching the vector groupby mode
[ https://issues.apache.org/jira/browse/HIVE-22588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22588: Attachment: HIVE-22588.2.patch Status: Patch Available (was: Open) > Flush the remaining rows for the rest of the grouping sets when switching the > vector groupby mode > - > > Key: HIVE-22588 > URL: https://issues.apache.org/jira/browse/HIVE-22588 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22588.1.patch, HIVE-22588.2.patch > > > Flush the remaining rows for the rest of the grouping sets when switching the > vector groupby mode -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22553) Expose locks and transactions in sys db
[
https://issues.apache.org/jira/browse/HIVE-22553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989488#comment-16989488
]
Hive QA commented on HIVE-22553:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 3m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19770/dev-support/hive-personality.sh
|
| git revision | master / 74ae418 |
| modules | C: metastore ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19770/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Expose locks and transactions in sys db
> ---
>
> Key: HIVE-22553
> URL: https://issues.apache.org/jira/browse/HIVE-22553
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22553.2.patch, HIVE-22553.3.patch,
> HIVE-22553.3.patch, HIVE-22553.patch
>
>
> Create new sysdb tables/views to access lock and transaction data.
> This allows to provide admins with live data about ongoing locks and
> transacions. Due to this being in the sys db access to this information can
> be restricted to select privileged users.
> Information about locks and compactions can be joined and accessed at the
> same time.
> Compaction related transactions would also be visible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22583) LLAP cache always misses with non-vectorized serde readers such as OpenCSV
[
https://issues.apache.org/jira/browse/HIVE-22583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989478#comment-16989478
]
Hive QA commented on HIVE-22583:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987578/HIVE-22583.0.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17759 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[unicode_data]
(batchId=87)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_io_etl]
(batchId=168)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19769/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19769/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19769/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987578 - PreCommit-HIVE-Build
> LLAP cache always misses with non-vectorized serde readers such as OpenCSV
> --
>
> Key: HIVE-22583
> URL: https://issues.apache.org/jira/browse/HIVE-22583
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22583.0.patch
>
>
> Although after the first read LLAP cache stores data of tables that are not
> using the LazySimple serde, the stored data is then never used in the future
> subsequent queries, causing a full cache miss and re-read each time.
> Problem is rooted in SerdeEncodedDataReader#cacheFileData is not taking care
> of creating an entry for the root/struct column of the table. The only cases
> this is taken care of are when a vectorized reader is used _(e.g.
> LazySimpleSerde's LazySimpleDeserializeRead)_, where
> SerdeEncodedDataReader#processAsyncCacheData takes care of this.
> This can be reproduced by either using a custom serde, like OpenCSV or using
> LazySimpleSerde, but turning off _hive.llap.io.encode.vector.serde.enabled_.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22583) LLAP cache always misses with non-vectorized serde readers such as OpenCSV
[
https://issues.apache.org/jira/browse/HIVE-22583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989447#comment-16989447
]
Hive QA commented on HIVE-22583:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
23s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19769/dev-support/hive-personality.sh
|
| git revision | master / 74ae418 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql llap-server itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19769/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP cache always misses with non-vectorized serde readers such as OpenCSV
> --
>
> Key: HIVE-22583
> URL: https://issues.apache.org/jira/browse/HIVE-22583
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22583.0.patch
>
>
> Although after the first read LLAP cache stores data of tables that are not
> using the LazySimple serde, the stored data is then never used in the future
> subsequent queries, causing a full cache miss and re-read each time.
> Problem is rooted in SerdeEncodedDataReader#cacheFileData is not taking care
> of creating an entry for the root/struct column of the table. The only cases
> this is taken care of are when a vectorized reader is used _(e.g.
> LazySimpleSerde's LazySimpleDeserializeRead)_, where
> SerdeEncodedDataReader#processAsyncCacheData takes care of this.
> This can be reproduced by either using a custom serde, like OpenCSV or using
> LazySimpleSerde, but turning off _hive.llap.io.encode.vector.serde.enabled_.
[jira] [Commented] (HIVE-22578) CBO genOPTree is not failsafe for CTAS and VIEW statements
[ https://issues.apache.org/jira/browse/HIVE-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989430#comment-16989430 ] Vineet Garg commented on HIVE-22578: bq. If OT generation for the new AST fails with a SemanticException I would think if this is the case it means there is something wrong with the query itself, otherwise genOptTree shouldn't be throwing SemanticException. [~hamvas.aron] Is there a test case? > CBO genOPTree is not failsafe for CTAS and VIEW statements > -- > > Key: HIVE-22578 > URL: https://issues.apache.org/jira/browse/HIVE-22578 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.0.0, 3.1.2 >Reporter: Aron Hamvas >Assignee: Aron Hamvas >Priority: Major > Fix For: 3.0.1, 3.1.3 > > Attachments: HIVE-22578.patch > > > If CBO fails during OT generation, it is supposed to skip CBO and fall back > to SemanticAnalyzer generating the OT for the original AST. However, for CTAS > and VIEW statements, the original AST is discarded and replaced with the new > AST. > If OT generation for the new AST fails with a SemanticException, the > compilation is destined to fail completely since we cannot fall back to the > original and possibly valid AST. > Also, the SemanticException sometimes gets hidden and misjudged as a missing > column statistics issue due to bad error handling. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22588) Flush the remaining rows for the rest of the grouping sets when switching the vector groupby mode
[ https://issues.apache.org/jira/browse/HIVE-22588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22588: Attachment: HIVE-22588.1.patch Status: Patch Available (was: Open) > Flush the remaining rows for the rest of the grouping sets when switching the > vector groupby mode > - > > Key: HIVE-22588 > URL: https://issues.apache.org/jira/browse/HIVE-22588 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22588.1.patch > > > Flush the remaining rows for the rest of the grouping sets when switching the > vector groupby mode -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22578) CBO genOPTree is not failsafe for CTAS and VIEW statements
[
https://issues.apache.org/jira/browse/HIVE-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989418#comment-16989418
]
Hive QA commented on HIVE-22578:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987577/HIVE-22578.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 139 failed/errored test(s), 17758 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[having]
(batchId=304)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ambiguous_col]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_filter]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_select]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_nullable_union]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket2] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket3] (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_1]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_3]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_8]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[case_sensitivity]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[combine3] (batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_union_table]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[equal_ns] (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_window]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[having] (batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_dynamicserde]
(batchId=98)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_lazyserde2]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_lazyserde]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_testxpath2]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_testxpath3]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_testxpath4]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_testxpath]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_by_range_rule_not_null]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_thrift] (batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown_negative]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_5] (batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_buckets] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nested_column_pruning]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonmr_fetch] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_avro_partition_uniontype]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_empty_table]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_nested_column_pruning]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_wise_fileformat18]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample1] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample3] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample5] (batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample8] (batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_20]
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_22]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats10] (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_ngrams] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_isnull_isnotnull]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_size] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union21] (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[uniquejoin
[jira] [Assigned] (HIVE-22588) Flush the remaining rows for the rest of the grouping sets when switching the vector groupby mode
[ https://issues.apache.org/jira/browse/HIVE-22588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan reassigned HIVE-22588: --- Assignee: Ramesh Kumar Thangarajan > Flush the remaining rows for the rest of the grouping sets when switching the > vector groupby mode > - > > Key: HIVE-22588 > URL: https://issues.apache.org/jira/browse/HIVE-22588 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > > Flush the remaining rows for the rest of the grouping sets when switching the > vector groupby mode -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22578) CBO genOPTree is not failsafe for CTAS and VIEW statements
[
https://issues.apache.org/jira/browse/HIVE-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989400#comment-16989400
]
Hive QA commented on HIVE-22578:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
34s{color} | {color:red} ql generated 1 new + 1531 unchanged - 1 fixed = 1532
total (was 1532) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(SemanticAnalyzer$PlannerContext)
At CalcitePlanner.java:is not thrown in
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(SemanticAnalyzer$PlannerContext)
At CalcitePlanner.java:[line 565] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19768/dev-support/hive-personality.sh
|
| git revision | master / 74ae418 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19768/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19768/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO genOPTree is not failsafe for CTAS and VIEW statements
> --
>
> Key: HIVE-22578
> URL: https://issues.apache.org/jira/browse/HIVE-22578
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0, 3.1.2
>Reporter: Aron Hamvas
>Assignee: Aron Hamvas
>Priority: Major
> Fix For: 3.0.1, 3.1.3
>
> Attachments: HIVE-22578.patch
>
>
> If CBO fails during OT generation, it is supposed to skip CBO and fall back
> to SemanticAnalyzer generating the OT for the original AST. However, for CTAS
> and VIEW statements, the original AST is discarded and replaced with the new
> AST.
> If OT generation for the new AST fails with a SemanticException, the
> compilation is destined to fail completely since we cannot fall back to the
> original and possibly valid AST.
> Also, the SemanticException sometimes gets hidden and misjudged as a missing
> column statistics issue due to bad error handling.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22548) Optimise Utilities.removeTempOrDuplicateFiles when moving files to final location
[
https://issues.apache.org/jira/browse/HIVE-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989389#comment-16989389
]
Hive QA commented on HIVE-22548:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987575/HIVE-22548.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17758 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19766/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19766/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19766/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987575 - PreCommit-HIVE-Build
> Optimise Utilities.removeTempOrDuplicateFiles when moving files to final
> location
> -
>
> Key: HIVE-22548
> URL: https://issues.apache.org/jira/browse/HIVE-22548
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.2
>Reporter: Rajesh Balamohan
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-22548.01.patch, HIVE-22548.02.patch
>
>
> {{Utilities.removeTempOrDuplicateFiles}}
> is very slow with cloud storage, as it executes {{listStatus}} twice and also
> runs in single threaded mode.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1629
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22548) Optimise Utilities.removeTempOrDuplicateFiles when moving files to final location
[
https://issues.apache.org/jira/browse/HIVE-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989366#comment-16989366
]
Hive QA commented on HIVE-22548:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 1 new + 105 unchanged - 0
fixed = 106 total (was 105) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19766/dev-support/hive-personality.sh
|
| git revision | master / 74ae418 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19766/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19766/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimise Utilities.removeTempOrDuplicateFiles when moving files to final
> location
> -
>
> Key: HIVE-22548
> URL: https://issues.apache.org/jira/browse/HIVE-22548
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.2
>Reporter: Rajesh Balamohan
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-22548.01.patch, HIVE-22548.02.patch
>
>
> {{Utilities.removeTempOrDuplicateFiles}}
> is very slow with cloud storage, as it executes {{listStatus}} twice and also
> runs in single threaded mode.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1629
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22582) Avoid reading table as ACID when table name is starting with "delta" , but table is not transactional and BI Split Strategy is used
[
https://issues.apache.org/jira/browse/HIVE-22582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989354#comment-16989354
]
Hive QA commented on HIVE-22582:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987573/HIVE-22582.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17758 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19765/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19765/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19765/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987573 - PreCommit-HIVE-Build
> Avoid reading table as ACID when table name is starting with "delta" , but
> table is not transactional and BI Split Strategy is used
> ---
>
> Key: HIVE-22582
> URL: https://issues.apache.org/jira/browse/HIVE-22582
> Project: Hive
> Issue Type: Bug
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Attachments: HIVE-22582.patch
>
>
> The issue is fixed in HIVE-22473 but missed a check for BI Split Strategy.
> Steps to reproduce:
> {code:java}
> set hive.exec.orc.split.strategy=BI;
> create table delta_result (a int) stored as orc
> tblproperties('transactional'='false');
> insert into delta_result select 1;
> select * from delta_result;
> {code}
> Exception Stack Trace:
> {code:java}
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: String index out of range: -1
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1929)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2016)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:461)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:430)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:336)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:576)
> ... 50 more
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.parsedDelta(AcidUtils.java:1128)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:921)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getLogicalLength(AcidUtils.java:2084)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$BISplitStrategy.getSplits(OrcInputFormat.java:1115)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1905)
> ... 55 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22582) Avoid reading table as ACID when table name is starting with "delta" , but table is not transactional and BI Split Strategy is used
[
https://issues.apache.org/jira/browse/HIVE-22582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989323#comment-16989323
]
Hive QA commented on HIVE-22582:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19765/dev-support/hive-personality.sh
|
| git revision | master / 74ae418 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19765/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Avoid reading table as ACID when table name is starting with "delta" , but
> table is not transactional and BI Split Strategy is used
> ---
>
> Key: HIVE-22582
> URL: https://issues.apache.org/jira/browse/HIVE-22582
> Project: Hive
> Issue Type: Bug
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Attachments: HIVE-22582.patch
>
>
> The issue is fixed in HIVE-22473 but missed a check for BI Split Strategy.
> Steps to reproduce:
> {code:java}
> set hive.exec.orc.split.strategy=BI;
> create table delta_result (a int) stored as orc
> tblproperties('transactional'='false');
> insert into delta_result select 1;
> select * from delta_result;
> {code}
> Exception Stack Trace:
> {code:java}
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: String index out of range: -1
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1929)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2016)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:461)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:430)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:336)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:576)
> ... 50 more
> Caused by: java.
[jira] [Resolved] (HIVE-22108) Cannot use structured field in a window function for a text table
[
https://issues.apache.org/jira/browse/HIVE-22108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrian Wang resolved HIVE-22108.
Resolution: Duplicate
Already fixed in HIVE-21104
> Cannot use structured field in a window function for a text table
> -
>
> Key: HIVE-22108
> URL: https://issues.apache.org/jira/browse/HIVE-22108
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Xianyin Xin
>Priority: Major
>
> Sample text which has a map as its field:
> {code:java}
> E1719F13B8213BA7EE1694B0D2DC838B,1563490705041,detailtype:wt|topansweruuid:|questionuuid:57291db409ee405cb572d347babd8416
> E8EB4433DB8F4CAB84FCDA769E2AE7BA,1563490701704,detailtype:wt|topansweruuid:|questionuuid:b26abb023eae4efc982924d35aad1f57{code}
> Create table,
> {code:java}
> CREATE TABLE temp_table(
> user_id string,
> ct STRING,
> ep MAP
> )
> ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
> COLLECTION ITEMS TERMINATED BY '|'
> MAP KEYS TERMINATED BY ":"
> STORED AS TEXTFILE
> LOCATION '/user/test/temp_table'; {code}
> Execute query,
> {code:java}
> SELECT lag(ep, 1) over (partition by user_id order by ct) as lag_ep
> from temp_table;{code}
> It thows,
> {code:java}
> 2019-08-14T10:46:01,626 INFO [pool-11-thread-1] exec.FileSinkOperator: Using
> serializer : class
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe[[[B@d44c263]:[_col0]:[map]]
> and formatter :
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat@751a939a
> 2019-08-14T10:46:01,663 ERROR [pool-11-thread-1] ExecReducer:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":"04E55B1D60F5D70807BF224997623188","reducesinkkey1":"1563490210162"},"value":{"_col0":{"detailtype":"wt","topansweruuid":"","questionuuid":"3efc91897aec41249d39e2d2a4a6c0e5"}}}
> at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:243)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryMap cannot be cast to
> java.util.Map
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardMapObjectInspector.getMap(StandardMapObjectInspector.java:85)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:451)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:372)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFLeadLag$GenericUDAFLeadLagEvaluator.iterate(GenericUDAFLeadLag.java:156)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:213)
> at
> org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.processRow(WindowingTableFunction.java:407)
> at
> org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.processRow(PTFOperator.java:325)
> at org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:139)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:234){code}
> However, simple select is OK:
> {code:java}
> select ep['detailtype'] from temp_table;{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22502) ConcurrentModificationException in TriggerValidatorRunnable stops trigger processing
[
https://issues.apache.org/jira/browse/HIVE-22502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989298#comment-16989298
]
Hive QA commented on HIVE-22502:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987559/HIVE-22502.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17758 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19764/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19764/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19764/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987559 - PreCommit-HIVE-Build
> ConcurrentModificationException in TriggerValidatorRunnable stops trigger
> processing
>
>
> Key: HIVE-22502
> URL: https://issues.apache.org/jira/browse/HIVE-22502
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22502.1.patch
>
>
> An other thread is modifying the list the contains the sessions while
> TriggerValidatorRunnable is traversing it. This causes the
> TriggerValidatorRunnable thread to die and triggers are no longer properly
> monitored.
>
> {code:java}
> <12>1 2019-11-14T00:31:12.187Z
> hiveserver2-0.hiveserver2-service.compute-1572769905-6965.svc.cluster.local
> hiveserver2 1 fa2f30b6-ffb3-11e9-93ba-0a257c2413a2 [mdc@18060
> class="tez.TriggerValidatorRunnable" level="WARN" thread="TriggerValidator"]
> TriggerValidatorRunnable caught exception.<12>1 2019-11-14T00:31:12.187Z
> hiveserver2-0.hiveserver2-service.compute-1572769905-6965.svc.cluster.local
> hiveserver2 1 fa2f30b6-ffb3-11e9-93ba-0a257c2413a2 [mdc@18060
> class="tez.TriggerValidatorRunnable" level="WARN" thread="TriggerValidator"]
> TriggerValidatorRunnable caught
> exception.java.util.ConcurrentModificationException at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at
> java.util.LinkedList$ListItr.next(LinkedList.java:888) at
> java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1044) at
> org.apache.hadoop.hive.ql.exec.tez.TriggerValidatorRunnable.run(TriggerValidatorRunnable.java:49)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22491) Use Collections emptyList
[ https://issues.apache.org/jira/browse/HIVE-22491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22491: -- Attachment: HIVE-22491.3.patch > Use Collections emptyList > - > > Key: HIVE-22491 > URL: https://issues.apache.org/jira/browse/HIVE-22491 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22491.1.patch, HIVE-22491.2.patch, > HIVE-22491.3.patch > > > [https://docs.oracle.com/javase/8/docs/api/?java/util/Collections.html] > Use Collections#emptyList instead of instantiating empty ArrayLists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22529) Make Debugging Stacktrace More Explicit
[
https://issues.apache.org/jira/browse/HIVE-22529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22529:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
[~vgarg] Thank you for the review! Pushed to master.
> Make Debugging Stacktrace More Explicit
> ---
>
> Key: HIVE-22529
> URL: https://issues.apache.org/jira/browse/HIVE-22529
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22529.1.patch, HIVE-22529.1.patch
>
>
> In some places, the following DEBUG logging was introduced:
> {code:java}
> LOG.debug("Message", new Exception());
> {code}
> The purpose of this is to log the stack trace of the Thread calling this
> debug logging method. However, the resulting log message includes the
> following:
> {code:none}
> 2019-11-19T08:13:31,392 DEBUG [Thread] Logger: Message
> java.lang.Exception: null
> at
> {code}
> To the observer, it appears that there was perhaps some sort of NPE. Add a
> message to the Exception being generated to make it more clear that this
> "Exception" is for debugging purposes and not an actual error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22502) ConcurrentModificationException in TriggerValidatorRunnable stops trigger processing
[
https://issues.apache.org/jira/browse/HIVE-22502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989283#comment-16989283
]
Hive QA commented on HIVE-22502:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19764/dev-support/hive-personality.sh
|
| git revision | master / bc7b38e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19764/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ConcurrentModificationException in TriggerValidatorRunnable stops trigger
> processing
>
>
> Key: HIVE-22502
> URL: https://issues.apache.org/jira/browse/HIVE-22502
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22502.1.patch
>
>
> An other thread is modifying the list the contains the sessions while
> TriggerValidatorRunnable is traversing it. This causes the
> TriggerValidatorRunnable thread to die and triggers are no longer properly
> monitored.
>
> {code:java}
> <12>1 2019-11-14T00:31:12.187Z
> hiveserver2-0.hiveserver2-service.compute-1572769905-6965.svc.cluster.local
> hiveserver2 1 fa2f30b6-ffb3-11e9-93ba-0a257c2413a2 [mdc@18060
> class="tez.TriggerValidatorRunnable" level="WARN" thread="TriggerValidator"]
> TriggerValidatorRunnable caught exception.<12>1 2019-11-14T00:31:12.187Z
> hiveserver2-0.hiveserver2-service.compute-1572769905-6965.svc.cluster.local
> hiveserver2 1 fa2f30b6-ffb3-11e9-93ba-0a257c2413a2 [mdc@18060
> class="tez.TriggerValidatorRunnable" level="WARN" thread="TriggerValidator"]
> TriggerValidatorRunnable caught
> exception.java.util.ConcurrentModificationException at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at
> java.util.LinkedList$ListItr.next(LinkedList.java:888) at
> java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1044) at
> org.apache.hadoop.hive.ql.exec.tez.TriggerValidatorRunnable.run(Tr
[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state
[
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pablo Junge updated HIVE-22566:
---
Description:
If you try dropping a table which is part of the definition of a created
materialized view, the table is not dropped, which is the desired state as it
is part of the materialized view.
However, there was a "drop" call to the table, so it tried to drop it but did
not succeed, leaving it in an inconsistent state.
Repro:
---
1) Create tables:
{code:java}
CREATE TABLE emps ( empid INT, deptno INT, name VARCHAR(256), salary FLOAT,
hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES ('transactional'='true');
CREATE TABLE depts ( deptno INT, deptname VARCHAR(256), locationid
INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
{code}
2) Create the VM:
{code:java}
CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps
JOIN depts ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
{code}
3) Following is in backend database at this point:
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
++---+---+--+---+
| 81 | 16 | 81 | emps | MANAGED_TABLE |
| 83 | 16 | 83 | depts | MANAGED_TABLE |
| 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
4) Let's drop the 'emps' table:
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
INFO : Compiling
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop
table emps
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time
taken: 0.05 seconds
INFO : Executing
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop
table emps
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time
taken: 10.281 seconds
INFO : OK
No rows affected (16.949 seconds)
{code}
No issue displayed
5) List tables:
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
INFO : Compiling
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show
tables
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time
taken: 0.041 seconds
INFO : Executing
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show
tables
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time
taken: 0.016 seconds
INFO : OK
+---+
| tab_name |
+---+
| depts |
| emps |
+---+
2 rows selected (0.08 seconds)
{code}
6) Now, from the backend-db point of view:
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
++---+---+--+---+
| 81 | 16 | NULL | emps | MANAGED_TABLE |
| 83 | 16 | 83 | depts | MANAGED_TABLE |
| 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
The table is left with NULL in SD_ID, making it not available.
7) From Metastore.log
{code:java}
2019-12-02T20:00:25,545 INFO [pool-6-thread-195]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(907)) - 196: source:172.25.34.150 drop_table :
tbl=hive.mvs.emps
2019-12-02T20:00:25,545 INFO [pool-6-thread-195]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(349)) - ugi=hive ip=172.25.34.150
cmd=source:172.25.34.150 drop_table : tbl=hive.mvs.emps
2019-12-02T20:00:25,580 INFO [pool-6-thread-195]:
metastore.ObjectStore$RetryingExecutor (ObjectStore.java:run(9966)) -
Attempting to acquire the DB log notification lock: 0 out of 10 retries
javax.jdo.JDODataStoreException: Error executing SQL query "select
"NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
~[datanucleus-api-jdo-4.2.4.jar:?]
[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state
[
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pablo Junge updated HIVE-22566:
---
Description:
If you try dropping a table which is part of the definition of a created
materialized view, the table is not dropped, which is the desired state as it
is part of the materialized view.
However, there was a "drop" call to the table, so it tried to drop it but did
not succeed, leaving it in an inconsistent state.
Repro:
---
1) Create tables:
{code:java}
CREATE TABLE emps ( empid INT, deptno INT, name VARCHAR(256), salary FLOAT,
hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES ('transactional'='true');
CREATE TABLE depts ( deptno INT, deptname VARCHAR(256), locationid
INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
{code}
2) Create the VM:
{code:java}
CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_dateFROM emps JOIN
depts ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
{code}
3) Following is in backend database at this point:
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
++---+---+--+---+
| 81 | 16 | 81 | emps | MANAGED_TABLE |
| 83 | 16 | 83 | depts | MANAGED_TABLE |
| 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
4) Let's drop the 'emps' table:
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
INFO : Compiling
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop
table emps
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time
taken: 0.05 seconds
INFO : Executing
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop
table emps
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time
taken: 10.281 seconds
INFO : OK
No rows affected (16.949 seconds)
{code}
No issue displayed
5) List tables:
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
INFO : Compiling
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show
tables
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time
taken: 0.041 seconds
INFO : Executing
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show
tables
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time
taken: 0.016 seconds
INFO : OK
+---+
| tab_name |
+---+
| depts |
| emps |
+---+
2 rows selected (0.08 seconds)
{code}
6) Now, from the backend-db point of view:
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
++---+---+--+---+
| 81 | 16 | NULL | emps | MANAGED_TABLE |
| 83 | 16 | 83 | depts | MANAGED_TABLE |
| 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
The table is left with NULL in SD_ID, making it not available.
7) From Metastore.log
{code:java}
2019-12-02T20:00:25,545 INFO [pool-6-thread-195]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(907)) - 196: source:172.25.34.150 drop_table :
tbl=hive.mvs.emps
2019-12-02T20:00:25,545 INFO [pool-6-thread-195]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(349)) - ugi=hive ip=172.25.34.150
cmd=source:172.25.34.150 drop_table : tbl=hive.mvs.emps
2019-12-02T20:00:25,580 INFO [pool-6-thread-195]:
metastore.ObjectStore$RetryingExecutor (ObjectStore.java:run(9966)) -
Attempting to acquire the DB log notification lock: 0 out of 10 retries
javax.jdo.JDODataStoreException: Error executing SQL query "select
"NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
~[datanucleus-api-jdo-4.2.4.jar:?]
[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state
[
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pablo Junge updated HIVE-22566:
---
Description:
If you try dropping a table which is part of the definition of a created
materialized view, the table is not dropped, which is the desired state as it
is part of the materialized view.
However, there was a "drop" call to the table, so it tried to drop it but did
not succeed, leaving it in an inconsistent state.
Repro:
---
1) Create tables:
{code:java}
CREATE TABLE emps ( empid INT, deptno INT, name VARCHAR(256), salary FLOAT,
hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES ('transactional'='true');
CREATE TABLE depts ( deptno INT, deptname VARCHAR(256), locationid
INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
{code}
2) Create the VM:
{code:java}
CREATE MATERIALIZED VIEW mv1ASSELECT empid, deptname, hire_dateFROM emps JOIN
depts ON (emps.deptno = depts.deptno)WHERE hire_date >= '2016-01-01';
{code}
3) Following is in backend database at this point:
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
++---+---+--+---+
| 81 | 16 | 81 | emps | MANAGED_TABLE |
| 83 | 16 | 83 | depts | MANAGED_TABLE |
| 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
4) Let's drop the 'emps' table:
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
INFO : Compiling
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop
table emps
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time
taken: 0.05 seconds
INFO : Executing
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop
table emps
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time
taken: 10.281 seconds
INFO : OK
No rows affected (16.949 seconds)
{code}
No issue displayed
5) List tables:
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
INFO : Compiling
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show
tables
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time
taken: 0.041 seconds
INFO : Executing
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show
tables
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time
taken: 0.016 seconds
INFO : OK
+---+
| tab_name |
+---+
| depts |
| emps |
+---+
2 rows selected (0.08 seconds)
{code}
6) Now, from the backend-db point of view:
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
++---+---+--+---+
| 81 | 16 | NULL | emps | MANAGED_TABLE |
| 83 | 16 | 83 | depts | MANAGED_TABLE |
| 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
The table is left with NULL in SD_ID, making it not available.
7) From Metastore.log
{code:java}
2019-12-02T20:00:25,545 INFO [pool-6-thread-195]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(907)) - 196: source:172.25.34.150 drop_table :
tbl=hive.mvs.emps
2019-12-02T20:00:25,545 INFO [pool-6-thread-195]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(349)) - ugi=hive ip=172.25.34.150
cmd=source:172.25.34.150 drop_table : tbl=hive.mvs.emps
2019-12-02T20:00:25,580 INFO [pool-6-thread-195]:
metastore.ObjectStore$RetryingExecutor (ObjectStore.java:run(9966)) -
Attempting to acquire the DB log notification lock: 0 out of 10 retries
javax.jdo.JDODataStoreException: Error executing SQL query "select
"NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
~[datanucleus-api-jdo-4.2.4.jar:?]
a
[jira] [Commented] (HIVE-22508) KeyWrapperComparator throws exception
[
https://issues.apache.org/jira/browse/HIVE-22508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989264#comment-16989264
]
Hive QA commented on HIVE-22508:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987529/HIVE-22508.1.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17775 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19763/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19763/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19763/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987529 - PreCommit-HIVE-Build
> KeyWrapperComparator throws exception
> -
>
> Key: HIVE-22508
> URL: https://issues.apache.org/jira/browse/HIVE-22508
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22508.1.patch, HIVE-22508.1.patch,
> HIVE-22508.1.patch
>
>
> TopNKeyOperator.KeyWrapperComparator throws exception when a new key and a
> copied key should be compared.
> The current implementation uses the standard object inspectors for all
> KeyWrapper instances. However when comparing untouched KeyWrappers the key
> object inspector should be used which can be extracted form
> Operator.inputObjectInspectors during initialization of the key's
> ExprNodeEvaluator.
> This can cause a ClassCastException when the comparator is used collections
> like TreeSet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22508) KeyWrapperComparator throws exception
[
https://issues.apache.org/jira/browse/HIVE-22508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989242#comment-16989242
]
Hive QA commented on HIVE-22508:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} serde: The patch generated 18 new + 261 unchanged - 0
fixed = 279 total (was 261) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 11 unchanged - 1 fixed
= 12 total (was 12) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} serde in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} ql generated 0 new + 1531 unchanged - 1 fixed = 1531
total (was 1532) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19763/dev-support/hive-personality.sh
|
| git revision | master / bc7b38e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19763/yetus/diff-checkstyle-serde.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19763/yetus/diff-checkstyle-ql.txt
|
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19763/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> KeyWrapperComparator throws exception
> -
>
> Key: HIVE-22508
> URL: https://issues.apache.org/jira/browse/HIVE-22508
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22508.1.patch, HIVE-22508.1.patch,
> HIVE-22508.1.patch
>
>
> TopNKeyOperator.KeyWrapperComparator throws exception when a new key and a
> copied key
[jira] [Updated] (HIVE-22573) HMS filter metadata using Ranger plugin
[ https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Mani updated HIVE-22573: --- Attachment: HIVE-22573.4.patch > HMS filter metadata using Ranger plugin > --- > > Key: HIVE-22573 > URL: https://issues.apache.org/jira/browse/HIVE-22573 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Ramesh Mani >Priority: Major > Attachments: HIVE-22573.2.patch, HIVE-22573.3.patch, > HIVE-22573.4.patch, HIVE-22573.patch > > > HMS needs to filter out rows based on Ranger plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22573) HMS filter metadata using Ranger plugin
[
https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989221#comment-16989221
]
Hive QA commented on HIVE-22573:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987520/HIVE-22573.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19762/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19762/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19762/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12987520/HIVE-22573.3.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987520 - PreCommit-HIVE-Build
> HMS filter metadata using Ranger plugin
> ---
>
> Key: HIVE-22573
> URL: https://issues.apache.org/jira/browse/HIVE-22573
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Ramesh Mani
>Priority: Major
> Attachments: HIVE-22573.2.patch, HIVE-22573.3.patch, HIVE-22573.patch
>
>
> HMS needs to filter out rows based on Ranger plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22573) HMS filter metadata using Ranger plugin
[
https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989219#comment-16989219
]
Hive QA commented on HIVE-22573:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987520/HIVE-22573.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19761/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19761/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19761/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12987520/HIVE-22573.3.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987520 - PreCommit-HIVE-Build
> HMS filter metadata using Ranger plugin
> ---
>
> Key: HIVE-22573
> URL: https://issues.apache.org/jira/browse/HIVE-22573
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Ramesh Mani
>Priority: Major
> Attachments: HIVE-22573.2.patch, HIVE-22573.3.patch, HIVE-22573.patch
>
>
> HMS needs to filter out rows based on Ranger plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22573) HMS filter metadata using Ranger plugin
[
https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989218#comment-16989218
]
Hive QA commented on HIVE-22573:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987520/HIVE-22573.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17762 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19760/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19760/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19760/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987520 - PreCommit-HIVE-Build
> HMS filter metadata using Ranger plugin
> ---
>
> Key: HIVE-22573
> URL: https://issues.apache.org/jira/browse/HIVE-22573
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Ramesh Mani
>Priority: Major
> Attachments: HIVE-22573.2.patch, HIVE-22573.3.patch, HIVE-22573.patch
>
>
> HMS needs to filter out rows based on Ranger plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22587) hive.stats.ndv.error parameter documentation issue in HiveConf.java
[
https://issues.apache.org/jira/browse/HIVE-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pablo Junge updated HIVE-22587:
---
Attachment: HIVE-22587.patch
Status: Patch Available (was: Open)
Changed definition of HIVE_STATS_NDV_ERROR
HIVE_STATS_NDV_ERROR("hive.stats.ndv.error", (float)20.0,
"The standard error allowed for NDV estimates, expressed in percentage. This
provides a tradeoff \n" +
"between accuracy and compute cost. A lower value for the error indicates
higher accuracy and a \n" +
"higher compute cost. (NDV means the number of distinct values.). It only
affects the FM-Sketch \n" +
"(not the HLL algorithm which is the default), where it computes the number of
necessary\n" +
" bitvectors to achieve the accuracy.")
> hive.stats.ndv.error parameter documentation issue in HiveConf.java
> ---
>
> Key: HIVE-22587
> URL: https://issues.apache.org/jira/browse/HIVE-22587
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Attachments: HIVE-22587.patch
>
>
> hive.stats.ndv.error parameter documentation should specify that it only
> affects the FM-Sketch algorithm
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22587) hive.stats.ndv.error parameter documentation issue in HiveConf.java
[ https://issues.apache.org/jira/browse/HIVE-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Junge reassigned HIVE-22587: -- > hive.stats.ndv.error parameter documentation issue in HiveConf.java > --- > > Key: HIVE-22587 > URL: https://issues.apache.org/jira/browse/HIVE-22587 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Pablo Junge >Assignee: Pablo Junge >Priority: Minor > > hive.stats.ndv.error parameter documentation should specify that it only > affects the FM-Sketch algorithm -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-21174) hive.stats.ndv.error parameter documentation issue
[ https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Junge resolved HIVE-21174. Resolution: Fixed > hive.stats.ndv.error parameter documentation issue > -- > > Key: HIVE-21174 > URL: https://issues.apache.org/jira/browse/HIVE-21174 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, > 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, > 2.3.4, 3.1.2 >Reporter: Pablo Junge >Assignee: Pablo Junge >Priority: Major > Fix For: 2.0.2, 2.1.2, 2.4.0, 2.2.1, 2.3.3, 3.0.1, 3.10, 3.2.0, > 2.3.4, 3.1.1, 3.1.0, 2.3.2, 2.3.1, 3.0.0, 2.3.0, 2.2.0, 2.1.1, 2.1.0, 2.0.1, > 2.0.0 > > > Hive documentation for hive.stats.ndv.error does not specify that > hive.stats.ndv.error will only affect FM Sketch and not HLL. > > https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21174) hive.stats.ndv.error parameter documentation issue
[ https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989213#comment-16989213 ] Pablo Junge commented on HIVE-21174: Wiki updated from Current documentation shows: Standard error allowed for NDV estimates, expressed in percentage. This provides a tradeoff between accuracy and compute cost. A lower value for the error indicates higher accuracy and a higher compute cost. (NDV means number of distinct values.) To The standard error allowed for NDV estimates, expressed in percentage. This provides a tradeoff between accuracy and compute cost. A lower value for the error indicates higher accuracy and a higher compute cost. (NDV means the number of distinct values.). It only affects the FM-Sketch (not the HLL algorithm which is the default), where it computes the number of necessary bitvectors to achieve the accuracy. > hive.stats.ndv.error parameter documentation issue > -- > > Key: HIVE-21174 > URL: https://issues.apache.org/jira/browse/HIVE-21174 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, > 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, > 2.3.4, 3.1.2 >Reporter: Pablo Junge >Assignee: Pablo Junge >Priority: Major > Fix For: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, > 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, > 2.3.4 > > > Hive documentation for hive.stats.ndv.error does not specify that > hive.stats.ndv.error will only affect FM Sketch and not HLL. > > https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21174) hive.stats.ndv.error parameter documentation issue
[ https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Junge reassigned HIVE-21174: -- Assignee: (was: Pablo Junge) > hive.stats.ndv.error parameter documentation issue > -- > > Key: HIVE-21174 > URL: https://issues.apache.org/jira/browse/HIVE-21174 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, > 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, > 2.3.4, 3.1.2 >Reporter: Pablo Junge >Priority: Major > Fix For: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, > 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, > 2.3.4 > > > Hive documentation for hive.stats.ndv.error does not specify that > hive.stats.ndv.error will only affect FM Sketch and not HLL. > > https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21174) hive.stats.ndv.error parameter documentation issue
[ https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Junge reassigned HIVE-21174: -- Assignee: Pablo Junge > hive.stats.ndv.error parameter documentation issue > -- > > Key: HIVE-21174 > URL: https://issues.apache.org/jira/browse/HIVE-21174 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, > 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, > 2.3.4, 3.1.2 >Reporter: Pablo Junge >Assignee: Pablo Junge >Priority: Major > Fix For: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, > 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, > 2.3.4 > > > Hive documentation for hive.stats.ndv.error does not specify that > hive.stats.ndv.error will only affect FM Sketch and not HLL. > > https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22573) HMS filter metadata using Ranger plugin
[
https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989197#comment-16989197
]
Hive QA commented on HIVE-22573:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 222 new + 185 unchanged - 3
fixed = 407 total (was 188) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
31s{color} | {color:red} ql generated 1 new + 1532 unchanged - 0 fixed = 1533
total (was 1532) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Write to static field
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.hiveAuthorizer
from instance method
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.onEvent(PreEventContext)
At HiveMetaStoreAuthorizer.java:from instance method
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.onEvent(PreEventContext)
At HiveMetaStoreAuthorizer.java:[line 102] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19760/dev-support/hive-personality.sh
|
| git revision | master / bc7b38e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19760/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19760/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19760/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HMS filter metadata using Ranger plugin
> ---
>
> Key: HIVE-22573
> URL: https://issues.apache.org/jira/browse/HIVE-22573
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Ramesh Mani
>Priority: Major
> Attachments: HIVE-22573.2.patch, HIVE-22573.3.patch, HIVE-22573.patch
>
>
> HMS needs to filter out rows based on Ranger plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989180#comment-16989180 ] Oliver Draese commented on HIVE-22572: -- Test failures are not related to patch. Re-attached patch to trigger new test run. > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.2.patch > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22318) Java.io.exception:Two readers for
[
https://issues.apache.org/jira/browse/HIVE-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989163#comment-16989163
]
Oliver Draese commented on HIVE-22318:
--
The fix seems to be in HIVE-16812.
Minor compaction doesn't filter correctly by bucketID. You end up with multiple
delete delta files, containing same tombstone RIDs, which leads to the above
exception.
> Java.io.exception:Two readers for
> -
>
> Key: HIVE-22318
> URL: https://issues.apache.org/jira/browse/HIVE-22318
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2
>Affects Versions: 3.1.0
>Reporter: max_c
>Priority: Major
> Attachments: hiveserver2 for exception.log
>
>
> I create a ACID table with ORC format:
>
> {noformat}
> CREATE TABLE `some.TableA`(
>
>)
> ROW FORMAT SERDE
>'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
>'bucketing_version'='2',
>'orc.compress'='SNAPPY',
>'transactional'='true',
>'transactional_properties'='default'){noformat}
> After executing merge into operation:
> {noformat}
> MERGE INTO some.TableA AS a USING (SELECT vend_no FROM some.TableB UNION ALL
> SELECT vend_no FROM some.TableC) AS b ON a.vend_no=b.vend_no WHEN MATCHED
> THEN DELETE
> {noformat}
> the problem happend(when selecting the TableA, the exception happens too):
> {noformat}
> java.io.IOException: java.io.IOException: Two readers for {originalWriteId:
> 4, bucket: 536870912(1.0.0), row: 2434, currentWriteId 25}: new
> [key={originalWriteId: 4, bucket: 536870912(1.0.0), row: 2434, currentWriteId
> 25}, nextRecord={2, 4, 536870912, 2434, 25, null}, reader=Hive ORC
> Reader(hdfs://hdpprod/warehouse/tablespace/managed/hive/some.db/tableA/delete_delta_015_026/bucket_1,
> 9223372036854775807)], old [key={originalWriteId: 4, bucket:
> 536870912(1.0.0), row: 2434, currentWriteId 25}, nextRecord={2, 4, 536870912,
> 2434, 25, null}, reader=Hive ORC
> Reader(hdfs://hdpprod/warehouse/tablespace/managed/hive/some.db/tableA/delete_delta_015_026/bucket_0{noformat}
> Through orc_tools I scan all the
> files(bucket_0,bucket_1,bucket_2) under delete_delta and find all
> rows of files are the same.I think this will cause the same
> key(RecordIdentifer) when scan the bucket_1 after bucket_0 but I
> don't know why all the rows are the same in these bucket files.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[
https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989142#comment-16989142
]
Hive QA commented on HIVE-22572:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987496/HIVE-22572.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17758 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=366)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19759/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19759/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19759/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987496 - PreCommit-HIVE-Build
> NullPointerException when using dynamic semijoin reduction
> --
>
> Key: HIVE-22572
> URL: https://issues.apache.org/jira/browse/HIVE-22572
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22572.1.patch, HIVE-22572.patch
>
>
> From HS2 logs
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[
https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989091#comment-16989091
]
Hive QA commented on HIVE-22572:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
16s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19759/dev-support/hive-personality.sh
|
| git revision | master / bc7b38e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19759/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NullPointerException when using dynamic semijoin reduction
> --
>
> Key: HIVE-22572
> URL: https://issues.apache.org/jira/browse/HIVE-22572
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22572.1.patch, HIVE-22572.patch
>
>
> From HS2 logs
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at org.apache.hadoop.hive.q
[jira] [Commented] (HIVE-22395) Add ability to read Druid metastore password from jceks
[
https://issues.apache.org/jira/browse/HIVE-22395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989063#comment-16989063
]
Hive QA commented on HIVE-22395:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987492/HIVE-22395.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17758 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19758/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19758/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19758/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987492 - PreCommit-HIVE-Build
> Add ability to read Druid metastore password from jceks
> ---
>
> Key: HIVE-22395
> URL: https://issues.apache.org/jira/browse/HIVE-22395
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22395.1.patch, HIVE-22395.2.patch,
> HIVE-22395.2.patch, HIVE-22395.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22570) Review of ExprNodeDesc.java
[ https://issues.apache.org/jira/browse/HIVE-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989051#comment-16989051 ] David Mollitor commented on HIVE-22570: --- [~vgarg] Can you also review this one please? Thanks! > Review of ExprNodeDesc.java > --- > > Key: HIVE-22570 > URL: https://issues.apache.org/jira/browse/HIVE-22570 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22570.1.patch, HIVE-22570.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22577) StringIndexOutOfBoundsException when getting sessionId from worker node name
[
https://issues.apache.org/jira/browse/HIVE-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22577:
-
Status: Open (was: Patch Available)
> StringIndexOutOfBoundsException when getting sessionId from worker node name
>
>
> Key: HIVE-22577
> URL: https://issues.apache.org/jira/browse/HIVE-22577
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22577.1.patch, HIVE-22577.2.patch
>
>
> When the node name is "worker-" the following exception is thrown
>
> {code:java}
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index
> out of range: -1 at java.lang.String.substring(String.java:1931) at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
> at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl$DynamicServiceInstanceSet.(LlapZookeeperRegistryImpl.java:313)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:462)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getApplicationId(LlapZookeeperRegistryImpl.java:469)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getApplicationId(LlapRegistryService.java:212)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getCustomSplitLocationProvider(Utils.java:77)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:53)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:140)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22577) StringIndexOutOfBoundsException when getting sessionId from worker node name
[
https://issues.apache.org/jira/browse/HIVE-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22577:
-
Attachment: HIVE-22577.2.patch
> StringIndexOutOfBoundsException when getting sessionId from worker node name
>
>
> Key: HIVE-22577
> URL: https://issues.apache.org/jira/browse/HIVE-22577
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22577.1.patch, HIVE-22577.2.patch
>
>
> When the node name is "worker-" the following exception is thrown
>
> {code:java}
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index
> out of range: -1 at java.lang.String.substring(String.java:1931) at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
> at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl$DynamicServiceInstanceSet.(LlapZookeeperRegistryImpl.java:313)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:462)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getApplicationId(LlapZookeeperRegistryImpl.java:469)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getApplicationId(LlapRegistryService.java:212)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getCustomSplitLocationProvider(Utils.java:77)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:53)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:140)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22577) StringIndexOutOfBoundsException when getting sessionId from worker node name
[
https://issues.apache.org/jira/browse/HIVE-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22577:
-
Status: Patch Available (was: Open)
> StringIndexOutOfBoundsException when getting sessionId from worker node name
>
>
> Key: HIVE-22577
> URL: https://issues.apache.org/jira/browse/HIVE-22577
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22577.1.patch, HIVE-22577.2.patch
>
>
> When the node name is "worker-" the following exception is thrown
>
> {code:java}
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index
> out of range: -1 at java.lang.String.substring(String.java:1931) at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
> at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl$DynamicServiceInstanceSet.(LlapZookeeperRegistryImpl.java:313)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:462)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getApplicationId(LlapZookeeperRegistryImpl.java:469)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getApplicationId(LlapRegistryService.java:212)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getCustomSplitLocationProvider(Utils.java:77)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:53)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:140)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19918) IllegalArgumentException when getContext().getVertexTaskResource().getMemory() returns -1
[
https://issues.apache.org/jira/browse/HIVE-19918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989042#comment-16989042
]
Daniil Bastrich commented on HIVE-19918:
But it would be useful to add at least a LOG message in this case. Because the
reason is not obvious initially.
Can we create a task for that?
> IllegalArgumentException when
> getContext().getVertexTaskResource().getMemory() returns -1
> -
>
> Key: HIVE-19918
> URL: https://issues.apache.org/jira/browse/HIVE-19918
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Priority: Major
>
> {code}
> Beeline version 4.0.0-SNAPSHOT by Apache Hive
> ;: jdbc:hive2://hs2.example.com:1/> CREATE TABLE pokes (foo INT, bar
> STRING)
> No rows affected (3.834 seconds)
> G) PARTITIONED BY (ds STRING);m:1/> CREATE TABLE invites (foo INT, bar
> STRIN
> No rows affected (0.175 seconds)
> 0: jdbc:hive2://hs2.example.com:1/> SELECT * FROM pokes, invites;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
> vertexName=Map 1, vertexId=vertex_1529099908701_0003_1_01,
> diagnostics=[Vertex vertex_1529099908701_0003_1_01 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: pokes initializer failed,
> vertex=vertex_1529099908701_0003_1_01 [Map 1],
> java.lang.IllegalArgumentException: Illegal Capacity: -10444
> at java.util.ArrayList.(ArrayList.java:157)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:339)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:519)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:768)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:211)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> ]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
> (state=08S01,code=2)
> {code}
> This is happening because
> [here|https://github.com/apache/hive/blob/bceb3dd82cbeb0334c3c33da7a21a74ad631810b/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java#L188]
> the return is -1 and afterwards we request a negative number of splits.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22395) Add ability to read Druid metastore password from jceks
[
https://issues.apache.org/jira/browse/HIVE-22395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989027#comment-16989027
]
Hive QA commented on HIVE-22395:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19758/dev-support/hive-personality.sh
|
| git revision | master / bc7b38e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: druid-handler U: druid-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19758/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add ability to read Druid metastore password from jceks
> ---
>
> Key: HIVE-22395
> URL: https://issues.apache.org/jira/browse/HIVE-22395
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22395.1.patch, HIVE-22395.2.patch,
> HIVE-22395.2.patch, HIVE-22395.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22570) Review of ExprNodeDesc.java
[
https://issues.apache.org/jira/browse/HIVE-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989005#comment-16989005
]
Hive QA commented on HIVE-22570:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987488/HIVE-22570.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17758 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19757/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19757/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19757/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987488 - PreCommit-HIVE-Build
> Review of ExprNodeDesc.java
> ---
>
> Key: HIVE-22570
> URL: https://issues.apache.org/jira/browse/HIVE-22570
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22570.1.patch, HIVE-22570.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21569) Bump guava version to 28.1-jre
[ https://issues.apache.org/jira/browse/HIVE-21569?focusedWorklogId=354461&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354461 ] ASF GitHub Bot logged work on HIVE-21569: - Author: ASF GitHub Bot Created on: 05/Dec/19 16:51 Start Date: 05/Dec/19 16:51 Worklog Time Spent: 10m Work Description: dlavati commented on pull request #861: HIVE-21569 Bump guava version to 28.1-jre URL: https://github.com/apache/hive/pull/861 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354461) Remaining Estimate: 0h Time Spent: 10m > Bump guava version to 28.1-jre > -- > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch, > HIVE-21569.03.patch, HIVE-21569.04.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Changes required to bump guava to 28.1-jre -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21569) Bump guava version to 28.1-jre
[ https://issues.apache.org/jira/browse/HIVE-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21569: -- Labels: pull-request-available (was: ) > Bump guava version to 28.1-jre > -- > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch, > HIVE-21569.03.patch, HIVE-21569.04.patch > > > Changes required to bump guava to 28.1-jre -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22571) Review of ExprNodeFieldDesc Class
[ https://issues.apache.org/jira/browse/HIVE-22571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22571: -- Status: Patch Available (was: Open) > Review of ExprNodeFieldDesc Class > - > > Key: HIVE-22571 > URL: https://issues.apache.org/jira/browse/HIVE-22571 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22571.1.patch, HIVE-22571.1.patch, > HIVE-22571.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22571) Review of ExprNodeFieldDesc Class
[ https://issues.apache.org/jira/browse/HIVE-22571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22571: -- Attachment: HIVE-22571.1.patch > Review of ExprNodeFieldDesc Class > - > > Key: HIVE-22571 > URL: https://issues.apache.org/jira/browse/HIVE-22571 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22571.1.patch, HIVE-22571.1.patch, > HIVE-22571.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22571) Review of ExprNodeFieldDesc Class
[ https://issues.apache.org/jira/browse/HIVE-22571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22571: -- Status: Open (was: Patch Available) > Review of ExprNodeFieldDesc Class > - > > Key: HIVE-22571 > URL: https://issues.apache.org/jira/browse/HIVE-22571 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22571.1.patch, HIVE-22571.1.patch, > HIVE-22571.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22570) Review of ExprNodeDesc.java
[
https://issues.apache.org/jira/browse/HIVE-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988958#comment-16988958
]
Hive QA commented on HIVE-22570:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
29s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19757/dev-support/hive-personality.sh
|
| git revision | master / bc7b38e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19757/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review of ExprNodeDesc.java
> ---
>
> Key: HIVE-22570
> URL: https://issues.apache.org/jira/browse/HIVE-22570
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22570.1.patch, HIVE-22570.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22569) PartitionPruner use Collections Class
[
https://issues.apache.org/jira/browse/HIVE-22569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22569:
--
Status: Patch Available (was: Open)
> PartitionPruner use Collections Class
> -
>
> Key: HIVE-22569
> URL: https://issues.apache.org/jira/browse/HIVE-22569
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22569.2.patch, HIVE-22569.2.patch,
> HIVE.22569.1.patch
>
>
> Make class {{PrunedPartitionList.java}} more immutable. It kinda is already.
> Immutable classes are easier to reason about.
> Use {{Collections.emptyXXX}} in {{PartitionPruner}} to save an object
> instantiation at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22569) PartitionPruner use Collections Class
[
https://issues.apache.org/jira/browse/HIVE-22569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22569:
--
Attachment: HIVE-22569.2.patch
> PartitionPruner use Collections Class
> -
>
> Key: HIVE-22569
> URL: https://issues.apache.org/jira/browse/HIVE-22569
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22569.2.patch, HIVE-22569.2.patch,
> HIVE.22569.1.patch
>
>
> Make class {{PrunedPartitionList.java}} more immutable. It kinda is already.
> Immutable classes are easier to reason about.
> Use {{Collections.emptyXXX}} in {{PartitionPruner}} to save an object
> instantiation at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22569) PartitionPruner use Collections Class
[
https://issues.apache.org/jira/browse/HIVE-22569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22569:
--
Status: Open (was: Patch Available)
> PartitionPruner use Collections Class
> -
>
> Key: HIVE-22569
> URL: https://issues.apache.org/jira/browse/HIVE-22569
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22569.2.patch, HIVE-22569.2.patch,
> HIVE.22569.1.patch
>
>
> Make class {{PrunedPartitionList.java}} more immutable. It kinda is already.
> Immutable classes are easier to reason about.
> Use {{Collections.emptyXXX}} in {{PartitionPruner}} to save an object
> instantiation at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22584) Flakyness in TestTaskExecutorService.testSetCapacity
[
https://issues.apache.org/jira/browse/HIVE-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988947#comment-16988947
]
Oliver Draese commented on HIVE-22584:
--
+1 LGTM
> Flakyness in TestTaskExecutorService.testSetCapacity
>
>
> Key: HIVE-22584
> URL: https://issues.apache.org/jira/browse/HIVE-22584
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22584.patch
>
>
> Very rarely the test fails:
> {code}
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity(TestTaskExecutorService.java:515)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> See:
> https://builds.apache.org/job/PreCommit-HIVE-Build/19739/testReport/org.apache.hadoop.hive.llap.daemon.impl/TestTaskExecutorService/testSetCapacity/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22569) PartitionPruner use Collections Class
[
https://issues.apache.org/jira/browse/HIVE-22569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988931#comment-16988931
]
Hive QA commented on HIVE-22569:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987481/HIVE-22569.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17758 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=366)
org.apache.hive.service.server.TestHS2HttpServer.testApiServletActiveSessions
(batchId=241)
org.apache.hive.service.server.TestHS2HttpServer.testApiServletHistoricalQueries
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19756/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19756/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19756/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987481 - PreCommit-HIVE-Build
> PartitionPruner use Collections Class
> -
>
> Key: HIVE-22569
> URL: https://issues.apache.org/jira/browse/HIVE-22569
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22569.2.patch, HIVE.22569.1.patch
>
>
> Make class {{PrunedPartitionList.java}} more immutable. It kinda is already.
> Immutable classes are easier to reason about.
> Use {{Collections.emptyXXX}} in {{PartitionPruner}} to save an object
> instantiation at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22568) Process compaction candidates in parallel by the Initiator
[ https://issues.apache.org/jira/browse/HIVE-22568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22568: -- Status: Patch Available (was: Open) > Process compaction candidates in parallel by the Initiator > -- > > Key: HIVE-22568 > URL: https://issues.apache.org/jira/browse/HIVE-22568 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22568.1.patch > > > `checkForCompaction` includes many file metadata checks and may be expensive. > Therefore, make sense using a thread pool here and running > `checkForCompactions` in parallel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22568) Process compaction candidates in parallel by the Initiator
[ https://issues.apache.org/jira/browse/HIVE-22568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22568: -- Attachment: (was: HIVE-22568.1.patch) > Process compaction candidates in parallel by the Initiator > -- > > Key: HIVE-22568 > URL: https://issues.apache.org/jira/browse/HIVE-22568 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22568.1.patch > > > `checkForCompaction` includes many file metadata checks and may be expensive. > Therefore, make sense using a thread pool here and running > `checkForCompactions` in parallel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22568) Process compaction candidates in parallel by the Initiator
[ https://issues.apache.org/jira/browse/HIVE-22568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22568: -- Attachment: HIVE-22568.1.patch > Process compaction candidates in parallel by the Initiator > -- > > Key: HIVE-22568 > URL: https://issues.apache.org/jira/browse/HIVE-22568 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22568.1.patch > > > `checkForCompaction` includes many file metadata checks and may be expensive. > Therefore, make sense using a thread pool here and running > `checkForCompactions` in parallel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22568) Process compaction candidates in parallel by the Initiator
[ https://issues.apache.org/jira/browse/HIVE-22568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22568: -- Attachment: HIVE-22568.1.patch > Process compaction candidates in parallel by the Initiator > -- > > Key: HIVE-22568 > URL: https://issues.apache.org/jira/browse/HIVE-22568 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22568.1.patch > > > `checkForCompaction` includes many file metadata checks and may be expensive. > Therefore, make sense using a thread pool here and running > `checkForCompactions` in parallel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22568) Process compaction candidates in parallel by the Initiator
[ https://issues.apache.org/jira/browse/HIVE-22568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22568: -- Attachment: (was: HIVE-22568.1.patch) > Process compaction candidates in parallel by the Initiator > -- > > Key: HIVE-22568 > URL: https://issues.apache.org/jira/browse/HIVE-22568 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22568.1.patch > > > `checkForCompaction` includes many file metadata checks and may be expensive. > Therefore, make sense using a thread pool here and running > `checkForCompactions` in parallel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21569) Bump guava version to 28.1-jre
[ https://issues.apache.org/jira/browse/HIVE-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-21569: Attachment: HIVE-21569.04.patch > Bump guava version to 28.1-jre > -- > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch, > HIVE-21569.03.patch, HIVE-21569.04.patch > > > Changes required to bump guava to 28.1-jre -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22553) Expose locks and transactions in sys db
[ https://issues.apache.org/jira/browse/HIVE-22553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22553: - Attachment: HIVE-22553.3.patch > Expose locks and transactions in sys db > --- > > Key: HIVE-22553 > URL: https://issues.apache.org/jira/browse/HIVE-22553 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22553.2.patch, HIVE-22553.3.patch, > HIVE-22553.3.patch, HIVE-22553.patch > > > Create new sysdb tables/views to access lock and transaction data. > This allows to provide admins with live data about ongoing locks and > transacions. Due to this being in the sys db access to this information can > be restricted to select privileged users. > Information about locks and compactions can be joined and accessed at the > same time. > Compaction related transactions would also be visible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22355) Beeline should not prompt for hive user and password when authentication is NONE
[ https://issues.apache.org/jira/browse/HIVE-22355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988897#comment-16988897 ] Naveen Gangam commented on HIVE-22355: -- [~matijhs] Just tuning in. I am trying to understand the problem we are seeing here. I understand that beeline prompts for password even when auth=NONE. Are we trying to resolve this issue specifically for sqoop or for something else? can we not use one of the workarounds to resolve this 1) beeline accepts a "-n" and a "-p" as command line arguments for username and password. we could use "" for both or any other values. 2) the beeline url itself can contain "user=" and "password=" as well. 3) Sqoop can also use beeline-site.xml file with named URLs in it or a .beeline.properties. So when it runs beeline, its not prompted for credentials regardless of the AUTH. Having said that, (apologies for taking a different direction) beeline should support the option for no credentials as JDBC supports DriverManager.getConnect(url) with just the URL. But should this be irrespective of AUTH value set in config or URL. This will allow the connection to fail if the server rejects it, instead of defaulting to something the user is unaware of. I have to give this a bit more thought but just throwing it out there for discussion. Thanks > Beeline should not prompt for hive user and password when authentication is > NONE > > > Key: HIVE-22355 > URL: https://issues.apache.org/jira/browse/HIVE-22355 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Mate Juhasz >Assignee: Mate Juhasz >Priority: Major > Attachments: HIVE-22355.1.patch, HIVE-22355.2.patch, > HIVE-22355.3.patch > > > Beeline - without adding the jdbc url - prompts for username and password in > case hive.server2.authentication=NONE, which is possibly pointless and can be > misleading for users as any input is accepted. > In addition, Sqoop has dropped hive cli recently in favor of beeline and if > there is no authentication set in Hive, Sqoop fails to connect as the process > stops waiting for the user/password input. > I think it would be nice to check the auth type "NONE" before reading unused > inputs from the console before this point: > https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/Commands.java#L1641 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22569) PartitionPruner use Collections Class
[
https://issues.apache.org/jira/browse/HIVE-22569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988895#comment-16988895
]
Hive QA commented on HIVE-22569:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
30s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19756/dev-support/hive-personality.sh
|
| git revision | master / bc7b38e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19756/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> PartitionPruner use Collections Class
> -
>
> Key: HIVE-22569
> URL: https://issues.apache.org/jira/browse/HIVE-22569
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22569.2.patch, HIVE.22569.1.patch
>
>
> Make class {{PrunedPartitionList.java}} more immutable. It kinda is already.
> Immutable classes are easier to reason about.
> Use {{Collections.emptyXXX}} in {{PartitionPruner}} to save an object
> instantiation at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-20843) RELY constraints on primary keys and foreign keys are not recognized
[ https://issues.apache.org/jira/browse/HIVE-20843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-20843: -- Fix Version/s: 3.0.0 Resolution: Duplicate Status: Resolved (was: Patch Available) HIVE-16575 fixes this issue > RELY constraints on primary keys and foreign keys are not recognized > > > Key: HIVE-20843 > URL: https://issues.apache.org/jira/browse/HIVE-20843 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 3.0.0 >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-20843.1-branch-2.patch, HIVE-20843.1.patch > > > Hive doesn't recognize RELY constraints after > https://issues.apache.org/jira/browse/HIVE-13076. The issue is in > BaseSemanticAnalyzer.java where we assign RELY. > An unrelated patch fixed this issue in later versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22571) Review of ExprNodeFieldDesc Class
[
https://issues.apache.org/jira/browse/HIVE-22571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988859#comment-16988859
]
Hive QA commented on HIVE-22571:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987480/HIVE-22571.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17754 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACIDNoAutogather - did not produce a TEST-*.xml
file (likely timed out) (batchId=257)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19755/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19755/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19755/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987480 - PreCommit-HIVE-Build
> Review of ExprNodeFieldDesc Class
> -
>
> Key: HIVE-22571
> URL: https://issues.apache.org/jira/browse/HIVE-22571
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22571.1.patch, HIVE-22571.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22571) Review of ExprNodeFieldDesc Class
[
https://issues.apache.org/jira/browse/HIVE-22571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988816#comment-16988816
]
Hive QA commented on HIVE-22571:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
33s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 0 unchanged - 0 fixed
= 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19755/dev-support/hive-personality.sh
|
| git revision | master / bc7b38e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19755/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19755/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review of ExprNodeFieldDesc Class
> -
>
> Key: HIVE-22571
> URL: https://issues.apache.org/jira/browse/HIVE-22571
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22571.1.patch, HIVE-22571.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[
https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988815#comment-16988815
]
Peter Vary commented on HIVE-22538:
---
[~jcamachorodriguez]: Any news here?
> RS deduplication does not always enforce
> hive.optimize.reducededuplication.min.reducer
> --
>
> Key: HIVE-22538
> URL: https://issues.apache.org/jira/browse/HIVE-22538
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22538.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For transactional tables, that property might be overriden to 1, which can
> lead to merging final aggregation into a single stage (hence leading to
> performance degradation). For instance, when autogather column stats is
> enabled, this can happen for the following query:
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> EXPLAIN
> CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS
> SELECT * FROM SRC x CLUSTER BY x.key;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22576) tez.queue.name setting is invalid when no default queue is set on yarn
[
https://issues.apache.org/jira/browse/HIVE-22576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hezhang updated HIVE-22576:
---
Priority: Blocker (was: Critical)
> tez.queue.name setting is invalid when no default queue is set on yarn
> ---
>
> Key: HIVE-22576
> URL: https://issues.apache.org/jira/browse/HIVE-22576
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.1, 2.3.2
>Reporter: hezhang
>Assignee: hezhang
>Priority: Blocker
> Fix For: 2.3.2
>
> Attachments: HIVE-22576.1.patch, image-2019-12-04-15-49-03-475.png
>
>
> h2. hive cli started:
> hive> set tez.queue.name=prod;
> hive>
> > select count(*) from test;
> Query ID = hadoop_20191204152859_d9f7fff6-c0a1-4984-9f56-dd958419992b
> Total jobs = 1
> Launching Job 1 out of 1
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
> err stack
>
> {code:java}
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.tez.dag.api.TezException:
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1575443228569_0002 to YARN : Application
> application_1575443228569_0002 submitted by user hadoop to unknown queue:
> default
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:163)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:144)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_191]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_191]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_191]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> ~[hadoop-common-2.7.3.jar:?]
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> ~[hadoop-common-2.7.3.jar:?]
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.tez.dag.api.TezException:
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1575443228569_0002 to YARN : Application
> application_1575443228569_0002 submitted by user hadoop to unknown queue:
> default
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[?:1.8.0_191]
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> ~[?:1.8.0_191]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:158)
> ~[hive-exec-2.3.2.jar:2.3.2]
> ... 20 more
> Caused by: org.apache.tez.dag.api.TezException:
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1575443228569_0002 to YARN : Application
> application_1575443228569_0002 submitted by user hadoop to unknown queue:
> default
> at org.apache.tez.client.TezClient.start(TezClient.java:492)
> ~[tez-api-0.8.5.jar:0.8.5]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:376)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:87)
>
[jira] [Assigned] (HIVE-22585) Clean up catalog/db/table name usage
[ https://issues.apache.org/jira/browse/HIVE-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati reassigned HIVE-22585: --- > Clean up catalog/db/table name usage > > > Key: HIVE-22585 > URL: https://issues.apache.org/jira/browse/HIVE-22585 > Project: Hive > Issue Type: Sub-task >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: refactor > > This is a followup to HIVE-21198 to address some additional improvement ideas > for the TableName object mentioned in > [https://github.com/apache/hive/pull/550] and attempt to remove all the fishy > usages of db/tablenames, as a number of places still rely on certain state > changes/black magic. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22577) StringIndexOutOfBoundsException when getting sessionId from worker node name
[
https://issues.apache.org/jira/browse/HIVE-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988776#comment-16988776
]
Hive QA commented on HIVE-22577:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987453/HIVE-22577.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17758 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19754/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19754/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19754/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987453 - PreCommit-HIVE-Build
> StringIndexOutOfBoundsException when getting sessionId from worker node name
>
>
> Key: HIVE-22577
> URL: https://issues.apache.org/jira/browse/HIVE-22577
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22577.1.patch
>
>
> When the node name is "worker-" the following exception is thrown
>
> {code:java}
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index
> out of range: -1 at java.lang.String.substring(String.java:1931) at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
> at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl$DynamicServiceInstanceSet.(LlapZookeeperRegistryImpl.java:313)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:462)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getApplicationId(LlapZookeeperRegistryImpl.java:469)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getApplicationId(LlapRegistryService.java:212)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getCustomSplitLocationProvider(Utils.java:77)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:53)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:140)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22584) Flakyness in TestTaskExecutorService.testSetCapacity
[
https://issues.apache.org/jira/browse/HIVE-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-22584:
--
Attachment: HIVE-22584.patch
> Flakyness in TestTaskExecutorService.testSetCapacity
>
>
> Key: HIVE-22584
> URL: https://issues.apache.org/jira/browse/HIVE-22584
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22584.patch
>
>
> Very rarely the test fails:
> {code}
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity(TestTaskExecutorService.java:515)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> See:
> https://builds.apache.org/job/PreCommit-HIVE-Build/19739/testReport/org.apache.hadoop.hive.llap.daemon.impl/TestTaskExecutorService/testSetCapacity/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22584) Flakyness in TestTaskExecutorService.testSetCapacity
[
https://issues.apache.org/jira/browse/HIVE-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-22584:
--
Status: Patch Available (was: Open)
Submitted the patch.
Thansk [~odraese] for pointing out.
> Flakyness in TestTaskExecutorService.testSetCapacity
>
>
> Key: HIVE-22584
> URL: https://issues.apache.org/jira/browse/HIVE-22584
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22584.patch
>
>
> Very rarely the test fails:
> {code}
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity(TestTaskExecutorService.java:515)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> See:
> https://builds.apache.org/job/PreCommit-HIVE-Build/19739/testReport/org.apache.hadoop.hive.llap.daemon.impl/TestTaskExecutorService/testSetCapacity/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-21569) Bump guava version to 28.1-jre
[ https://issues.apache.org/jira/browse/HIVE-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21569 started by David Lavati. --- > Bump guava version to 28.1-jre > -- > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch, > HIVE-21569.03.patch > > > Changes required to bump guava to 28.1-jre -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22584) Flakyness in TestTaskExecutorService.testSetCapacity
[
https://issues.apache.org/jira/browse/HIVE-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-22584:
-
> Flakyness in TestTaskExecutorService.testSetCapacity
>
>
> Key: HIVE-22584
> URL: https://issues.apache.org/jira/browse/HIVE-22584
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
>
> Very rarely the test fails:
> {code}
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity(TestTaskExecutorService.java:515)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> See:
> https://builds.apache.org/job/PreCommit-HIVE-Build/19739/testReport/org.apache.hadoop.hive.llap.daemon.impl/TestTaskExecutorService/testSetCapacity/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22308) Add missing support of Azure Blobstore schemes
[
https://issues.apache.org/jira/browse/HIVE-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988768#comment-16988768
]
David Lavati commented on HIVE-22308:
-
I missed the fact that the UriSchemes file mentioned in the description is only
available from hadoop 3.2.0, currently we're at 3.1.0.
> Add missing support of Azure Blobstore schemes
> --
>
> Key: HIVE-22308
> URL: https://issues.apache.org/jira/browse/HIVE-22308
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22308.02.patch, HIVE-22308.02.patch,
> HIVE-22308.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Azure has been used as a filesystem for Hive, but its various schemes aren't
> registered under
> {{HiveConf.HIVE_BLOBSTORE_SUPPORTED_SCHEMES.}}
> Found the list of elements in:
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemUriSchemes.java
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-20972) Enable TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
[ https://issues.apache.org/jira/browse/HIVE-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-20972: Attachment: HIVE-20972.02.patch > Enable TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit] > -- > > Key: HIVE-20972 > URL: https://issues.apache.org/jira/browse/HIVE-20972 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20972.01.patch, HIVE-20972.02.patch, > HIVE-20972.02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21569) Bump guava version to 28.1-jre
[ https://issues.apache.org/jira/browse/HIVE-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988764#comment-16988764 ] David Lavati edited comment on HIVE-21569 at 12/5/19 12:49 PM: --- The guava 14 copy for Spark introduced [here|https://github.com/apache/hive/commit/b0daad1fefa46c8ad2a9e076db343d19ae840342#diff-558075ec170fa171d85802433e3083abR65-R116] might need to get shaded. was (Author: dlavati): The guava 14 copy for Spark introduced [here [https://github.com/apache/hive/commit/b0daad1fefa46c8ad2a9e076db343d19ae840342#diff-558075ec170fa171d85802433e3083abR65-R116] ] might need to get shaded. > Bump guava version to 28.1-jre > -- > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch, > HIVE-21569.03.patch > > > Changes required to bump guava to 28.1-jre -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21569) Bump guava version to 28.1-jre
[ https://issues.apache.org/jira/browse/HIVE-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988764#comment-16988764 ] David Lavati commented on HIVE-21569: - The guava 14 copy for Spark introduced [here [https://github.com/apache/hive/commit/b0daad1fefa46c8ad2a9e076db343d19ae840342#diff-558075ec170fa171d85802433e3083abR65-R116] ] might need to get shaded. > Bump guava version to 28.1-jre > -- > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch, > HIVE-21569.03.patch > > > Changes required to bump guava to 28.1-jre -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21569) Bump guava version to 28.1-jre
[ https://issues.apache.org/jira/browse/HIVE-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-21569: Attachment: HIVE-21569.03.patch > Bump guava version to 28.1-jre > -- > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch, > HIVE-21569.03.patch > > > Changes required to bump guava to 28.1-jre -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[
https://issues.apache.org/jira/browse/HIVE-21213?focusedWorklogId=354254&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354254
]
ASF GitHub Bot logged work on HIVE-21213:
-
Author: ASF GitHub Bot
Created on: 05/Dec/19 12:36
Start Date: 05/Dec/19 12:36
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #587:
HIVE-21213 : Acid table bootstrap replication needs to handle directory created
by compaction with txn id
URL: https://github.com/apache/hive/pull/587#discussion_r354280694
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -463,7 +464,29 @@ public static Path
getCopyDestination(ReplChangeManager.FileInfo fileInfo, Path
String[] subDirs = fileInfo.getSubDir().split(Path.SEPARATOR);
Path destination = destRoot;
for (String subDir: subDirs) {
- destination = new Path(destination, subDir);
+ // If the directory is created by compactor, then the directory will
have the transaction id also.
+ // In case of replication, the same txn id can not be used at target, as
the txn with same id might be a
+ // aborted or live txn at target.
+ // In case of bootstrap load, we copy only the committed data, so the
directory with only write id
+ // can be created. The validity txn id can be removed from the directory
name.
+ // TODO : Support for incremental load flow. This can be done once
replication of compaction is decided.
+ if (AcidUtils.getVisibilityTxnId(subDir) > 0) {
Review comment:
What does it mean to have visibility txn id to be less than 0?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354254)
Time Spent: 40m (was: 0.5h)
> Acid table bootstrap replication needs to handle directory created by
> compaction with txn id
>
>
> Key: HIVE-21213
> URL: https://issues.apache.org/jira/browse/HIVE-21213
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21213.01.patch, HIVE-21213.02.patch,
> HIVE-21213.03.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The current implementation of compaction uses the txn id in the directory
> name. This is used to isolate the queries from reading the directory until
> compaction has finished and to avoid the compactor marking used earlier. In
> case of replication, during bootstrap , directory is copied as it is with the
> same name from source to destination cluster. But the directory created by
> compaction with txn id can not be copied as the txn list at target may be
> different from source. The txn id which is valid at source may be an aborted
> txn at target. So conversion logic is required to create a new directory with
> valid txn at target and dump the data to the newly created directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[
https://issues.apache.org/jira/browse/HIVE-21213?focusedWorklogId=354258&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354258
]
ASF GitHub Bot logged work on HIVE-21213:
-
Author: ASF GitHub Bot
Created on: 05/Dec/19 12:36
Start Date: 05/Dec/19 12:36
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #587:
HIVE-21213 : Acid table bootstrap replication needs to handle directory created
by compaction with txn id
URL: https://github.com/apache/hive/pull/587#discussion_r354286755
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -670,4 +678,63 @@ public void testMultiDBTxn() throws Throwable {
replica.run("drop database " + dbName1 + " cascade");
replica.run("drop database " + dbName2 + " cascade");
}
+
+ private void runCompaction(String dbName, String tblName, CompactionType
compactionType) throws Throwable {
+HiveConf hiveConf = new HiveConf(primary.getConf());
+TxnStore txnHandler = TxnUtils.getTxnStore(hiveConf);
+txnHandler.compact(new CompactionRequest(dbName, tblName, compactionType));
+hiveConf.setBoolVar(HiveConf.ConfVars.COMPACTOR_CRUD_QUERY_BASED, false);
+runWorker(hiveConf);
+runCleaner(hiveConf);
+ }
+
+ private FileStatus[] getDirsInTableLoc(WarehouseInstance wh, String db,
String table) throws Throwable {
+Path tblLoc = new Path(wh.getTable(db, table).getSd().getLocation());
+FileSystem fs = tblLoc.getFileSystem(wh.getConf());
+return fs.listStatus(tblLoc, EximUtil.getDirectoryFilter(fs));
+ }
+
+ @Test
+ public void testAcidTablesBootstrapWithCompaction() throws Throwable {
+ String tableName = testName.getMethodName();
+ primary.run("use " + primaryDbName)
+.run("create table " + tableName + " (id int) clustered by(id)
into 3 buckets stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("insert into " + tableName + " values(1)")
+.run("insert into " + tableName + " values(2)");
+runCompaction(primaryDbName, tableName, CompactionType.MAJOR);
Review comment:
What about a minor compaction?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354258)
> Acid table bootstrap replication needs to handle directory created by
> compaction with txn id
>
>
> Key: HIVE-21213
> URL: https://issues.apache.org/jira/browse/HIVE-21213
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21213.01.patch, HIVE-21213.02.patch,
> HIVE-21213.03.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The current implementation of compaction uses the txn id in the directory
> name. This is used to isolate the queries from reading the directory until
> compaction has finished and to avoid the compactor marking used earlier. In
> case of replication, during bootstrap , directory is copied as it is with the
> same name from source to destination cluster. But the directory created by
> compaction with txn id can not be copied as the txn list at target may be
> different from source. The txn id which is valid at source may be an aborted
> txn at target. So conversion logic is required to create a new directory with
> valid txn at target and dump the data to the newly created directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[
https://issues.apache.org/jira/browse/HIVE-21213?focusedWorklogId=354255&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354255
]
ASF GitHub Bot logged work on HIVE-21213:
-
Author: ASF GitHub Bot
Created on: 05/Dec/19 12:36
Start Date: 05/Dec/19 12:36
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #587:
HIVE-21213 : Acid table bootstrap replication needs to handle directory created
by compaction with txn id
URL: https://github.com/apache/hive/pull/587#discussion_r354281539
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
##
@@ -1467,7 +1467,7 @@ public void
replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaEx
// Schedule Major compaction on all the partitions/table to clean aborted
data
if (numAbortedWrites > 0) {
- CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(),
rqst.getTableName(),
+ CompactionRequest compactRqst = new CompactionRequest(dbName, tblName,
Review comment:
Why do we need this change here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354255)
Time Spent: 50m (was: 40m)
> Acid table bootstrap replication needs to handle directory created by
> compaction with txn id
>
>
> Key: HIVE-21213
> URL: https://issues.apache.org/jira/browse/HIVE-21213
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21213.01.patch, HIVE-21213.02.patch,
> HIVE-21213.03.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The current implementation of compaction uses the txn id in the directory
> name. This is used to isolate the queries from reading the directory until
> compaction has finished and to avoid the compactor marking used earlier. In
> case of replication, during bootstrap , directory is copied as it is with the
> same name from source to destination cluster. But the directory created by
> compaction with txn id can not be copied as the txn list at target may be
> different from source. The txn id which is valid at source may be an aborted
> txn at target. So conversion logic is required to create a new directory with
> valid txn at target and dump the data to the newly created directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[
https://issues.apache.org/jira/browse/HIVE-21213?focusedWorklogId=354252&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354252
]
ASF GitHub Bot logged work on HIVE-21213:
-
Author: ASF GitHub Bot
Created on: 05/Dec/19 12:36
Start Date: 05/Dec/19 12:36
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #587:
HIVE-21213 : Acid table bootstrap replication needs to handle directory created
by compaction with txn id
URL: https://github.com/apache/hive/pull/587#discussion_r354277500
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -463,7 +464,29 @@ public static Path
getCopyDestination(ReplChangeManager.FileInfo fileInfo, Path
String[] subDirs = fileInfo.getSubDir().split(Path.SEPARATOR);
Path destination = destRoot;
for (String subDir: subDirs) {
- destination = new Path(destination, subDir);
+ // If the directory is created by compactor, then the directory will
have the transaction id also.
+ // In case of replication, the same txn id can not be used at target, as
the txn with same id might be a
+ // aborted or live txn at target.
+ // In case of bootstrap load, we copy only the committed data, so the
directory with only write id
+ // can be created. The validity txn id can be removed from the directory
name.
+ // TODO : Support for incremental load flow. This can be done once
replication of compaction is decided.
+ if (AcidUtils.getVisibilityTxnId(subDir) > 0) {
+if (subDir.startsWith(AcidUtils.BASE_PREFIX)) {
+ AcidUtils.ParsedBase pb = AcidUtils.ParsedBase.parseBase(new
Path(subDir));
+ destination = new Path(destination,
AcidUtils.baseDir(pb.getWriteId()));
Review comment:
Looks like, we are removing the transaction id from the compactor and just
leaving writeId there. Are we replicating the corresponding transaction? How do
we know whether this writeId is visible. I am assuming that a failed compaction
will leave a directory behind but its writeId won't be visible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354252)
Time Spent: 0.5h (was: 20m)
> Acid table bootstrap replication needs to handle directory created by
> compaction with txn id
>
>
> Key: HIVE-21213
> URL: https://issues.apache.org/jira/browse/HIVE-21213
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21213.01.patch, HIVE-21213.02.patch,
> HIVE-21213.03.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The current implementation of compaction uses the txn id in the directory
> name. This is used to isolate the queries from reading the directory until
> compaction has finished and to avoid the compactor marking used earlier. In
> case of replication, during bootstrap , directory is copied as it is with the
> same name from source to destination cluster. But the directory created by
> compaction with txn id can not be copied as the txn list at target may be
> different from source. The txn id which is valid at source may be an aborted
> txn at target. So conversion logic is required to create a new directory with
> valid txn at target and dump the data to the newly created directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[
https://issues.apache.org/jira/browse/HIVE-21213?focusedWorklogId=354256&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354256
]
ASF GitHub Bot logged work on HIVE-21213:
-
Author: ASF GitHub Bot
Created on: 05/Dec/19 12:36
Start Date: 05/Dec/19 12:36
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #587:
HIVE-21213 : Acid table bootstrap replication needs to handle directory created
by compaction with txn id
URL: https://github.com/apache/hive/pull/587#discussion_r354286299
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -670,4 +678,63 @@ public void testMultiDBTxn() throws Throwable {
replica.run("drop database " + dbName1 + " cascade");
replica.run("drop database " + dbName2 + " cascade");
}
+
+ private void runCompaction(String dbName, String tblName, CompactionType
compactionType) throws Throwable {
+HiveConf hiveConf = new HiveConf(primary.getConf());
+TxnStore txnHandler = TxnUtils.getTxnStore(hiveConf);
+txnHandler.compact(new CompactionRequest(dbName, tblName, compactionType));
+hiveConf.setBoolVar(HiveConf.ConfVars.COMPACTOR_CRUD_QUERY_BASED, false);
+runWorker(hiveConf);
+runCleaner(hiveConf);
+ }
+
+ private FileStatus[] getDirsInTableLoc(WarehouseInstance wh, String db,
String table) throws Throwable {
+Path tblLoc = new Path(wh.getTable(db, table).getSd().getLocation());
+FileSystem fs = tblLoc.getFileSystem(wh.getConf());
+return fs.listStatus(tblLoc, EximUtil.getDirectoryFilter(fs));
+ }
+
+ @Test
+ public void testAcidTablesBootstrapWithCompaction() throws Throwable {
+ String tableName = testName.getMethodName();
+ primary.run("use " + primaryDbName)
+.run("create table " + tableName + " (id int) clustered by(id)
into 3 buckets stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("insert into " + tableName + " values(1)")
+.run("insert into " + tableName + " values(2)");
+runCompaction(primaryDbName, tableName, CompactionType.MAJOR);
+WarehouseInstance.Tuple bootstrapDump = primary.dump(primaryDbName, null);
+replica.load(replicatedDbName, bootstrapDump.dumpLocation);
+replica.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[] {tableName})
+.run("repl status " + replicatedDbName)
+.verifyResult(bootstrapDump.lastReplicationId)
+.run("select id from " + tableName + " order by id")
+.verifyResults(new String[]{"1", "2"});
+
+FileStatus[] dirsInLoadPath = getDirsInTableLoc(primary, primaryDbName,
tableName);
+long writeId = -1;
+for (FileStatus fileStatus : dirsInLoadPath) {
+ if (fileStatus.getPath().getName().startsWith(AcidUtils.BASE_PREFIX)) {
+writeId =
AcidUtils.ParsedBase.parseBase(fileStatus.getPath()).getWriteId();
+
assertTrue(AcidUtils.getVisibilityTxnId(fileStatus.getPath().getName()) != -1);
+break;
+ }
+}
+//compaction is done so there should be a base directory.
+assertTrue(writeId != -1);
+
+dirsInLoadPath = getDirsInTableLoc(replica, replicatedDbName, tableName);
+for (FileStatus fileStatus : dirsInLoadPath) {
+ if (fileStatus.getPath().getName().startsWith(AcidUtils.BASE_PREFIX)) {
+assertTrue(writeId ==
AcidUtils.ParsedBase.parseBase(fileStatus.getPath()).getWriteId());
+
assertTrue(AcidUtils.getVisibilityTxnId(fileStatus.getPath().getName()) == -1);
+writeId = -1;
+break;
+ }
+}
+//make sure that it has done the verification.
+assertTrue(writeId == -1);
Review comment:
Using writeId again for verification saves a variable but it's not so much
readable. May be you want to save writeId on the replica in a separate variable
and compare the writeId from source that on the target.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354256)
Time Spent: 1h (was: 50m)
> Acid table bootstrap replication needs to handle directory created by
> compaction with txn id
>
>
> Key: HIVE-21213
> URL: https://issues.apache.org/jira/browse/HIVE-21213
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, repl
>Reporter: mahesh ku
[jira] [Work logged] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[
https://issues.apache.org/jira/browse/HIVE-21213?focusedWorklogId=354253&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354253
]
ASF GitHub Bot logged work on HIVE-21213:
-
Author: ASF GitHub Bot
Created on: 05/Dec/19 12:36
Start Date: 05/Dec/19 12:36
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #587:
HIVE-21213 : Acid table bootstrap replication needs to handle directory created
by compaction with txn id
URL: https://github.com/apache/hive/pull/587#discussion_r354280208
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -463,7 +464,29 @@ public static Path
getCopyDestination(ReplChangeManager.FileInfo fileInfo, Path
String[] subDirs = fileInfo.getSubDir().split(Path.SEPARATOR);
Path destination = destRoot;
for (String subDir: subDirs) {
- destination = new Path(destination, subDir);
+ // If the directory is created by compactor, then the directory will
have the transaction id also.
+ // In case of replication, the same txn id can not be used at target, as
the txn with same id might be a
+ // aborted or live txn at target.
+ // In case of bootstrap load, we copy only the committed data, so the
directory with only write id
+ // can be created. The validity txn id can be removed from the directory
name.
+ // TODO : Support for incremental load flow. This can be done once
replication of compaction is decided.
+ if (AcidUtils.getVisibilityTxnId(subDir) > 0) {
+if (subDir.startsWith(AcidUtils.BASE_PREFIX)) {
+ AcidUtils.ParsedBase pb = AcidUtils.ParsedBase.parseBase(new
Path(subDir));
+ destination = new Path(destination,
AcidUtils.baseDir(pb.getWriteId()));
+} else if (subDir.startsWith(AcidUtils.DELTA_PREFIX)) {
+ AcidUtils.ParsedDeltaLight pdl =
AcidUtils.ParsedDeltaLight.parse(new Path(subDir));
+ destination = new Path(destination,
AcidUtils.deltaSubdir(pdl.getMinWriteId(), pdl.getMaxWriteId()));
+} else if (subDir.startsWith(AcidUtils.DELETE_DELTA_PREFIX)) {
+ AcidUtils.ParsedDeltaLight pdl =
AcidUtils.ParsedDeltaLight.parse(new Path(subDir));
+ destination = new Path(destination,
AcidUtils.deleteDeltaSubdir(pdl.getMinWriteId(), pdl.getMaxWriteId()));
Review comment:
We are ignoring the statement id, I think because it's -1 always. Is that
future proof?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354253)
Time Spent: 40m (was: 0.5h)
> Acid table bootstrap replication needs to handle directory created by
> compaction with txn id
>
>
> Key: HIVE-21213
> URL: https://issues.apache.org/jira/browse/HIVE-21213
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21213.01.patch, HIVE-21213.02.patch,
> HIVE-21213.03.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The current implementation of compaction uses the txn id in the directory
> name. This is used to isolate the queries from reading the directory until
> compaction has finished and to avoid the compactor marking used earlier. In
> case of replication, during bootstrap , directory is copied as it is with the
> same name from source to destination cluster. But the directory created by
> compaction with txn id can not be copied as the txn list at target may be
> different from source. The txn id which is valid at source may be an aborted
> txn at target. So conversion logic is required to create a new directory with
> valid txn at target and dump the data to the newly created directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[
https://issues.apache.org/jira/browse/HIVE-21213?focusedWorklogId=354251&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354251
]
ASF GitHub Bot logged work on HIVE-21213:
-
Author: ASF GitHub Bot
Created on: 05/Dec/19 12:36
Start Date: 05/Dec/19 12:36
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #587:
HIVE-21213 : Acid table bootstrap replication needs to handle directory created
by compaction with txn id
URL: https://github.com/apache/hive/pull/587#discussion_r354223809
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -463,7 +464,29 @@ public static Path
getCopyDestination(ReplChangeManager.FileInfo fileInfo, Path
String[] subDirs = fileInfo.getSubDir().split(Path.SEPARATOR);
Path destination = destRoot;
for (String subDir: subDirs) {
- destination = new Path(destination, subDir);
+ // If the directory is created by compactor, then the directory will
have the transaction id also.
+ // In case of replication, the same txn id can not be used at target, as
the txn with same id might be a
+ // aborted or live txn at target.
+ // In case of bootstrap load, we copy only the committed data, so the
directory with only write id
+ // can be created. The validity txn id can be removed from the directory
name.
+ // TODO : Support for incremental load flow. This can be done once
replication of compaction is decided.
Review comment:
Is this TODO resolved? If yes, please remove it. If it needs to be resolved,
may be we should resolve it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354251)
Time Spent: 20m (was: 10m)
> Acid table bootstrap replication needs to handle directory created by
> compaction with txn id
>
>
> Key: HIVE-21213
> URL: https://issues.apache.org/jira/browse/HIVE-21213
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21213.01.patch, HIVE-21213.02.patch,
> HIVE-21213.03.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The current implementation of compaction uses the txn id in the directory
> name. This is used to isolate the queries from reading the directory until
> compaction has finished and to avoid the compactor marking used earlier. In
> case of replication, during bootstrap , directory is copied as it is with the
> same name from source to destination cluster. But the directory created by
> compaction with txn id can not be copied as the txn list at target may be
> different from source. The txn id which is valid at source may be an aborted
> txn at target. So conversion logic is required to create a new directory with
> valid txn at target and dump the data to the newly created directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[
https://issues.apache.org/jira/browse/HIVE-21213?focusedWorklogId=354257&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354257
]
ASF GitHub Bot logged work on HIVE-21213:
-
Author: ASF GitHub Bot
Created on: 05/Dec/19 12:36
Start Date: 05/Dec/19 12:36
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #587:
HIVE-21213 : Acid table bootstrap replication needs to handle directory created
by compaction with txn id
URL: https://github.com/apache/hive/pull/587#discussion_r354286550
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -670,4 +678,63 @@ public void testMultiDBTxn() throws Throwable {
replica.run("drop database " + dbName1 + " cascade");
replica.run("drop database " + dbName2 + " cascade");
}
+
+ private void runCompaction(String dbName, String tblName, CompactionType
compactionType) throws Throwable {
+HiveConf hiveConf = new HiveConf(primary.getConf());
+TxnStore txnHandler = TxnUtils.getTxnStore(hiveConf);
+txnHandler.compact(new CompactionRequest(dbName, tblName, compactionType));
+hiveConf.setBoolVar(HiveConf.ConfVars.COMPACTOR_CRUD_QUERY_BASED, false);
+runWorker(hiveConf);
+runCleaner(hiveConf);
+ }
+
+ private FileStatus[] getDirsInTableLoc(WarehouseInstance wh, String db,
String table) throws Throwable {
+Path tblLoc = new Path(wh.getTable(db, table).getSd().getLocation());
+FileSystem fs = tblLoc.getFileSystem(wh.getConf());
+return fs.listStatus(tblLoc, EximUtil.getDirectoryFilter(fs));
+ }
+
+ @Test
+ public void testAcidTablesBootstrapWithCompaction() throws Throwable {
+ String tableName = testName.getMethodName();
+ primary.run("use " + primaryDbName)
+.run("create table " + tableName + " (id int) clustered by(id)
into 3 buckets stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
Review comment:
Should we add a test for partitioned table as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354257)
> Acid table bootstrap replication needs to handle directory created by
> compaction with txn id
>
>
> Key: HIVE-21213
> URL: https://issues.apache.org/jira/browse/HIVE-21213
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21213.01.patch, HIVE-21213.02.patch,
> HIVE-21213.03.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The current implementation of compaction uses the txn id in the directory
> name. This is used to isolate the queries from reading the directory until
> compaction has finished and to avoid the compactor marking used earlier. In
> case of replication, during bootstrap , directory is copied as it is with the
> same name from source to destination cluster. But the directory created by
> compaction with txn id can not be copied as the txn list at target may be
> different from source. The txn id which is valid at source may be an aborted
> txn at target. So conversion logic is required to create a new directory with
> valid txn at target and dump the data to the newly created directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22546) Postgres schema not using quoted identifiers for certain tables
[
https://issues.apache.org/jira/browse/HIVE-22546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22546:
-
Attachment: HIVE-22546.3.patch
> Postgres schema not using quoted identifiers for certain tables
> ---
>
> Key: HIVE-22546
> URL: https://issues.apache.org/jira/browse/HIVE-22546
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Affects Versions: 4.0.0
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22546.2.patch, HIVE-22546.3.patch,
> HIVE-22546.3.patch, HIVE-22546.4.patch, HIVE-22546.patch
>
>
> In the latest postgresql schema
> (standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql)
> the following tables have lowercase table and column names:
> {code:java}
> aux_table
> compaction_queue
> completed_compactions
> completed_txn_components
> hive_locks
> materialization_rebuild_locks
> min_history_level
> next_compaction_queue_id
> next_lock_id
> next_txn_id
> next_write_id
> repl_txn_map
> runtime_stats
> txn_components
> txn_to_write_id
> txns
> write_set{code}
> As these tables are referenced from the Hive sys database, the queries to
> these tables will fail with a "Table not found" error.
> The problem is that the table and column names are not enclosed in quotes, so
> postgres will turn these identifiers into lowercase.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22553) Expose locks and transactions in sys db
[ https://issues.apache.org/jira/browse/HIVE-22553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22553: - Attachment: HIVE-22553.3.patch > Expose locks and transactions in sys db > --- > > Key: HIVE-22553 > URL: https://issues.apache.org/jira/browse/HIVE-22553 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22553.2.patch, HIVE-22553.3.patch, HIVE-22553.patch > > > Create new sysdb tables/views to access lock and transaction data. > This allows to provide admins with live data about ongoing locks and > transacions. Due to this being in the sys db access to this information can > be restricted to select privileged users. > Information about locks and compactions can be joined and accessed at the > same time. > Compaction related transactions would also be visible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22582) Avoid reading table as ACID when table name is starting with "delta" , but table is not transactional and BI Split Strategy is used
[
https://issues.apache.org/jira/browse/HIVE-22582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988726#comment-16988726
]
Peter Vary edited comment on HIVE-22582 at 12/5/19 12:04 PM:
-
+1, pending test
was (Author: pvary):
+1
> Avoid reading table as ACID when table name is starting with "delta" , but
> table is not transactional and BI Split Strategy is used
> ---
>
> Key: HIVE-22582
> URL: https://issues.apache.org/jira/browse/HIVE-22582
> Project: Hive
> Issue Type: Bug
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Attachments: HIVE-22582.patch
>
>
> The issue is fixed in HIVE-22473 but missed a check for BI Split Strategy.
> Steps to reproduce:
> {code:java}
> set hive.exec.orc.split.strategy=BI;
> create table delta_result (a int) stored as orc
> tblproperties('transactional'='false');
> insert into delta_result select 1;
> select * from delta_result;
> {code}
> Exception Stack Trace:
> {code:java}
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: String index out of range: -1
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1929)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2016)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:461)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:430)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:336)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:576)
> ... 50 more
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.parsedDelta(AcidUtils.java:1128)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:921)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getLogicalLength(AcidUtils.java:2084)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$BISplitStrategy.getSplits(OrcInputFormat.java:1115)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1905)
> ... 55 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22582) Avoid reading table as ACID when table name is starting with "delta" , but table is not transactional and BI Split Strategy is used
[
https://issues.apache.org/jira/browse/HIVE-22582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988726#comment-16988726
]
Peter Vary commented on HIVE-22582:
---
+1
> Avoid reading table as ACID when table name is starting with "delta" , but
> table is not transactional and BI Split Strategy is used
> ---
>
> Key: HIVE-22582
> URL: https://issues.apache.org/jira/browse/HIVE-22582
> Project: Hive
> Issue Type: Bug
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Attachments: HIVE-22582.patch
>
>
> The issue is fixed in HIVE-22473 but missed a check for BI Split Strategy.
> Steps to reproduce:
> {code:java}
> set hive.exec.orc.split.strategy=BI;
> create table delta_result (a int) stored as orc
> tblproperties('transactional'='false');
> insert into delta_result select 1;
> select * from delta_result;
> {code}
> Exception Stack Trace:
> {code:java}
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: String index out of range: -1
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1929)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2016)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:461)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:430)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:336)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:576)
> ... 50 more
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.parsedDelta(AcidUtils.java:1128)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:921)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getLogicalLength(AcidUtils.java:2084)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$BISplitStrategy.getSplits(OrcInputFormat.java:1115)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1905)
> ... 55 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22577) StringIndexOutOfBoundsException when getting sessionId from worker node name
[
https://issues.apache.org/jira/browse/HIVE-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988716#comment-16988716
]
Hive QA commented on HIVE-22577:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} llap-client in master has 27 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} llap-client: The patch generated 3 new + 40 unchanged
- 3 fixed = 43 total (was 43) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19754/dev-support/hive-personality.sh
|
| git revision | master / bc7b38e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19754/yetus/diff-checkstyle-llap-client.txt
|
| modules | C: llap-client U: llap-client |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19754/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> StringIndexOutOfBoundsException when getting sessionId from worker node name
>
>
> Key: HIVE-22577
> URL: https://issues.apache.org/jira/browse/HIVE-22577
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22577.1.patch
>
>
> When the node name is "worker-" the following exception is thrown
>
> {code:java}
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index
> out of range: -1 at java.lang.String.substring(String.java:1931) at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
> at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl$DynamicServiceInstanceSet.(LlapZookeeperRegistryImpl.java:313)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:462)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getApplicat
[jira] [Updated] (HIVE-21569) Bump guava version to 28.1-jre
[ https://issues.apache.org/jira/browse/HIVE-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-21569: Summary: Bump guava version to 28.1-jre (was: Bump guava version to 28.0-jre) > Bump guava version to 28.1-jre > -- > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch > > > Changes required to bump guava to 28.0-jre -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21569) Bump guava version to 28.1-jre
[ https://issues.apache.org/jira/browse/HIVE-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-21569: Description: Changes required to bump guava to 28.1-jre (was: Changes required to bump guava to 28.0-jre) > Bump guava version to 28.1-jre > -- > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch > > > Changes required to bump guava to 28.1-jre -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21569) Bump guava version to 28.0-jre
[ https://issues.apache.org/jira/browse/HIVE-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988706#comment-16988706 ] David Lavati commented on HIVE-21569: - I've modified the scope of this issue to the actual bump, we'll see how the tests go. > Bump guava version to 28.0-jre > -- > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch > > > Changes required to bump guava to 28.0-jre -- This message was sent by Atlassian Jira (v8.3.4#803005)
