[jira] [Updated] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.21.patch > Show Bucketing version for ReduceSinkOp in explain extended plan > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch, > HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch, > HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch, > HIVE-21304.21.patch > > > Show Bucketing version for ReduceSinkOp in explain extended plan. > This helps identify what hashing algorithm is being used by by ReduceSinkOp. > > cc [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23095) NDV might be overestimated for a table with ~70 value
[ https://issues.apache.org/jira/browse/HIVE-23095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23095: Attachment: HIVE-23095.04.patch > NDV might be overestimated for a table with ~70 value > - > > Key: HIVE-23095 > URL: https://issues.apache.org/jira/browse/HIVE-23095 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23095.01.patch, HIVE-23095.02.patch, > HIVE-23095.03.patch, HIVE-23095.04.patch, HIVE-23095.04.patch, > HIVE-23095.04.patch > > Time Spent: 10m > Remaining Estimate: 0h > > uncovered during looking into HIVE-23082 > https://issues.apache.org/jira/browse/HIVE-23082?focusedCommentId=17067773=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17067773 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23082) PK/FK stat rescale doesn't work in some cases
[
https://issues.apache.org/jira/browse/HIVE-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-23082:
Attachment: HIVE-23082.03.patch
> PK/FK stat rescale doesn't work in some cases
> -
>
> Key: HIVE-23082
> URL: https://issues.apache.org/jira/browse/HIVE-23082
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23082.01.patch, HIVE-23082.02.patch,
> HIVE-23082.03.patch, HIVE-23082.03.patch, HIVE-23082.03.patch
>
>
> As a result in Joins may retain the original estimate; see MAPJOIN_33 in this
> plan ; which retained the estimate of SEL_32
> {code}
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Map 1 <- Map 2 (BROADCAST_EDGE)|
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:12 |
> | Stage-1|
> | Map 1 vectorized |
> | File Output Operator [FS_36] |
> | Limit [LIM_35] (rows=12 width=4) |
> | Number of rows:12|
> | Select Operator [SEL_34] (rows=5040 width=4) |
> | Output:["_col0"] |
> | Map Join Operator [MAPJOIN_33] (rows=5040 width=8) |
> | Conds:SEL_32._col0=RS_30._col0(Inner) |
> | <-Map 2 [BROADCAST_EDGE] vectorized|
> | BROADCAST [RS_30]|
> | PartitionCols:_col0|
> | Select Operator [SEL_29] (rows=1 width=8) |
> | Output:["_col0"] |
> | Filter Operator [FIL_28] (rows=1 width=108) |
> | predicate:((r_reason_id = 'reason 66') and r_reason_sk
> is not null) |
> | TableScan [TS_3] (rows=2 width=108) |
> |
> default@rx0,reason,Tbl:COMPLETE,Col:COMPLETE,Output:["r_reason_id","r_reason_sk"]
> |
> | <-Select Operator [SEL_32] (rows=5040 width=7) |
> | Output:["_col0"] |
> | Filter Operator [FIL_31] (rows=5040 width=7) |
> | predicate:sr_reason_sk is not null |
> | TableScan [TS_0] (rows=5112 width=7) |
> |
> default@sr0,store_returns,Tbl:COMPLETE,Col:COMPLETE,Output:["sr_reason_sk"] |
> ||
> ++
> {code}
> repro:
> {code}
> set hive.query.results.cache.enabled=false;
> set hive.explain.user=true;
> drop table if exists default.rx0;
> drop table if exists default.sr0;
> create table rx0 (r_reason_id string, r_reason_sk bigint);
> create table sr0 (sr_reason_sk bigint);
> insert into rx0 values ('',1),('GEAA',70);
> insert into sr0 values (NULL),(1),(2),(3),(4),(5),(6),(7),(8),(9),(10),
> (11),(12),(13),(14),(15),(16),(17),(18),(19),(20),(21),(22),(23),(24),(25),
> (26),(27),(28),(29),(30),(31),(32),(33),(34),(35),(36),(37),(38),(39),(40),
> (41),(42),(43),(44),(45),(46),(47),(48),(49),(50),(51),(52),(53),(54),(55),
> (56),(57),(58),(59),(60),(61),(62),(63),(64),(65),(66),(67),(68),(69),(70);
> insert into sr0 select a.* from sr0 a,sr0 b;
> -- |sr0| ~ 5112
> explain select 1
> from default.sr0 store_returns , default.rx0 reason
> where sr_reason_sk = r_reason_sk
> and r_reason_id = 'reason 66'
> limit 12;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23030) Enable sketch union-s to be rolled up
[ https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23030: Attachment: HIVE-23030.06.patch > Enable sketch union-s to be rolled up > - > > Key: HIVE-23030 > URL: https://issues.apache.org/jira/browse/HIVE-23030 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23030.01.patch, HIVE-23030.02.patch, > HIVE-23030.03.patch, HIVE-23030.04.patch, HIVE-23030.04.patch, > HIVE-23030.05.patch, HIVE-23030.06.patch, HIVE-23030.06.patch, > HIVE-23030.06.patch, HIVE-23030.06.patch > > Time Spent: 5h > Remaining Estimate: 0h > > Enabling rolling up sketch aggregates could enable the matching of > materialized views created for higher dimensions to be applied for lower > dimension cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23130) User friendly error message when MV rewriting fails
[
https://issues.apache.org/jira/browse/HIVE-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074281#comment-17074281
]
Hive QA commented on HIVE-23130:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21416/dev-support/hive-personality.sh
|
| git revision | master / 7f9ce94 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21416/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> User friendly error message when MV rewriting fails
> ---
>
> Key: HIVE-23130
> URL: https://issues.apache.org/jira/browse/HIVE-23130
> Project: Hive
> Issue Type: Task
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Attachments: HIVE-23130.1.patch, HIVE-23130.1.patch
>
>
> If materialized view rewriting fails because an unsupported SQL clause or
> operator is used we got an error message like this:
> {code}
> FAILED: SemanticException Cannot enable automatic rewriting for materialized
> view. Unsupported RelNode type HiveSortExchange encountered in the query plan
> {code}
> This refers to the *HiveSortExchange* operator. This is introduced to CBO
> plan if the statement has *sort by* clause which may not clear for the user.
> {code}
> create materialized view cmv_mat_view as select a, b, c from cmv_basetable
> sort by a;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Status: In Progress (was: Patch Available) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch, > HIVE-23039.06.patch, HIVE-23039.07.patch, HIVE-23039.08.patch, > HIVE-23039.09.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Attachment: HIVE-23039.09.patch Status: Patch Available (was: In Progress) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch, > HIVE-23039.06.patch, HIVE-23039.07.patch, HIVE-23039.08.patch, > HIVE-23039.09.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23132) Add test of Explain CBO of Merge statements
[ https://issues.apache.org/jira/browse/HIVE-23132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23132: -- Status: Patch Available (was: Open) > Add test of Explain CBO of Merge statements > --- > > Key: HIVE-23132 > URL: https://issues.apache.org/jira/browse/HIVE-23132 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23132.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23132) Add test of Explain CBO of Merge statements
[ https://issues.apache.org/jira/browse/HIVE-23132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23132: -- Attachment: HIVE-23132.1.patch > Add test of Explain CBO of Merge statements > --- > > Key: HIVE-23132 > URL: https://issues.apache.org/jira/browse/HIVE-23132 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23132.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23132) Add test of Explain CBO of Merge statements
[ https://issues.apache.org/jira/browse/HIVE-23132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa reassigned HIVE-23132: - > Add test of Explain CBO of Merge statements > --- > > Key: HIVE-23132 > URL: https://issues.apache.org/jira/browse/HIVE-23132 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Status: In Progress (was: Patch Available) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch, > HIVE-23039.06.patch, HIVE-23039.07.patch, HIVE-23039.08.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Attachment: HIVE-23039.08.patch Status: Patch Available (was: In Progress) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch, > HIVE-23039.06.patch, HIVE-23039.07.patch, HIVE-23039.08.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23118) Option for exposing compile time counters as tez counters
[
https://issues.apache.org/jira/browse/HIVE-23118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074271#comment-17074271
]
Hive QA commented on HIVE-23118:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998675/HIVE-23118.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18168 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21415/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21415/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21415/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998675 - PreCommit-HIVE-Build
> Option for exposing compile time counters as tez counters
> -
>
> Key: HIVE-23118
> URL: https://issues.apache.org/jira/browse/HIVE-23118
> Project: Hive
> Issue Type: Improvement
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23118.1.patch, HIVE-23118.2.patch,
> HIVE-23118.3.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> TezCounters currently are runtime only. Some compile time information from
> optimizer can be exposed as counters which can then be used by workload
> management to make runtime decisions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Attachment: HIVE-23039.07.patch Status: Patch Available (was: In Progress) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch, > HIVE-23039.06.patch, HIVE-23039.07.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Status: In Progress (was: Patch Available) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch, > HIVE-23039.06.patch, HIVE-23039.07.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23118) Option for exposing compile time counters as tez counters
[
https://issues.apache.org/jira/browse/HIVE-23118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074256#comment-17074256
]
Hive QA commented on HIVE-23118:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 5 new + 95 unchanged - 0 fixed
= 100 total (was 95) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21415/dev-support/hive-personality.sh
|
| git revision | master / 7f9ce94 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21415/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21415/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Option for exposing compile time counters as tez counters
> -
>
> Key: HIVE-23118
> URL: https://issues.apache.org/jira/browse/HIVE-23118
> Project: Hive
> Issue Type: Improvement
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23118.1.patch, HIVE-23118.2.patch,
> HIVE-23118.3.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> TezCounters currently are runtime only. Some compile time information from
> optimizer can be exposed as counters which can then be used by workload
> management to make runtime decisions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074251#comment-17074251
]
Ganesha Shreedhara commented on HIVE-21492:
---
For some reason test didn't pick the patch (cancelling and resubmitting the
patch didn't work). I have created a new patch to make sure that it gets picked
by Hive QA job in the next run.
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch,
> HIVE-21492.4.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-21492:
--
Attachment: HIVE-21492.4.patch
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch,
> HIVE-21492.4.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-21492:
--
Status: Patch Available (was: In Progress)
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch,
> HIVE-21492.4.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-21492:
--
Status: In Progress (was: Patch Available)
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074239#comment-17074239
]
Hive QA commented on HIVE-23039:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998673/HIVE-23039.06.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 18179 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.repl.TestReplDumpTask.removeDBPropertyToPreventRenameWhenBootstrapDumpOfTableFails
(batchId=356)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testCheckPointing
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testCheckPointingInDumpFailure
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testCheckPointingMetadataDumpFailure
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testCheckPointing
(batchId=270)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testCheckPointingInDumpFailure
(batchId=270)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testCheckPointingMetadataDumpFailure
(batchId=270)
org.apache.hive.minikdc.TestSSLWithMiniKdc.testConnection (batchId=308)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21414/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21414/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21414/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998673 - PreCommit-HIVE-Build
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch,
> HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch,
> HIVE-23039.06.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23130) User friendly error message when MV rewriting fails
[
https://issues.apache.org/jira/browse/HIVE-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23130:
--
Attachment: HIVE-23130.1.patch
> User friendly error message when MV rewriting fails
> ---
>
> Key: HIVE-23130
> URL: https://issues.apache.org/jira/browse/HIVE-23130
> Project: Hive
> Issue Type: Task
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Attachments: HIVE-23130.1.patch, HIVE-23130.1.patch
>
>
> If materialized view rewriting fails because an unsupported SQL clause or
> operator is used we got an error message like this:
> {code}
> FAILED: SemanticException Cannot enable automatic rewriting for materialized
> view. Unsupported RelNode type HiveSortExchange encountered in the query plan
> {code}
> This refers to the *HiveSortExchange* operator. This is introduced to CBO
> plan if the statement has *sort by* clause which may not clear for the user.
> {code}
> create materialized view cmv_mat_view as select a, b, c from cmv_basetable
> sort by a;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23130) User friendly error message when MV rewriting fails
[
https://issues.apache.org/jira/browse/HIVE-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23130:
--
Status: Patch Available (was: Open)
> User friendly error message when MV rewriting fails
> ---
>
> Key: HIVE-23130
> URL: https://issues.apache.org/jira/browse/HIVE-23130
> Project: Hive
> Issue Type: Task
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Attachments: HIVE-23130.1.patch, HIVE-23130.1.patch
>
>
> If materialized view rewriting fails because an unsupported SQL clause or
> operator is used we got an error message like this:
> {code}
> FAILED: SemanticException Cannot enable automatic rewriting for materialized
> view. Unsupported RelNode type HiveSortExchange encountered in the query plan
> {code}
> This refers to the *HiveSortExchange* operator. This is introduced to CBO
> plan if the statement has *sort by* clause which may not clear for the user.
> {code}
> create materialized view cmv_mat_view as select a, b, c from cmv_basetable
> sort by a;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23130) User friendly error message when MV rewriting fails
[
https://issues.apache.org/jira/browse/HIVE-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23130:
--
Status: Open (was: Patch Available)
> User friendly error message when MV rewriting fails
> ---
>
> Key: HIVE-23130
> URL: https://issues.apache.org/jira/browse/HIVE-23130
> Project: Hive
> Issue Type: Task
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Attachments: HIVE-23130.1.patch, HIVE-23130.1.patch
>
>
> If materialized view rewriting fails because an unsupported SQL clause or
> operator is used we got an error message like this:
> {code}
> FAILED: SemanticException Cannot enable automatic rewriting for materialized
> view. Unsupported RelNode type HiveSortExchange encountered in the query plan
> {code}
> This refers to the *HiveSortExchange* operator. This is introduced to CBO
> plan if the statement has *sort by* clause which may not clear for the user.
> {code}
> create materialized view cmv_mat_view as select a, b, c from cmv_basetable
> sort by a;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23120) TopNKey related tests should be run by TestMiniLlapLocalCliDriver only
[ https://issues.apache.org/jira/browse/HIVE-23120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23120: -- Status: Patch Available (was: Open) > TopNKey related tests should be run by TestMiniLlapLocalCliDriver only > -- > > Key: HIVE-23120 > URL: https://issues.apache.org/jira/browse/HIVE-23120 > Project: Hive > Issue Type: Task > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23120.1.patch, HIVE-23120.2.patch > > > TopNKey optimization is only used when the execution framework is Tez. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23120) TopNKey related tests should be run by TestMiniLlapLocalCliDriver only
[ https://issues.apache.org/jira/browse/HIVE-23120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23120: -- Attachment: HIVE-23120.2.patch > TopNKey related tests should be run by TestMiniLlapLocalCliDriver only > -- > > Key: HIVE-23120 > URL: https://issues.apache.org/jira/browse/HIVE-23120 > Project: Hive > Issue Type: Task > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23120.1.patch, HIVE-23120.2.patch > > > TopNKey optimization is only used when the execution framework is Tez. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23120) TopNKey related tests should be run by TestMiniLlapLocalCliDriver only
[ https://issues.apache.org/jira/browse/HIVE-23120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23120: -- Status: Open (was: Patch Available) > TopNKey related tests should be run by TestMiniLlapLocalCliDriver only > -- > > Key: HIVE-23120 > URL: https://issues.apache.org/jira/browse/HIVE-23120 > Project: Hive > Issue Type: Task > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23120.1.patch, HIVE-23120.2.patch > > > TopNKey optimization is only used when the execution framework is Tez. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23006) Compiler support for Probe MapJoin
[ https://issues.apache.org/jira/browse/HIVE-23006?focusedWorklogId=415296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415296 ] ASF GitHub Bot logged work on HIVE-23006: - Author: ASF GitHub Bot Created on: 03/Apr/20 02:56 Start Date: 03/Apr/20 02:56 Worklog Time Spent: 10m Work Description: jcamachor commented on issue #952: HIVE-23006 ProbeDecode compiler support URL: https://github.com/apache/hive/pull/952#issuecomment-608201286 @pgaref , thanks for changes. Patch looks good. About the name, I think it is fine (probe side for the filtering generated statically in case of expressions or dynamically for joins). I think main remaining issue is related to selection in case of multiple MJ operators. I was thinking that for the time being, we could make the policy pluggable via config and have two very simple ones: 1) keep the one with lowest `ndv_JOIN_probe_key_column / ndv_TS_target_column` ratio, or 2) keep those with ratio below a certain threshold specified through a config value. Alternatively, in this patch you could maybe go only with option 1 and tackle other in follow-up. You have the information about the stats in the operators themselves (follow `op.getStatistics()` calls to see how it is retrieved per column basis). Thus, I would make that decision in the TezCompiler (once we gather all the context information) rather than GenTezUtils. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415296) Time Spent: 1h 10m (was: 1h) > Compiler support for Probe MapJoin > -- > > Key: HIVE-23006 > URL: https://issues.apache.org/jira/browse/HIVE-23006 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > The decision of pushing down information to the Record reader (potentially > reducing decoding time by row-level filtering) should be done at query > compilation time. > This patch adds an extra optimisation step with the goal of finding Table > Scan operators that could reduce the number of rows decoded at runtime using > extra available information. > It currently looks for all the available MapJoin operators that could use the > smaller HashTable on the probing side (where TS is) to filter-out rows that > would never match. > To do so the HashTable information is pushed down to the TS properties and > then propagated as part of MapWork. > If the a single TS is used by multiple operators (shared-word), this rule can > not be applied. > This rule can be extended to support static filter expressions like: > _select * from sales where sold_state = 'PR';_ > This optimisation manly targets the Tez execution engine running on Llap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23006) Compiler support for Probe MapJoin
[ https://issues.apache.org/jira/browse/HIVE-23006?focusedWorklogId=415295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415295 ] ASF GitHub Bot logged work on HIVE-23006: - Author: ASF GitHub Bot Created on: 03/Apr/20 02:55 Start Date: 03/Apr/20 02:55 Worklog Time Spent: 10m Work Description: jcamachor commented on issue #952: HIVE-23006 ProbeDecode compiler support URL: https://github.com/apache/hive/pull/952#issuecomment-608201286 @pgaref , thanks for changes. Patch looks good. About the name, I think it is fine (probe side for the filtering generated statically in case of expressions or dynamically for joins). I think main remaining issue is related to selection in case of multiple MJ operators. I was thinking that for the time being, we could make the policy pluggable via config and have two very simple ones: 1) keep the one with lowest `ndv_JOIN_key_column / ndv_TS_target_column` ratio, or 2) keep those with ratio below a certain threshold specified through a config value. Alternatively, in this patch you could maybe go only with option 1 and tackle other in follow-up. You have the information about the stats in the operators themselves (follow `op.getStatistics()` calls to see how it is retrieved per column basis). Thus, I would make that decision in the TezCompiler (once we gather all the context information) rather than GenTezUtils. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415295) Time Spent: 1h (was: 50m) > Compiler support for Probe MapJoin > -- > > Key: HIVE-23006 > URL: https://issues.apache.org/jira/browse/HIVE-23006 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > The decision of pushing down information to the Record reader (potentially > reducing decoding time by row-level filtering) should be done at query > compilation time. > This patch adds an extra optimisation step with the goal of finding Table > Scan operators that could reduce the number of rows decoded at runtime using > extra available information. > It currently looks for all the available MapJoin operators that could use the > smaller HashTable on the probing side (where TS is) to filter-out rows that > would never match. > To do so the HashTable information is pushed down to the TS properties and > then propagated as part of MapWork. > If the a single TS is used by multiple operators (shared-word), this rule can > not be applied. > This rule can be extended to support static filter expressions like: > _select * from sales where sold_state = 'PR';_ > This optimisation manly targets the Tez execution engine running on Llap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23006) Compiler support for Probe MapJoin
[
https://issues.apache.org/jira/browse/HIVE-23006?focusedWorklogId=415294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415294
]
ASF GitHub Bot logged work on HIVE-23006:
-
Author: ASF GitHub Bot
Created on: 03/Apr/20 02:54
Start Date: 03/Apr/20 02:54
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #952: HIVE-23006
ProbeDecode compiler support
URL: https://github.com/apache/hive/pull/952#discussion_r402708575
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
##
@@ -1483,17 +1489,64 @@ private void
removeSemijoinsParallelToMapJoin(OptimizeTezProcContext procCtx)
}
}
-if (semijoins.size() > 0) {
- for (ReduceSinkOperator rs : semijoins.keySet()) {
-if (LOG.isDebugEnabled()) {
- LOG.debug("Semijoin optimization with parallel edge to map join.
Removing semijoin "
- + OperatorUtils.getOpNamePretty(rs) + " - " +
OperatorUtils.getOpNamePretty(semijoins.get(rs)));
+if
(!procCtx.conf.getBoolVar(ConfVars.TEZ_DYNAMIC_SEMIJOIN_REDUCTION_FOR_MAPJOIN))
{
+ if (semijoins.size() > 0) {
Review comment:
You may still need to get rid of the SJ branches for the map join operators
here if `HIVE_MAPJOIN_PROBEDECODE_ENABLED`? Or do you need to keep them?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415294)
Time Spent: 50m (was: 40m)
> Compiler support for Probe MapJoin
> --
>
> Key: HIVE-23006
> URL: https://issues.apache.org/jira/browse/HIVE-23006
> Project: Hive
> Issue Type: Sub-task
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The decision of pushing down information to the Record reader (potentially
> reducing decoding time by row-level filtering) should be done at query
> compilation time.
> This patch adds an extra optimisation step with the goal of finding Table
> Scan operators that could reduce the number of rows decoded at runtime using
> extra available information.
> It currently looks for all the available MapJoin operators that could use the
> smaller HashTable on the probing side (where TS is) to filter-out rows that
> would never match.
> To do so the HashTable information is pushed down to the TS properties and
> then propagated as part of MapWork.
> If the a single TS is used by multiple operators (shared-word), this rule can
> not be applied.
> This rule can be extended to support static filter expressions like:
> _select * from sales where sold_state = 'PR';_
> This optimisation manly targets the Tez execution engine running on Llap.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23006) Compiler support for Probe MapJoin
[ https://issues.apache.org/jira/browse/HIVE-23006?focusedWorklogId=415293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415293 ] ASF GitHub Bot logged work on HIVE-23006: - Author: ASF GitHub Bot Created on: 03/Apr/20 02:53 Start Date: 03/Apr/20 02:53 Worklog Time Spent: 10m Work Description: jcamachor commented on issue #952: HIVE-23006 ProbeDecode compiler support URL: https://github.com/apache/hive/pull/952#issuecomment-608201286 @pgaref , thanks for changes. Patch looks good. About the name, I think it is fine (probe side for the filtering generated statically in case of expressions or dynamically for joins). I think main remaining issue is related to selection in case of multiple MJ operators. I was thinking that for the time being, we could make the policy pluggable via config and have three very simple ones: 1) keep all of them, 2) keep the one with lowest `ndv_JOIN_key_column / ndv_TS_target_column` ratio, or 3) keep those with ratio below a certain threshold specified through a config value. Alternatively, in this patch you could maybe go only with option 2 and tackle others in follow-up. You have the information about the stats in the operators themselves (follow `op.getStatistics()` calls to see how it is retrieved per column basis). Thus, I would make that decision in the TezCompiler (once we gather all the context information) rather than GenTezUtils. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415293) Time Spent: 40m (was: 0.5h) > Compiler support for Probe MapJoin > -- > > Key: HIVE-23006 > URL: https://issues.apache.org/jira/browse/HIVE-23006 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The decision of pushing down information to the Record reader (potentially > reducing decoding time by row-level filtering) should be done at query > compilation time. > This patch adds an extra optimisation step with the goal of finding Table > Scan operators that could reduce the number of rows decoded at runtime using > extra available information. > It currently looks for all the available MapJoin operators that could use the > smaller HashTable on the probing side (where TS is) to filter-out rows that > would never match. > To do so the HashTable information is pushed down to the TS properties and > then propagated as part of MapWork. > If the a single TS is used by multiple operators (shared-word), this rule can > not be applied. > This rule can be extended to support static filter expressions like: > _select * from sales where sold_state = 'PR';_ > This optimisation manly targets the Tez execution engine running on Llap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074210#comment-17074210
]
Hive QA commented on HIVE-23039:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 117 unchanged - 1
fixed = 118 total (was 118) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} itests/hive-unit: The patch generated 42 new + 661
unchanged - 0 fixed = 703 total (was 661) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} ql generated 0 new + 1515 unchanged - 13 fixed =
1515 total (was 1528) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21414/dev-support/hive-personality.sh
|
| git revision | master / 7f9ce94 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21414/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21414/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21414/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch,
> HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch,
> HIVE-23039.06.patch
>
>
[jira] [Updated] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-21492:
--
Status: Patch Available (was: Open)
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-21492:
--
Status: Open (was: Patch Available)
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20728) Enable flaky test back: stat_estimate_related_col.q
[
https://issues.apache.org/jira/browse/HIVE-20728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074195#comment-17074195
]
Jesus Camacho Rodriguez commented on HIVE-20728:
+1
> Enable flaky test back: stat_estimate_related_col.q
> ---
>
> Key: HIVE-20728
> URL: https://issues.apache.org/jira/browse/HIVE-20728
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Jesus Camacho Rodriguez
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20728.01.patch
>
>
> Seems like sometimes the actually read rows by the operator is not 1 ; but
> 64...
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing stat_estimate_related_col.q
> 491c491
> < Statistics: Num rows: 1/64 Data size: 8 Basic stats: COMPLETE
> Column stats: COMPLETE
> ---
> > Statistics: Num rows: 1/1 Data size: 8 Basic stats: COMPLETE
> > Column stats: COMPLETE
> 505c505
> < Statistics: Num rows: 1/64 Data size: 8 Basic stats: COMPLETE
> Column stats: COMPLETE
> ---
> > Statistics: Num rows: 1/1 Data size: 8 Basic stats: COMPLETE
> > Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074194#comment-17074194
]
Hive QA commented on HIVE-21492:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998415/HIVE-21492.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21413/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21413/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21413/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12998415/HIVE-21492.3.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998415 - PreCommit-HIVE-Build
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23118) Option for exposing compile time counters as tez counters
[ https://issues.apache.org/jira/browse/HIVE-23118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-23118: - Attachment: HIVE-23118.3.patch > Option for exposing compile time counters as tez counters > - > > Key: HIVE-23118 > URL: https://issues.apache.org/jira/browse/HIVE-23118 > Project: Hive > Issue Type: Improvement >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23118.1.patch, HIVE-23118.2.patch, > HIVE-23118.3.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > TezCounters currently are runtime only. Some compile time information from > optimizer can be exposed as counters which can then be used by workload > management to make runtime decisions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23118) Option for exposing compile time counters as tez counters
[
https://issues.apache.org/jira/browse/HIVE-23118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074193#comment-17074193
]
Hive QA commented on HIVE-23118:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998662/HIVE-23118.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18168 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21412/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21412/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21412/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998662 - PreCommit-HIVE-Build
> Option for exposing compile time counters as tez counters
> -
>
> Key: HIVE-23118
> URL: https://issues.apache.org/jira/browse/HIVE-23118
> Project: Hive
> Issue Type: Improvement
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23118.1.patch, HIVE-23118.2.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> TezCounters currently are runtime only. Some compile time information from
> optimizer can be exposed as counters which can then be used by workload
> management to make runtime decisions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22967) Support hive.reloadable.aux.jars.path for Hive on Tez
[
https://issues.apache.org/jira/browse/HIVE-22967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074191#comment-17074191
]
Toshihiro Suzuki commented on HIVE-22967:
-
[~Rajkumar Singh] Could you please review this when you gat a chance?
> Support hive.reloadable.aux.jars.path for Hive on Tez
> -
>
> Key: HIVE-22967
> URL: https://issues.apache.org/jira/browse/HIVE-22967
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Toshihiko Uchida
>Assignee: Toshihiko Uchida
>Priority: Minor
> Attachments: HIVE-22967.1.patch, HIVE-22967.2.patch
>
>
> The jars in hive.reloadable.aux.jars.path are not localized in Tez containers.
> As a result, any query utilizing those reloadable jars fails for Hive on Tez
> due to ClassNotFoundException.
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
> vertexName=Map 1, vertexId=vertex_1578856704640_0087_1_00, diagnostics=[Task
> failed, taskId=task_1578856704640_0087_1_00_01, diagnostics=[TaskAttempt
> 0 failed, info=[Error: Error while running task ( failure) :
> attempt_1578856704640_0087_1_00_01_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:354)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.example.hive.udf.Lower
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOperator.java:71)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.initializeOp(VectorFilterOperator.java:83)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:573)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:525)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:386)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.initializeMapOperator(VectorMapOperator.java:591)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:317)
> ... 17 more
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.example.hive.udf.Lower
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:134)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.isStateful(FunctionRegistry.java:1492)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:111)
> at
>
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Attachment: HIVE-23039.06.patch Status: Patch Available (was: In Progress) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch, > HIVE-23039.06.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Status: In Progress (was: Patch Available) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch, > HIVE-23039.06.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23118) Option for exposing compile time counters as tez counters
[
https://issues.apache.org/jira/browse/HIVE-23118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074170#comment-17074170
]
Hive QA commented on HIVE-23118:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 5 new + 95 unchanged - 0 fixed
= 100 total (was 95) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21412/dev-support/hive-personality.sh
|
| git revision | master / 7f9ce94 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21412/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21412/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Option for exposing compile time counters as tez counters
> -
>
> Key: HIVE-23118
> URL: https://issues.apache.org/jira/browse/HIVE-23118
> Project: Hive
> Issue Type: Improvement
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23118.1.patch, HIVE-23118.2.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> TezCounters currently are runtime only. Some compile time information from
> optimizer can be exposed as counters which can then be used by workload
> management to make runtime decisions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074169#comment-17074169
]
Ferdinand Xu edited comment on HIVE-21492 at 4/3/20, 12:47 AM:
---
Yes, I manually started another job to double check this (see #21413).
was (Author: ferd):
Yes, I manually started another job to double check this.
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074169#comment-17074169
]
Ferdinand Xu commented on HIVE-21492:
-
Yes, I manually started another job to double check this.
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23118) Option for exposing compile time counters as tez counters
[
https://issues.apache.org/jira/browse/HIVE-23118?focusedWorklogId=415118=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415118
]
ASF GitHub Bot logged work on HIVE-23118:
-
Author: ASF GitHub Bot
Created on: 02/Apr/20 23:07
Start Date: 02/Apr/20 23:07
Worklog Time Spent: 10m
Work Description: prasanthj commented on pull request #966: HIVE-23118:
Option for exposing compile time counters as tez counters
URL: https://github.com/apache/hive/pull/966#discussion_r402646605
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java
##
@@ -185,6 +188,12 @@ public int monitorExecution() {
if (wmContext != null) {
Set desiredCounters = wmContext.getSubscribedCounters();
TezCounters dagCounters = status.getDAGCounters();
+ // if initial counters exists, merge it with dag counters to get
aggregated view
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415118)
Time Spent: 0.5h (was: 20m)
> Option for exposing compile time counters as tez counters
> -
>
> Key: HIVE-23118
> URL: https://issues.apache.org/jira/browse/HIVE-23118
> Project: Hive
> Issue Type: Improvement
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23118.1.patch, HIVE-23118.2.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> TezCounters currently are runtime only. Some compile time information from
> optimizer can be exposed as counters which can then be used by workload
> management to make runtime decisions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23118) Option for exposing compile time counters as tez counters
[ https://issues.apache.org/jira/browse/HIVE-23118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-23118: - Attachment: HIVE-23118.2.patch > Option for exposing compile time counters as tez counters > - > > Key: HIVE-23118 > URL: https://issues.apache.org/jira/browse/HIVE-23118 > Project: Hive > Issue Type: Improvement >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23118.1.patch, HIVE-23118.2.patch > > Time Spent: 20m > Remaining Estimate: 0h > > TezCounters currently are runtime only. Some compile time information from > optimizer can be exposed as counters which can then be used by workload > management to make runtime decisions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[
https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074138#comment-17074138
]
Hive QA commented on HIVE-23067:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998655/HIVE-23067.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21411/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21411/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21411/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-04-02 22:56:46.438
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-21411/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-04-02 22:56:46.441
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 7f9ce94 HIVE-23094: Implement Explain CBO of Update and Delete
statements (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 7f9ce94 HIVE-23094: Implement Explain CBO of Update and Delete
statements (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-04-02 22:56:47.755
+ rm -rf ../yetus_PreCommit-HIVE-Build-21411
+ mkdir ../yetus_PreCommit-HIVE-Build-21411
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-21411
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-21411/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:190
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java'
with conflicts.
Going to apply patch with: git apply -p0
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:190
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java'
with conflicts.
U
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-21411
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998655 - PreCommit-HIVE-Build
> Use batch DB calls in TxnHandler for commitTxn and abortTxns
>
>
> Key: HIVE-23067
> URL: https://issues.apache.org/jira/browse/HIVE-23067
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch,
> HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch,
> HIVE-23067.2.patch, HIVE-23067.4.patch, HIVE-23067.5.patch,
> HIVE-23067.5.patch, HIVE-23067.5.patch, HIVE-23067.5.patch
>
>
> To reduce the number of database calls and network roundtrips, we could use
> more batching in TxnHandler, where currently in many places we call insert
> commands in loops sequentially.
> Some examples:
> * openTxns (TXNS, REPL_TXN_MAP)
> * commitTxn
[jira] [Commented] (HIVE-23128) SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
[
https://issues.apache.org/jira/browse/HIVE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074135#comment-17074135
]
Hive QA commented on HIVE-23128:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998653/HIVE-23128.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 18167 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[whroot_external1]
(batchId=162)
org.apache.hadoop.hive.metastore.client.TestGetTableMeta.catalogPatternsDontWork[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestGetTableMeta.noSuchCatalog[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestGetTableMeta.tablesInDifferentCatalog[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestGetTableMeta.testGetTableMetaCaseSensitive[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestGetTableMeta.testGetTableMetaNullNoDbNoTbl[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestGetTableMeta.testGetTableMetaNullOrEmptyDb[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestGetTableMeta.testGetTableMetaNullOrEmptyTbl[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestGetTableMeta.testGetTableMetaNullOrEmptyTypes[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestGetTableMeta.testGetTableMeta[Remote]
(batchId=230)
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
(batchId=285)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21410/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21410/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21410/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998653 - PreCommit-HIVE-Build
> SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
> --
>
> Key: HIVE-23128
> URL: https://issues.apache.org/jira/browse/HIVE-23128
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.4.0, 3.1.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23128.1.patch
>
>
> {code:sql}
> show create table `sample_07`;
> show create table `default`.`sample_07`;
> show create table `default.sample_07`;
> {code}
> {code:none|title=Results}
> CREATE TABLE `sample_07`(...)
> CREATE TABLE `default.sample_07`(...)
> CREATE TABLE `default.sample_07`(...);
> {code}
> All three {{show create table}} statements complete in Hive 2.x and 3.x and
> generate {{CREATE TABLE}} statements as show above. The first result is
> correct because it does not include the database name, however, the
> subsequent two results are invalid: each field must be quoted individually.
> This causes a failure in recent versions of Hive because "SemanticException
> Table or database name may not contain dot(.) character."
> {quote}If any components of a multiple-part name require quoting, quote them
> individually rather than quoting the name as a whole. For example, write
> `my-table`.`my-column`, not `my-table.my-column`.
> * [https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html]{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23128) SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
[
https://issues.apache.org/jira/browse/HIVE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074110#comment-17074110
]
Hive QA commented on HIVE-23128:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 0 unchanged - 0 fixed
= 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21410/dev-support/hive-personality.sh
|
| git revision | master / 7f9ce94 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21410/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21410/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
> --
>
> Key: HIVE-23128
> URL: https://issues.apache.org/jira/browse/HIVE-23128
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.4.0, 3.1.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23128.1.patch
>
>
> {code:sql}
> show create table `sample_07`;
> show create table `default`.`sample_07`;
> show create table `default.sample_07`;
> {code}
> {code:none|title=Results}
> CREATE TABLE `sample_07`(...)
> CREATE TABLE `default.sample_07`(...)
> CREATE TABLE `default.sample_07`(...);
> {code}
> All three {{show create table}} statements complete in Hive 2.x and 3.x and
> generate {{CREATE TABLE}} statements as show above. The first result is
> correct because it does not include the database name, however, the
> subsequent two results are invalid: each field must be quoted individually.
> This causes a failure in recent versions of Hive because "SemanticException
> Table or database name may not contain dot(.) character."
> {quote}If any components of a multiple-part name require quoting, quote them
> individually rather than quoting the name as a whole. For example, write
> `my-table`.`my-column`, not `my-table.my-column`.
> * [https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html]{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23118) Option for exposing compile time counters as tez counters
[ https://issues.apache.org/jira/browse/HIVE-23118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074108#comment-17074108 ] Jesus Camacho Rodriguez commented on HIVE-23118: Left a small comment in the patch. Other than that, LGTM. +1 (pending tests) > Option for exposing compile time counters as tez counters > - > > Key: HIVE-23118 > URL: https://issues.apache.org/jira/browse/HIVE-23118 > Project: Hive > Issue Type: Improvement >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23118.1.patch > > Time Spent: 20m > Remaining Estimate: 0h > > TezCounters currently are runtime only. Some compile time information from > optimizer can be exposed as counters which can then be used by workload > management to make runtime decisions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23118) Option for exposing compile time counters as tez counters
[
https://issues.apache.org/jira/browse/HIVE-23118?focusedWorklogId=415070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415070
]
ASF GitHub Bot logged work on HIVE-23118:
-
Author: ASF GitHub Bot
Created on: 02/Apr/20 21:48
Start Date: 02/Apr/20 21:48
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #966: HIVE-23118:
Option for exposing compile time counters as tez counters
URL: https://github.com/apache/hive/pull/966#discussion_r402615414
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java
##
@@ -185,6 +188,12 @@ public int monitorExecution() {
if (wmContext != null) {
Set desiredCounters = wmContext.getSubscribedCounters();
TezCounters dagCounters = status.getDAGCounters();
+ // if initial counters exists, merge it with dag counters to get
aggregated view
Review comment:
Can we move this if block to a utility static function `mergeCounters`? It
seems it is repeated in a couple of places, may be useful in other cases in the
future.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 415070)
Time Spent: 20m (was: 10m)
> Option for exposing compile time counters as tez counters
> -
>
> Key: HIVE-23118
> URL: https://issues.apache.org/jira/browse/HIVE-23118
> Project: Hive
> Issue Type: Improvement
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23118.1.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> TezCounters currently are runtime only. Some compile time information from
> optimizer can be exposed as counters which can then be used by workload
> management to make runtime decisions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23067: -- Attachment: HIVE-23067.5.patch > Use batch DB calls in TxnHandler for commitTxn and abortTxns > > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch, > HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch, > HIVE-23067.2.patch, HIVE-23067.4.patch, HIVE-23067.5.patch, > HIVE-23067.5.patch, HIVE-23067.5.patch, HIVE-23067.5.patch > > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23128) SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
[
https://issues.apache.org/jira/browse/HIVE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23128:
--
Description:
{code:sql}
show create table `sample_07`;
show create table `default`.`sample_07`;
show create table `default.sample_07`;
{code}
{code:none|title=Results}
CREATE TABLE `sample_07`(...)
CREATE TABLE `default.sample_07`(...)
CREATE TABLE `default.sample_07`(...);
{code}
All three {{show create table}} statements complete in Hive 2.x and 3.x and
generate {{CREATE TABLE}} statements as show above. The first result is correct
because it does not include the database name, however, the subsequent two
results are invalid: each field must be quoted individually. This causes a
failure in recent versions of Hive because "SemanticException Table or database
name may not contain dot(.) character."
{quote}If any components of a multiple-part name require quoting, quote them
individually rather than quoting the name as a whole. For example, write
`my-table`.`my-column`, not `my-table.my-column`.
* [https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html]{quote}
was:
{code:sql}
show create table `sample_07`;
show create table `default`.`sample_07`;
show create table `default.sample_07`;
{code}
{code:none|title=Results}
CREATE TABLE `sample_07`(...)
CREATE TABLE `default.sample_07`(...)
CREATE TABLE `default.sample_07`(...);
{code}
All three queries complete in Hive 2.x (not sure about 3.x) but the results are
consistent in both versions. The first result is correct because it does not
include the database name, however, the subsequent two results are invalid:
each field must be quoted individually.
{quote}
If any components of a multiple-part name require quoting, quote them
individually rather than quoting the name as a whole. For example, write
`my-table`.`my-column`, not `my-table.my-column`.
* https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html
{quote}
> SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
> --
>
> Key: HIVE-23128
> URL: https://issues.apache.org/jira/browse/HIVE-23128
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.4.0, 3.1.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23128.1.patch
>
>
> {code:sql}
> show create table `sample_07`;
> show create table `default`.`sample_07`;
> show create table `default.sample_07`;
> {code}
> {code:none|title=Results}
> CREATE TABLE `sample_07`(...)
> CREATE TABLE `default.sample_07`(...)
> CREATE TABLE `default.sample_07`(...);
> {code}
> All three {{show create table}} statements complete in Hive 2.x and 3.x and
> generate {{CREATE TABLE}} statements as show above. The first result is
> correct because it does not include the database name, however, the
> subsequent two results are invalid: each field must be quoted individually.
> This causes a failure in recent versions of Hive because "SemanticException
> Table or database name may not contain dot(.) character."
> {quote}If any components of a multiple-part name require quoting, quote them
> individually rather than quoting the name as a whole. For example, write
> `my-table`.`my-column`, not `my-table.my-column`.
> * [https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html]{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23128) SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
[
https://issues.apache.org/jira/browse/HIVE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23128:
--
Attachment: HIVE-23128.1.patch
> SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
> --
>
> Key: HIVE-23128
> URL: https://issues.apache.org/jira/browse/HIVE-23128
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.4.0, 3.1.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23128.1.patch
>
>
> {code:sql}
> show create table `sample_07`;
> show create table `default`.`sample_07`;
> show create table `default.sample_07`;
> {code}
> {code:none|title=Results}
> CREATE TABLE `sample_07`(...)
> CREATE TABLE `default.sample_07`(...)
> CREATE TABLE `default.sample_07`(...);
> {code}
> All three queries complete in Hive 2.x (not sure about 3.x) but the results
> are consistent in both versions. The first result is correct because it does
> not include the database name, however, the subsequent two results are
> invalid: each field must be quoted individually.
> {quote}
> If any components of a multiple-part name require quoting, quote them
> individually rather than quoting the name as a whole. For example, write
> `my-table`.`my-column`, not `my-table.my-column`.
> * https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23128) SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
[
https://issues.apache.org/jira/browse/HIVE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23128:
--
Status: Patch Available (was: Open)
> SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
> --
>
> Key: HIVE-23128
> URL: https://issues.apache.org/jira/browse/HIVE-23128
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.4.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23128.1.patch
>
>
> {code:sql}
> show create table `sample_07`;
> show create table `default`.`sample_07`;
> show create table `default.sample_07`;
> {code}
> {code:none|title=Results}
> CREATE TABLE `sample_07`(...)
> CREATE TABLE `default.sample_07`(...)
> CREATE TABLE `default.sample_07`(...);
> {code}
> All three queries complete in Hive 2.x (not sure about 3.x) but the results
> are consistent in both versions. The first result is correct because it does
> not include the database name, however, the subsequent two results are
> invalid: each field must be quoted individually.
> {quote}
> If any components of a multiple-part name require quoting, quote them
> individually rather than quoting the name as a whole. For example, write
> `my-table`.`my-column`, not `my-table.my-column`.
> * https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23128) SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
[
https://issues.apache.org/jira/browse/HIVE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-23128:
-
Assignee: David Mollitor
> SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
> --
>
> Key: HIVE-23128
> URL: https://issues.apache.org/jira/browse/HIVE-23128
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.4.0, 3.1.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> {code:sql}
> show create table `sample_07`;
> show create table `default`.`sample_07`;
> show create table `default.sample_07`;
> {code}
> {code:none|title=Results}
> CREATE TABLE `sample_07`(...)
> CREATE TABLE `default.sample_07`(...)
> CREATE TABLE `default.sample_07`(...);
> {code}
> All three queries complete in Hive 2.x (not sure about 3.x) but the results
> are consistent in both versions. The first result is correct because it does
> not include the database name, however, the subsequent two results are
> invalid: each field must be quoted individually.
> {quote}
> If any components of a multiple-part name require quoting, quote them
> individually rather than quoting the name as a whole. For example, write
> `my-table`.`my-column`, not `my-table.my-column`.
> * https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23067: -- Attachment: HIVE-23067.5.patch > Use batch DB calls in TxnHandler for commitTxn and abortTxns > > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch, > HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch, > HIVE-23067.2.patch, HIVE-23067.4.patch, HIVE-23067.5.patch, > HIVE-23067.5.patch, HIVE-23067.5.patch > > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23094) Implement Explain CBO of Update and Delete statements
[
https://issues.apache.org/jira/browse/HIVE-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23094:
---
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~kkasa]!
> Implement Explain CBO of Update and Delete statements
> -
>
> Key: HIVE-23094
> URL: https://issues.apache.org/jira/browse/HIVE-23094
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23094.1.patch
>
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create table acidtlb(a int, b int) clustered by (a) into 2 buckets stored as
> orc TBLPROPERTIES ('transactional'='true');
> explain cbo
> update acidtlb set b=777;
> {code}
> doesn't print CBO plan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23094) Implement Explain CBO of Update and Delete statements
[
https://issues.apache.org/jira/browse/HIVE-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074072#comment-17074072
]
Jesus Camacho Rodriguez commented on HIVE-23094:
+1
Can we create a new JIRA and add some merge statements too? Will that work?
Thanks
> Implement Explain CBO of Update and Delete statements
> -
>
> Key: HIVE-23094
> URL: https://issues.apache.org/jira/browse/HIVE-23094
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23094.1.patch
>
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create table acidtlb(a int, b int) clustered by (a) into 2 buckets stored as
> orc TBLPROPERTIES ('transactional'='true');
> explain cbo
> update acidtlb set b=777;
> {code}
> doesn't print CBO plan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23124) Review of SQLOperation Class
[
https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074064#comment-17074064
]
Hive QA commented on HIVE-23124:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998473/HIVE-23124.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18167 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21391/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21391/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21391/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998473 - PreCommit-HIVE-Build
> Review of SQLOperation Class
>
>
> Key: HIVE-23124
> URL: https://issues.apache.org/jira/browse/HIVE-23124
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23124.1.patch
>
>
> * Use ConcurrentHashMap instead of synchronized methods to improve
> multi-threaded access
> * Use JDK 8 facilities where applicable
> * General cleanup
> * Better log messages and Exception messages
> * Use {{switch}} statement instead of if/else blocks
> * Checkstyle fixes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23131) Remove ql/src/test/results/clientnegative/orc_type_promotion3_acid.q
[ https://issues.apache.org/jira/browse/HIVE-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23131: -- Status: Patch Available (was: Open) > Remove ql/src/test/results/clientnegative/orc_type_promotion3_acid.q > > > Key: HIVE-23131 > URL: https://issues.apache.org/jira/browse/HIVE-23131 > Project: Hive > Issue Type: Improvement >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23131.01.patch > > > ql/src/test/results/clientnegative/orc_type_promotion3_acid.q is a q file in > the results directory, got there by accident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23131) Remove ql/src/test/results/clientnegative/orc_type_promotion3_acid.q
[ https://issues.apache.org/jira/browse/HIVE-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23131: -- Attachment: HIVE-23131.01.patch > Remove ql/src/test/results/clientnegative/orc_type_promotion3_acid.q > > > Key: HIVE-23131 > URL: https://issues.apache.org/jira/browse/HIVE-23131 > Project: Hive > Issue Type: Improvement >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23131.01.patch > > > ql/src/test/results/clientnegative/orc_type_promotion3_acid.q is a q file in > the results directory, got there by accident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23131) Remove ql/src/test/results/clientnegative/orc_type_promotion3_acid.q
[ https://issues.apache.org/jira/browse/HIVE-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-23131: - > Remove ql/src/test/results/clientnegative/orc_type_promotion3_acid.q > > > Key: HIVE-23131 > URL: https://issues.apache.org/jira/browse/HIVE-23131 > Project: Hive > Issue Type: Improvement >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > > ql/src/test/results/clientnegative/orc_type_promotion3_acid.q is a q file in > the results directory, got there by accident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23123) Disable export/import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Summary: Disable export/import of views and materialized views (was:
Disable export/Import of views and materialized views)
> Disable export/import of views and materialized views
> -
>
> Key: HIVE-23123
> URL: https://issues.apache.org/jira/browse/HIVE-23123
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
> import and export can be done by using the
> {code:java}
> export table ...
> import table ...
> {code}
> commands. The document doesn't mention views or materialized views at all,
> and in fact we don't support commands like
> {code:java}
> export view ...
> import view ...
> export materialized view ...
> import materialized view ...
> {code}
> they can not be parsed at all. The word table is often used though in a
> broader sense, when it means all table like entities, including views and
> materialized views. For example the various Table classes may represent any
> of these as well.
> If I try to export a view with the export table ... command, it goes fine. A
> _metadata file will be created, but no data directory, which is what we'd
> expect. If I try to import it back, an exception is thrown due to the lack of
> the data dir:
> {code:java}
> java.lang.AssertionError: null==getPath() for exim_view
> at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
> {code}
> Still the view gets imported successfully, as data movement wasn't even
> necessary.
> If we try to export a materialized view which is transactional, then this
> exception occurs:
> {code:java}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
> exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
> at
> org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
> {code}
> So the export process can not handle it, as the temporary table is not
> getting created.
>
> The import command handling have a lot of codes dedicated to importing views
> and materialized views, which suggests that we support the importing (and
> thus also suggests implicitly that we support the exporting) of views and
> materialiezed views.
>
> So the conclusion is that we have to decide if we support exporting/importing
> of views and materialized views.
> If we decide not to support them then:
> - export process should throw an exception if a view or materialized view is
> the subject
> - the codes specific to view imports should be removed
> If we decide to support them, then:
> - the commands mentioned above above should be introduced
> - exception should be thrown if not the proper command used (e.g. export
> view on a table)
> - the exceptions mentioned above should be fixed
>
> I prefer not to support them, I don't think we should support the exporting /
> importing of views. The point of exporting / importing is the transfer of
> data, not DDL, it causes more issues than it solves. Our current
> documentation also suggests that
[jira] [Commented] (HIVE-23124) Review of SQLOperation Class
[
https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074020#comment-17074020
]
Hive QA commented on HIVE-23124:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} service in master has 50 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} service: The patch generated 1 new + 14 unchanged - 1
fixed = 15 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21391/dev-support/hive-personality.sh
|
| git revision | master / 29d6967 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21391/yetus/diff-checkstyle-service.txt
|
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21391/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review of SQLOperation Class
>
>
> Key: HIVE-23124
> URL: https://issues.apache.org/jira/browse/HIVE-23124
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23124.1.patch
>
>
> * Use ConcurrentHashMap instead of synchronized methods to improve
> multi-threaded access
> * Use JDK 8 facilities where applicable
> * General cleanup
> * Better log messages and Exception messages
> * Use {{switch}} statement instead of if/else blocks
> * Checkstyle fixes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23106) Cleanup CalcitePlanner genOPTree exception handling
[ https://issues.apache.org/jira/browse/HIVE-23106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23106: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks for your contribution [~jfs]! > Cleanup CalcitePlanner genOPTree exception handling > --- > > Key: HIVE-23106 > URL: https://issues.apache.org/jira/browse/HIVE-23106 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-23106.1.patch, HIVE-23106.2.patch > > > The logic where genOPTree handles exceptions during CBO planning is a bit > twisty and could use some cleanup and comments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Attachment: HIVE-23039.05.patch Status: Patch Available (was: In Progress) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Status: In Progress (was: Patch Available) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch, HIVE-23039.05.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23123) Disable export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Summary: Disable export/Import of views and materialized views (was:
Export/Import of views and materialized views)
> Disable export/Import of views and materialized views
> -
>
> Key: HIVE-23123
> URL: https://issues.apache.org/jira/browse/HIVE-23123
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
> import and export can be done by using the
> {code:java}
> export table ...
> import table ...
> {code}
> commands. The document doesn't mention views or materialized views at all,
> and in fact we don't support commands like
> {code:java}
> export view ...
> import view ...
> export materialized view ...
> import materialized view ...
> {code}
> they can not be parsed at all. The word table is often used though in a
> broader sense, when it means all table like entities, including views and
> materialized views. For example the various Table classes may represent any
> of these as well.
> If I try to export a view with the export table ... command, it goes fine. A
> _metadata file will be created, but no data directory, which is what we'd
> expect. If I try to import it back, an exception is thrown due to the lack of
> the data dir:
> {code:java}
> java.lang.AssertionError: null==getPath() for exim_view
> at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
> {code}
> Still the view gets imported successfully, as data movement wasn't even
> necessary.
> If we try to export a materialized view which is transactional, then this
> exception occurs:
> {code:java}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
> exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
> at
> org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
> {code}
> So the export process can not handle it, as the temporary table is not
> getting created.
>
> The import command handling have a lot of codes dedicated to importing views
> and materialized views, which suggests that we support the importing (and
> thus also suggests implicitly that we support the exporting) of views and
> materialiezed views.
>
> So the conclusion is that we have to decide if we support exporting/importing
> of views and materialized views.
> If we decide not to support them then:
> - export process should throw an exception if a view or materialized view is
> the subject
> - the codes specific to view imports should be removed
> If we decide to support them, then:
> - the commands mentioned above above should be introduced
> - exception should be thrown if not the proper command used (e.g. export
> view on a table)
> - the exceptions mentioned above should be fixed
> I prefer not to support them, I don't think we should support the exporting /
> importing of views. The point of exporting / importing is the transfer of
> data, not DDL, it causes more issues than it solves. Our current
> documentation also suggests that it is only
[jira] [Updated] (HIVE-23123) Disable export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Description:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of these as well.
If I try to export a view with the export table ... command, it goes fine. A
_metadata file will be created, but no data directory, which is what we'd
expect. If I try to import it back, an exception is thrown due to the lack of
the data dir:
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
{code}
Still the view gets imported successfully, as data movement wasn't even
necessary.
If we try to export a materialized view which is transactional, then this
exception occurs:
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
at
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
{code}
So the export process can not handle it, as the temporary table is not getting
created.
The import command handling have a lot of codes dedicated to importing views
and materialized views, which suggests that we support the importing (and thus
also suggests implicitly that we support the exporting) of views and
materialiezed views.
So the conclusion is that we have to decide if we support exporting/importing
of views and materialized views.
If we decide not to support them then:
- export process should throw an exception if a view or materialized view is
the subject
- the codes specific to view imports should be removed
If we decide to support them, then:
- the commands mentioned above above should be introduced
- exception should be thrown if not the proper command used (e.g. export view
on a table)
- the exceptions mentioned above should be fixed
I prefer not to support them, I don't think we should support the exporting /
importing of views. The point of exporting / importing is the transfer of data,
not DDL, it causes more issues than it solves. Our current documentation also
suggests that it is only supported for tables.
was:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any
[jira] [Commented] (HIVE-23089) Add constraint checks to CBO plan
[
https://issues.apache.org/jira/browse/HIVE-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073986#comment-17073986
]
Hive QA commented on HIVE-23089:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998472/HIVE-23089.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 18163 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=171)
org.apache.hive.jdbc.TestNewGetSplitsFormatReturnPath.testInvalidReferenceCountScenario
(batchId=295)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21390/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21390/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21390/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998472 - PreCommit-HIVE-Build
> Add constraint checks to CBO plan
> -
>
> Key: HIVE-23089
> URL: https://issues.apache.org/jira/browse/HIVE-23089
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23089.1.patch
>
>
> {code}
> create table acid_uami(i int,
> de decimal(5,2) constraint nn1 not null enforced,
> vc varchar(128) constraint nn2 not null enforced) clustered
> by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> explain
> update acid_uami set de=null where i=1;
> {code}
> Non-CBO path:
> {code:java}
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> {code}
> CBO path:
> {code:java}
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: (i = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: (i = 1) (type: boolean)
> ...
> Reduce Operator Tree:
> ...
> Filter Operator
> predicate: enforce_constraint((null is not null and _col3 is not
> null)) (type: boolean)
> {code}
> In CBO path the enforce_constraint function is added to the plan when CBO
> plan is already generated and optimized.
> {code}
> HiveSortExchange(distribution=[any], collation=[[0]])
> HiveProject(row__id=[$5], i=[CAST(1):INTEGER], _o__c2=[null:NULL], vc=[$2])
> HiveFilter(condition=[=($0, 1)])
> HiveTableScan(table=[[default, acid_uami]], table:alias=[acid_uami])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23089) Add constraint checks to CBO plan
[
https://issues.apache.org/jira/browse/HIVE-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073933#comment-17073933
]
Hive QA commented on HIVE-23089:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} ql: The patch generated 0 new + 476 unchanged - 1
fixed = 476 total (was 477) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21390/dev-support/hive-personality.sh
|
| git revision | master / c3afb57 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21390/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add constraint checks to CBO plan
> -
>
> Key: HIVE-23089
> URL: https://issues.apache.org/jira/browse/HIVE-23089
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23089.1.patch
>
>
> {code}
> create table acid_uami(i int,
> de decimal(5,2) constraint nn1 not null enforced,
> vc varchar(128) constraint nn2 not null enforced) clustered
> by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> explain
> update acid_uami set de=null where i=1;
> {code}
> Non-CBO path:
> {code:java}
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> {code}
> CBO path:
> {code:java}
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: (i = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: (i = 1) (type: boolean)
> ...
> Reduce Operator Tree:
> ...
> Filter Operator
> predicate: enforce_constraint((null is not null and _col3
[jira] [Commented] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073919#comment-17073919
]
Thejas Nair commented on HIVE-22995:
+1
> Add support for location for managed tables on database
> ---
>
> Key: HIVE-22995
> URL: https://issues.apache.org/jira/browse/HIVE-22995
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22995.1.patch, HIVE-22995.10.patch,
> HIVE-22995.11.patch, HIVE-22995.12.patch, HIVE-22995.2.patch,
> HIVE-22995.3.patch, HIVE-22995.4.patch, HIVE-22995.5.patch,
> HIVE-22995.6.patch, HIVE-22995.7.patch, HIVE-22995.8.patch,
> HIVE-22995.9.patch, Hive Metastore Support for Tenant-based storage
> heirarchy.pdf
>
>
> I have attached the initial spec to this jira.
> Default location for database would be the external table base directory.
> Managed location can be optionally specified.
> {code}
> CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
> [COMMENT database_comment]
> [LOCATION hdfs_path]
> [MANAGEDLOCATION hdfs_path]
> [WITH DBPROPERTIES (property_name=property_value, ...)];
> ALTER (DATABASE|SCHEMA) database_name SET
> MANAGEDLOCATION
> hdfs_path;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23109) Query-based compaction omits database
[
https://issues.apache.org/jira/browse/HIVE-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073918#comment-17073918
]
László Pintér commented on HIVE-23109:
--
Thanks for the patch [~klcopp]. Pushed to master.
> Query-based compaction omits database
> -
>
> Key: HIVE-23109
> URL: https://issues.apache.org/jira/browse/HIVE-23109
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23109.01.patch, HIVE-23109.02.patch
>
>
> E.g. MM major compaction query looks like:
> {code:java}
> insert into tmp_table select * from src_table;
> {code}
> it should be
> {code:java}
> insert into tmp_table select * from src_db.src_table;
> {code}
> Therefore compaction fails if db of source table isn't default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23109) Query-based compaction omits database
[
https://issues.apache.org/jira/browse/HIVE-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Pintér updated HIVE-23109:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Query-based compaction omits database
> -
>
> Key: HIVE-23109
> URL: https://issues.apache.org/jira/browse/HIVE-23109
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23109.01.patch, HIVE-23109.02.patch
>
>
> E.g. MM major compaction query looks like:
> {code:java}
> insert into tmp_table select * from src_table;
> {code}
> it should be
> {code:java}
> insert into tmp_table select * from src_db.src_table;
> {code}
> Therefore compaction fails if db of source table isn't default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23113) Clean Up HiveCallableStatement
[
https://issues.apache.org/jira/browse/HIVE-23113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073913#comment-17073913
]
Hive QA commented on HIVE-23113:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998461/HIVE-23113.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18163 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21389/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21389/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21389/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998461 - PreCommit-HIVE-Build
> Clean Up HiveCallableStatement
> --
>
> Key: HIVE-23113
> URL: https://issues.apache.org/jira/browse/HIVE-23113
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23113.1.patch, HIVE-23113.1.patch
>
>
> * Add a useful class comment
> * Remove all non-javadoc comments
> * Remove 'TODO' tags
> * Add {{@override}} tags
> * Checkstyle formatting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23113) Clean Up HiveCallableStatement
[
https://issues.apache.org/jira/browse/HIVE-23113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073854#comment-17073854
]
Hive QA commented on HIVE-23113:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} jdbc: The patch generated 0 new + 0 unchanged - 1
fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21389/dev-support/hive-personality.sh
|
| git revision | master / c3afb57 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21389/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean Up HiveCallableStatement
> --
>
> Key: HIVE-23113
> URL: https://issues.apache.org/jira/browse/HIVE-23113
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23113.1.patch, HIVE-23113.1.patch
>
>
> * Add a useful class comment
> * Remove all non-javadoc comments
> * Remove 'TODO' tags
> * Add {{@override}} tags
> * Checkstyle formatting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23120) TopNKey related tests should be run by TestMiniLlapLocalCliDriver only
[ https://issues.apache.org/jira/browse/HIVE-23120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073850#comment-17073850 ] Jesus Camacho Rodriguez commented on HIVE-23120: [~kkasa], the new .q.out files that you be in llap directory are not included in the patch. Other than that, LGTM. +1 (pending tests) > TopNKey related tests should be run by TestMiniLlapLocalCliDriver only > -- > > Key: HIVE-23120 > URL: https://issues.apache.org/jira/browse/HIVE-23120 > Project: Hive > Issue Type: Task > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23120.1.patch > > > TopNKey optimization is only used when the execution framework is Tez. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23130) User friendly error message when MV rewriting fails
[
https://issues.apache.org/jira/browse/HIVE-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073845#comment-17073845
]
Jesus Camacho Rodriguez commented on HIVE-23130:
+1 (pending tests)
> User friendly error message when MV rewriting fails
> ---
>
> Key: HIVE-23130
> URL: https://issues.apache.org/jira/browse/HIVE-23130
> Project: Hive
> Issue Type: Task
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Attachments: HIVE-23130.1.patch
>
>
> If materialized view rewriting fails because an unsupported SQL clause or
> operator is used we got an error message like this:
> {code}
> FAILED: SemanticException Cannot enable automatic rewriting for materialized
> view. Unsupported RelNode type HiveSortExchange encountered in the query plan
> {code}
> This refers to the *HiveSortExchange* operator. This is introduced to CBO
> plan if the statement has *sort by* clause which may not clear for the user.
> {code}
> create materialized view cmv_mat_view as select a, b, c from cmv_basetable
> sort by a;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073834#comment-17073834
]
Hive QA commented on HIVE-23122:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998463/HIVE-23122.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18162 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21388/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21388/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21388/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998463 - PreCommit-HIVE-Build
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state
[
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pablo Junge updated HIVE-22566:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Drop table involved in materialized view leaves the table in inconsistent
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.05.patch,
> HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created
> materialized view, the table is not dropped, which is the desired state as it
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did
> not succeed, leaving it in an inconsistent state.
>
> Repro:
> ---
> 1) Create tables:
>
> {code:java}
> CREATE TABLE emps ( empid INT, deptno INT, name VARCHAR(256), salary
> FLOAT, hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES
> ('transactional'='true');
> CREATE TABLE depts ( deptno INT, deptname VARCHAR(256), locationid
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>
> 2) Create the VM:
>
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps
> JOIN depts ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>
> 3) Following is in backend database at this point:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | 81 | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>
> 4) Let's drop the 'emps' table:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO : Compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 0.05 seconds
> INFO : Executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 10.281 seconds
> INFO : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>
> 5) List tables:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO : Compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.041 seconds
> INFO : Executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.016 seconds
> INFO : OK
> +---+
> | tab_name |
> +---+
> | depts |
> | emps |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>
> 6) Now, from the backend-db point of view:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | NULL | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>
> 7) From
[jira] [Updated] (HIVE-5312) Let HiveServer2 run simultaneously in HTTP (over thrift) and Binary (normal thrift transport) mode
[ https://issues.apache.org/jira/browse/HIVE-5312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-5312: - Attachment: HIVE-5312.4.patch > Let HiveServer2 run simultaneously in HTTP (over thrift) and Binary (normal > thrift transport) mode > --- > > Key: HIVE-5312 > URL: https://issues.apache.org/jira/browse/HIVE-5312 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Vaibhav Gumashta >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-5132.branch-2.3.patch, HIVE-5312.1.patch, > HIVE-5312.2.patch, HIVE-5312.3.patch, HIVE-5312.4.patch > > > [HIVE-4763|https://issues.apache.org/jira/browse/HIVE-4763] adds support for > HTTP transport over thrift. With that, HS2 can be configured to run either > using using HTTP or using normal thrift binary transport. Ideally HS2 should > be supporting both modes simultaneously and the client should be able to > specify the mode used in serving the request. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073784#comment-17073784
]
Hive QA commented on HIVE-23122:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21388/dev-support/hive-personality.sh
|
| git revision | master / 8739c58 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21388/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22534) ACID: Improve Compactor thread logging
[ https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Pintér updated HIVE-22534: - Attachment: HIVE-22534.13.patch > ACID: Improve Compactor thread logging > -- > > Key: HIVE-22534 > URL: https://issues.apache.org/jira/browse/HIVE-22534 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: László Pintér >Priority: Minor > Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch, > HIVE-22534.03.patch, HIVE-22534.04.patch, HIVE-22534.05.patch, > HIVE-22534.06.patch, HIVE-22534.07.patch, HIVE-22534.08.patch, > HIVE-22534.09.patch, HIVE-22534.10.patch, HIVE-22534.11.patch, > HIVE-22534.12.patch, HIVE-22534.13.patch > > > Make sure that it is easy to find issues when one of the compactor thread > fails. > Maybe: > * MDC - with iteration / threadname - so we can easily grep the logs for a > given run > * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073727#comment-17073727
]
László Bodor commented on HIVE-23122:
-
thanks for taking a look [~rajesh.balamohan]! are you sure that it's needed on
DEBUG level? it's already on INFO level, and this logging only involves
primitives, so an expectable amount of characters...perhaps we can add the new
extended line on DEBUG level, and the old one on INFO level?
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23067: -- Attachment: HIVE-23067.5.patch > Use batch DB calls in TxnHandler for commitTxn and abortTxns > > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch, > HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch, > HIVE-23067.2.patch, HIVE-23067.4.patch, HIVE-23067.5.patch, HIVE-23067.5.patch > > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073722#comment-17073722
]
Hive QA commented on HIVE-21304:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 6 new + 907 unchanged - 6
fixed = 913 total (was 913) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
4s{color} | {color:red} ql generated 3 new + 1528 unchanged - 0 fixed = 1531
total (was 1528) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Suspicious comparison of Integer references in
org.apache.hadoop.hive.ql.optimizer.BucketVersionPopulator$BucketingVersionResult.merge(BucketVersionPopulator$BucketingVersionResult)
At BucketVersionPopulator.java:in
org.apache.hadoop.hive.ql.optimizer.BucketVersionPopulator$BucketingVersionResult.merge(BucketVersionPopulator$BucketingVersionResult)
At BucketVersionPopulator.java:[line 64] |
| | Suspicious comparison of Integer references in
org.apache.hadoop.hive.ql.optimizer.BucketVersionPopulator$BucketingVersionResult.merge2(BucketVersionPopulator$BucketingVersionResult)
At BucketVersionPopulator.java:in
org.apache.hadoop.hive.ql.optimizer.BucketVersionPopulator$BucketingVersionResult.merge2(BucketVersionPopulator$BucketingVersionResult)
At BucketVersionPopulator.java:[line 74] |
| | Nullcheck of table_desc at line 8208 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean) At
SemanticAnalyzer.java:8208 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean) At
SemanticAnalyzer.java:[line 8201] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21387/dev-support/hive-personality.sh
|
| git revision |
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073717#comment-17073717
]
Rajesh Balamohan commented on HIVE-23122:
-
Given that this is in hotpath, we can move to debug level, if this adds
significant logging in certain jobs.
+1.
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18083) Support UTF8 in MySQL Metastore Backend
[
https://issues.apache.org/jira/browse/HIVE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073713#comment-17073713
]
David Mollitor commented on HIVE-18083:
---
If Hive is going to be a drop-in replacement for MySQL/MariaDB (or something
approaching that) then it needs to support UTF-8:
{quote}
Certain objects within MySQL, including database, table, index, column, alias,
view, stored procedure, partition, tablespace, resource group and other object
names are known as identifiers.
...
# Permitted characters in quoted identifiers include the full Unicode Basic
Multilingual Plane (BMP), except U+
{quote}
* https://dev.mysql.com/doc/refman/8.0/en/identifiers.html
> Support UTF8 in MySQL Metastore Backend
> ---
>
> Key: HIVE-18083
> URL: https://issues.apache.org/jira/browse/HIVE-18083
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: David Mollitor
>Priority: Major
>
> {code:sql|title=hive-schema-2.2.0.mysql.sql}
> CREATE TABLE IF NOT EXISTS `COLUMNS_V2` (
> `CD_ID` bigint(20) NOT NULL,
> `COMMENT` varchar(256) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
> `COLUMN_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
> `TYPE_NAME` varchar(4000) DEFAULT NULL,
> `INTEGER_IDX` int(11) NOT NULL,
> PRIMARY KEY (`CD_ID`,`COLUMN_NAME`),
> KEY `COLUMNS_V2_N49` (`CD_ID`),
> CONSTRAINT `COLUMNS_V2_FK1` FOREIGN KEY (`CD_ID`) REFERENCES `CDS` (`CD_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> {code}
> Hive explicitly defines a {{CHARACTER SET latin1 COLLATE latin1_bin}} in the
> schema design. This explicit definition should either be removed, so that it
> can fallback onto the database administrator's defaults, or changed to
> {{CHARACTER SET utf8 COLLATE utf8_bin}} to change the explicit definition to
> utf8.
> This will allow Hive to support UTF8 characters in MySQL backend databases
> for our international friends.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[
https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073693#comment-17073693
]
Hive QA commented on HIVE-23067:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998567/HIVE-23067.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18162 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_ingest]
(batchId=185)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21386/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21386/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21386/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998567 - PreCommit-HIVE-Build
> Use batch DB calls in TxnHandler for commitTxn and abortTxns
>
>
> Key: HIVE-23067
> URL: https://issues.apache.org/jira/browse/HIVE-23067
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch,
> HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch,
> HIVE-23067.2.patch, HIVE-23067.4.patch, HIVE-23067.5.patch
>
>
> To reduce the number of database calls and network roundtrips, we could use
> more batching in TxnHandler, where currently in many places we call insert
> commands in loops sequentially.
> Some examples:
> * openTxns (TXNS, REPL_TXN_MAP)
> * commitTxn (COMPLETED_TXN_COMPONENTS)
> * replTableWriteIdState (TXN_TO_WRITE_ID)
> * allocateTableWriteIds (TXN_TO_WRITE_ID)
> *
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Attachment: HIVE-23039.04.patch Status: Patch Available (was: In Progress) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Status: In Progress (was: Patch Available) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch, HIVE-23039.04.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23052: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~Marton Bod] and [~dkuzmenko] for the review! > Optimize lock enqueueing in TxnHandler > -- > > Key: HIVE-23052 > URL: https://issues.apache.org/jira/browse/HIVE-23052 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23052.1.patch, HIVE-23052.10.patch, > HIVE-23052.11.patch, HIVE-23052.12.patch, HIVE-23052.13.patch, > HIVE-23052.14.patch, HIVE-23052.2.patch, HIVE-23052.3.patch, > HIVE-23052.4.patch, HIVE-23052.5.patch, HIVE-23052.6.patch, > HIVE-23052.7.patch, HIVE-23052.8.patch, HIVE-23052.8.patch, > HIVE-23052.8.patch, HIVE-23052.9.patch > > > * Reduce scope of next_lock_id select-for-update by moving the txn_component > inserts before the S4U + inserting the hive_locks entries before the S4U > (first with a temp ID, which will be replaced later in a single update). This > helps decrease the overall time that the next_lock_id table is locked, > thereby increasing concurrency > * Insert txn_components in a batch instead of one-by-one (also in > TxnHandler::addDynamicPartition) > * Increment next_lock_id and update hive_locks table in a single batch > statement > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[
https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073648#comment-17073648
]
Hive QA commented on HIVE-23067:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 2 new + 558 unchanged - 2 fixed = 560 total (was 560) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
24s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 190 unchanged - 0 fixed = 191 total (was 190) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | A prepared statement is generated from a nonconstant String in
org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(CommitTxnRequest)
At TxnHandler.java:from a nonconstant String in
org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(CommitTxnRequest)
At TxnHandler.java:[line 1299] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21386/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21386/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21386/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21386/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use batch DB calls in TxnHandler for commitTxn and abortTxns
>
>
> Key: HIVE-23067
> URL: https://issues.apache.org/jira/browse/HIVE-23067
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch,
> HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch,
> HIVE-23067.2.patch, HIVE-23067.4.patch, HIVE-23067.5.patch
>
>
> To reduce the number of database calls and network roundtrips, we could use
> more batching in TxnHandler, where currently in many places we call insert
> commands in loops sequentially.
>
[jira] [Assigned] (HIVE-22750) Consolidate LockType naming
[
https://issues.apache.org/jira/browse/HIVE-22750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Bod reassigned HIVE-22750:
-
Assignee: Marton Bod (was: Zoltan Chovan)
> Consolidate LockType naming
> ---
>
> Key: HIVE-22750
> URL: https://issues.apache.org/jira/browse/HIVE-22750
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Zoltan Chovan
>Assignee: Marton Bod
>Priority: Minor
>
> Extend enum with string literal to remove unnecessary `id` to `char` casting
> for the LockType:
> {code:java}
> switch (lockType) {
> case EXCLUSIVE:
> lockChar = LOCK_EXCLUSIVE;
> break;
> case SHARED_READ:
> lockChar = LOCK_SHARED;
> break;
> case SHARED_WRITE:
> lockChar = LOCK_SEMI_SHARED;
> break;
> }
> {code}
> Consolidate LockType naming in code and schema upgrade scripts:
> {code:java}
> CASE WHEN HL.`HL_LOCK_TYPE` = 'e' THEN 'exclusive' WHEN HL.`HL_LOCK_TYPE` =
> 'r' THEN 'shared' WHEN HL.`HL_LOCK_TYPE` = 'w' THEN *'semi-shared'* END AS
> LOCK_TYPE,
> {code}
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073626#comment-17073626
]
Hive QA commented on HIVE-23052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998429/HIVE-23052.14.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18162 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21385/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21385/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21385/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998429 - PreCommit-HIVE-Build
> Optimize lock enqueueing in TxnHandler
> --
>
> Key: HIVE-23052
> URL: https://issues.apache.org/jira/browse/HIVE-23052
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23052.1.patch, HIVE-23052.10.patch,
> HIVE-23052.11.patch, HIVE-23052.12.patch, HIVE-23052.13.patch,
> HIVE-23052.14.patch, HIVE-23052.2.patch, HIVE-23052.3.patch,
> HIVE-23052.4.patch, HIVE-23052.5.patch, HIVE-23052.6.patch,
> HIVE-23052.7.patch, HIVE-23052.8.patch, HIVE-23052.8.patch,
> HIVE-23052.8.patch, HIVE-23052.9.patch
>
>
> * Reduce scope of next_lock_id select-for-update by moving the txn_component
> inserts before the S4U + inserting the hive_locks entries before the S4U
> (first with a temp ID, which will be replaced later in a single update). This
> helps decrease the overall time that the next_lock_id table is locked,
> thereby increasing concurrency
> * Insert txn_components in a batch instead of one-by-one (also in
> TxnHandler::addDynamicPartition)
> * Increment next_lock_id and update hive_locks table in a single batch
> statement
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23129) Cast invalid string to date returns incorrect result
[
https://issues.apache.org/jira/browse/HIVE-23129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073624#comment-17073624
]
Renukaprasad C commented on HIVE-23129:
---
java.time.LocalDate.parse(CharSequence, DateTimeFormatter)
This Java API is being used to convert the date, this internally add the excess
months (>12) as year and remaing months gets converted (months % 12).
This same applies to date also.
LocalDate.parse("2020-1-36", PARSE_FORMATTER);
Results to - 2020-02-05
If required to handle, date validation should be done from hive side.
> Cast invalid string to date returns incorrect result
>
>
> Key: HIVE-23129
> URL: https://issues.apache.org/jira/browse/HIVE-23129
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: Yuming Wang
>Priority: Major
>
> {noformat}
> hive> select cast('2020-20-20' as date);
> OK
> 2021-08-20
> Time taken: 4.436 seconds, Fetched: 1 row(s)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23107) Remove MIN_HISTORY_LEVEL table
[ https://issues.apache.org/jira/browse/HIVE-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Pintér updated HIVE-23107: - Attachment: HIVE-23107.03.patch > Remove MIN_HISTORY_LEVEL table > -- > > Key: HIVE-23107 > URL: https://issues.apache.org/jira/browse/HIVE-23107 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Attachments: HIVE-23107.01.patch, HIVE-23107.02.patch, > HIVE-23107.03.patch > > > MIN_HISTORY_LEVEL table is used in two places: > * Cleaner uses it to decide if the files can be removed - this could be > replaced by adding a new column to compaction_queue storing the next_txn_id > when the change was committed, and before cleaning checking the minimum open > transaction id in the TXNS table > * Initiator uses it to decide if some items from TXN_TO_WRITE_ID table can > be removed. This could be replaced by using the WRITE_SET.WS_COMMIT_ID. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23076) Add batching for openTxn
[ https://issues.apache.org/jira/browse/HIVE-23076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23076: -- Attachment: HIVE-23076.10.patch > Add batching for openTxn > > > Key: HIVE-23076 > URL: https://issues.apache.org/jira/browse/HIVE-23076 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-23076.10.patch, HIVE-23076.2.patch, > HIVE-23076.3.patch, HIVE-23076.4.patch, HIVE-23076.5.patch, > HIVE-23076.6.patch, HIVE-23076.7.patch, HIVE-23076.8.patch, > HIVE-23076.9.patch, HIVE-23076.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073572#comment-17073572
]
Hive QA commented on HIVE-23052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 36 new + 535 unchanged - 36 fixed = 571 total (was 571) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 189 unchanged - 1 fixed = 189 total (was 190) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21385/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21385/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21385/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize lock enqueueing in TxnHandler
> --
>
> Key: HIVE-23052
> URL: https://issues.apache.org/jira/browse/HIVE-23052
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23052.1.patch, HIVE-23052.10.patch,
> HIVE-23052.11.patch, HIVE-23052.12.patch, HIVE-23052.13.patch,
> HIVE-23052.14.patch, HIVE-23052.2.patch, HIVE-23052.3.patch,
> HIVE-23052.4.patch, HIVE-23052.5.patch, HIVE-23052.6.patch,
> HIVE-23052.7.patch, HIVE-23052.8.patch, HIVE-23052.8.patch,
> HIVE-23052.8.patch, HIVE-23052.9.patch
>
>
> * Reduce scope of next_lock_id select-for-update by moving the txn_component
> inserts before the S4U + inserting the hive_locks entries before the S4U
> (first with a temp ID, which will be replaced later in a single update). This
> helps decrease the overall time that the next_lock_id table is locked,
> thereby increasing concurrency
> * Insert txn_components in a batch instead of one-by-one (also in
> TxnHandler::addDynamicPartition)
> * Increment next_lock_id and update hive_locks table in a single batch
> statement
>
--
This message was sent by Atlassian Jira
