[jira] [Assigned] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled
[
https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li reassigned HIVE-14557:
-
Assignee: Rui Li
> Nullpointer When both SkewJoin and Mapjoin Enabled
> ---
>
> Key: HIVE-14557
> URL: https://issues.apache.org/jira/browse/HIVE-14557
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Nemon Lou
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-14557.patch
>
>
> The following sql failed with return code 2 on mr.
> {noformat}
> create table a(id int,id1 int);
> create table b(id int,id1 int);
> create table c(id int,id1 int);
> set hive.optimize.skewjoin=true;
> select a.id,b.id,c.id1 from a,b,c where a.id=b.id and a.id1=c.id1;
> {noformat}
> Error log as follows:
> {noformat}
> 2016-08-17 21:13:42,081 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> Id =0
>
> Id =21
>
> Id =28
>
> Id =16
>
> <\Children>
> Id = 28 null<\Parent>
> <\FS>
> <\Children>
> Id = 21 nullId = 33
> Id =33
> null
> <\Children>
> <\Parent>
> <\HASHTABLEDUMMY><\Parent>
> <\MAPJOIN>
> <\Children>
> Id = 0 null<\Parent>
> <\TS>
> <\Children>
> <\Parent>
> <\MAP>
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[21]
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Initializing dummy operator
> 2016-08-17 21:13:42,086 INFO [main]
> org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0,
> RECORDS_IN:0,
> 2016-08-17 21:13:42,087 ERROR [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing
> operators - failing tree
> 2016-08-17 21:13:42,088 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:682)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled
[
https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-14557:
--
Attachment: HIVE-14557.2.patch
> Nullpointer When both SkewJoin and Mapjoin Enabled
> ---
>
> Key: HIVE-14557
> URL: https://issues.apache.org/jira/browse/HIVE-14557
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Nemon Lou
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-14557.2.patch, HIVE-14557.patch
>
>
> The following sql failed with return code 2 on mr.
> {noformat}
> create table a(id int,id1 int);
> create table b(id int,id1 int);
> create table c(id int,id1 int);
> set hive.optimize.skewjoin=true;
> select a.id,b.id,c.id1 from a,b,c where a.id=b.id and a.id1=c.id1;
> {noformat}
> Error log as follows:
> {noformat}
> 2016-08-17 21:13:42,081 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> Id =0
>
> Id =21
>
> Id =28
>
> Id =16
>
> <\Children>
> Id = 28 null<\Parent>
> <\FS>
> <\Children>
> Id = 21 nullId = 33
> Id =33
> null
> <\Children>
> <\Parent>
> <\HASHTABLEDUMMY><\Parent>
> <\MAPJOIN>
> <\Children>
> Id = 0 null<\Parent>
> <\TS>
> <\Children>
> <\Parent>
> <\MAP>
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[21]
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Initializing dummy operator
> 2016-08-17 21:13:42,086 INFO [main]
> org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0,
> RECORDS_IN:0,
> 2016-08-17 21:13:42,087 ERROR [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing
> operators - failing tree
> 2016-08-17 21:13:42,088 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:682)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled
[
https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569817#comment-16569817
]
Rui Li commented on HIVE-14557:
---
Upload a patch based on Nemon's solution.
> Nullpointer When both SkewJoin and Mapjoin Enabled
> ---
>
> Key: HIVE-14557
> URL: https://issues.apache.org/jira/browse/HIVE-14557
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Nemon Lou
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-14557.2.patch, HIVE-14557.patch
>
>
> The following sql failed with return code 2 on mr.
> {noformat}
> create table a(id int,id1 int);
> create table b(id int,id1 int);
> create table c(id int,id1 int);
> set hive.optimize.skewjoin=true;
> select a.id,b.id,c.id1 from a,b,c where a.id=b.id and a.id1=c.id1;
> {noformat}
> Error log as follows:
> {noformat}
> 2016-08-17 21:13:42,081 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> Id =0
>
> Id =21
>
> Id =28
>
> Id =16
>
> <\Children>
> Id = 28 null<\Parent>
> <\FS>
> <\Children>
> Id = 21 nullId = 33
> Id =33
> null
> <\Children>
> <\Parent>
> <\HASHTABLEDUMMY><\Parent>
> <\MAPJOIN>
> <\Children>
> Id = 0 null<\Parent>
> <\TS>
> <\Children>
> <\Parent>
> <\MAP>
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[21]
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Initializing dummy operator
> 2016-08-17 21:13:42,086 INFO [main]
> org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0,
> RECORDS_IN:0,
> 2016-08-17 21:13:42,087 ERROR [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing
> operators - failing tree
> 2016-08-17 21:13:42,088 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:682)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled
[
https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569848#comment-16569848
]
Hive QA commented on HIVE-14557:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 2301 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13056/dev-support/hive-personality.sh
|
| git revision | master / 20c95c1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13056/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Nullpointer When both SkewJoin and Mapjoin Enabled
> ---
>
> Key: HIVE-14557
> URL: https://issues.apache.org/jira/browse/HIVE-14557
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Nemon Lou
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-14557.2.patch, HIVE-14557.patch
>
>
> The following sql failed with return code 2 on mr.
> {noformat}
> create table a(id int,id1 int);
> create table b(id int,id1 int);
> create table c(id int,id1 int);
> set hive.optimize.skewjoin=true;
> select a.id,b.id,c.id1 from a,b,c where a.id=b.id and a.id1=c.id1;
> {noformat}
> Error log as follows:
> {noformat}
> 2016-08-17 21:13:42,081 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> Id =0
>
> Id =21
>
> Id =28
>
> Id =16
>
> <\Children>
> Id = 28 null<\Parent>
> <\FS>
> <\Children>
> Id = 21 nullId = 33
> Id =33
> null
> <\Children>
> <\Parent>
> <\HASHTABLEDUMMY><\Parent>
> <\MAPJOIN>
> <\Children>
> Id = 0 null<\Parent>
> <\TS>
> <\Children>
> <\Parent>
> <\MAP>
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[21]
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Initializing dummy operator
> 2016-08-17 21:13:42,086 INFO [main]
> org.apache.hadoop.hive.ql.exec.Ma
[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled
[
https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569870#comment-16569870
]
Hive QA commented on HIVE-14557:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934433/HIVE-14557.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14863 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13056/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13056/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13056/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934433 - PreCommit-HIVE-Build
> Nullpointer When both SkewJoin and Mapjoin Enabled
> ---
>
> Key: HIVE-14557
> URL: https://issues.apache.org/jira/browse/HIVE-14557
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Nemon Lou
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-14557.2.patch, HIVE-14557.patch
>
>
> The following sql failed with return code 2 on mr.
> {noformat}
> create table a(id int,id1 int);
> create table b(id int,id1 int);
> create table c(id int,id1 int);
> set hive.optimize.skewjoin=true;
> select a.id,b.id,c.id1 from a,b,c where a.id=b.id and a.id1=c.id1;
> {noformat}
> Error log as follows:
> {noformat}
> 2016-08-17 21:13:42,081 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> Id =0
>
> Id =21
>
> Id =28
>
> Id =16
>
> <\Children>
> Id = 28 null<\Parent>
> <\FS>
> <\Children>
> Id = 21 nullId = 33
> Id =33
> null
> <\Children>
> <\Parent>
> <\HASHTABLEDUMMY><\Parent>
> <\MAPJOIN>
> <\Children>
> Id = 0 null<\Parent>
> <\TS>
> <\Children>
> <\Parent>
> <\MAP>
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[21]
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Initializing dummy operator
> 2016-08-17 21:13:42,086 INFO [main]
> org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0,
> RECORDS_IN:0,
> 2016-08-17 21:13:42,087 ERROR [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing
> operators - failing tree
> 2016-08-17 21:13:42,088 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:682)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569874#comment-16569874 ] Sahil Takiar commented on HIVE-14162: - [~ngangam], [~aihuaxu] could you take a look at this patch. I created an RB which a detailed description of the code changes - https://reviews.apache.org/r/68223/ > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Thomas Scott >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-14162.1.patch, HIVE-14162.2.patch, > HIVE-14162.3.patch, HIVE-14162.4.patch, HIVE-14162.5.patch, > HIVE-14162.6.patch, HIVE-14162.7.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18858) System properties in job configuration not resolved when submitting MR job
[
https://issues.apache.org/jira/browse/HIVE-18858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569875#comment-16569875
]
Puneet Jain commented on HIVE-18858:
Hi,
This seems to have broken working scenarios with Hive MR. We now see
hadoop.tmp.dir is always set to /tmp/hadoop-hive (in job.xml). This creates
problems on a multi-tenant hadoop cluster since ownership of tmp folder is set
to the user who executes the jobs first and other users fails to write to tmp
folder.
E.g. User1 run job and /tmp/hadoop-hive is created on worker node with
ownership to user1 and sibsequently user2 tries to run a job and job fails due
to no write permission on /tmp/hadoop-hive/
Old behavior allowed multiple tenants to write to their respective tmp folders
which was secure and contention free. User1 - /tmp/hadoop-user1, User2 -
/tmp/hadoop-user2.
Thanks
Puneet
> System properties in job configuration not resolved when submitting MR job
> --
>
> Key: HIVE-18858
> URL: https://issues.apache.org/jira/browse/HIVE-18858
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
> Environment: Hadoop 3.0.0
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18858.1.patch, HIVE-18858.2.patch,
> HIVE-18858.3.patch
>
>
> Since [this hadoop
> commit|https://github.com/apache/hadoop/commit/5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1]
> that was first released in 3.0.0, Configuration has a restricted mode, that
> disables the resolution of system properties (that happens when retrieving a
> configuration option).
> This leads to test failures when switching to Hadoop 3.0.0 (instead of
> 3.0.0-beta1), since we're relying on the [substitution of
> test.tmp.dir|https://github.com/apache/hive/blob/05d4719eefc56676a3e0e8f706e1c5e5e1f6b345/data/conf/hive-site.xml#L37]
> during the [maven
> build|https://github.com/apache/hive/blob/05d4719eefc56676a3e0e8f706e1c5e5e1f6b345/pom.xml#L83].
> See test results on HIVE-18327.
> When we're passing job configurations to Hadoop, I believe there's no way to
> disable the restricted mode, since we go through some Hadoop MR calls first,
> see here:
> {code}
> "HiveServer2-Background-Pool: Thread-105@9500" prio=5 tid=0x69 nid=NA runnable
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.conf.Configuration.addResourceObject(Configuration.java:970)
> - locked <0x2fe6> (a org.apache.hadoop.mapred.JobConf)
> at
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:895)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:476)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:162)
> at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:788)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:254)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
> at
> java.security.AccessController.doPrivileged(AccessController.java:-1)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571)
> at
> java.security.AccessController.doPrivileged(AccessController.java:-1)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:415)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:149)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2314)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1985)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1687)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1438)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1432)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:248)
> at
[jira] [Updated] (HIVE-19937) Intern fields in MapWork on deserialization
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19937:
Attachment: HIVE-19937.6.patch
> Intern fields in MapWork on deserialization
> ---
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch, HIVE-19937.2.patch,
> HIVE-19937.3.patch, HIVE-19937.4.patch, HIVE-19937.5.patch,
> HIVE-19937.6.patch, post-patch-report.html, report.html
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19937) Intern fields in MapWork on deserialization
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19937:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Thanks for taking a look Vihang. I addressed your comments and attached an
updated patch. Since the change was just to add new comments, I don't think its
necessary to re-run Hive QA, so I went ahead and pushed this to master.
> Intern fields in MapWork on deserialization
> ---
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19937.1.patch, HIVE-19937.2.patch,
> HIVE-19937.3.patch, HIVE-19937.4.patch, HIVE-19937.5.patch,
> HIVE-19937.6.patch, post-patch-report.html, report.html
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled
[
https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569935#comment-16569935
]
Rui Li commented on HIVE-14557:
---
[~aihuaxu], could you please verify whether the patch can solve your issue?
> Nullpointer When both SkewJoin and Mapjoin Enabled
> ---
>
> Key: HIVE-14557
> URL: https://issues.apache.org/jira/browse/HIVE-14557
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Nemon Lou
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-14557.2.patch, HIVE-14557.patch
>
>
> The following sql failed with return code 2 on mr.
> {noformat}
> create table a(id int,id1 int);
> create table b(id int,id1 int);
> create table c(id int,id1 int);
> set hive.optimize.skewjoin=true;
> select a.id,b.id,c.id1 from a,b,c where a.id=b.id and a.id1=c.id1;
> {noformat}
> Error log as follows:
> {noformat}
> 2016-08-17 21:13:42,081 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> Id =0
>
> Id =21
>
> Id =28
>
> Id =16
>
> <\Children>
> Id = 28 null<\Parent>
> <\FS>
> <\Children>
> Id = 21 nullId = 33
> Id =33
> null
> <\Children>
> <\Parent>
> <\HASHTABLEDUMMY><\Parent>
> <\MAPJOIN>
> <\Children>
> Id = 0 null<\Parent>
> <\TS>
> <\Children>
> <\Parent>
> <\MAP>
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[21]
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Initializing dummy operator
> 2016-08-17 21:13:42,086 INFO [main]
> org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0,
> RECORDS_IN:0,
> 2016-08-17 21:13:42,087 ERROR [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing
> operators - failing tree
> 2016-08-17 21:13:42,088 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:682)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19008) Improve Spark session id logging
[ https://issues.apache.org/jira/browse/HIVE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-19008: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master, thanks Aihua for the review! > Improve Spark session id logging > > > Key: HIVE-19008 > URL: https://issues.apache.org/jira/browse/HIVE-19008 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19008.1.patch, HIVE-19008.2.patch > > > HoS users have two session ids, one id for the Hive session and another id > for the Spark session, both are UUIDs. > I think some improvements could be made here: > The Spark session id could just be a counter that is incremented for each new > Spark session within a Hive session. Each Spark session is still globally > identifiable by its associated Hive session id + its own counter. This may > make more sense since the Hive session - Spark session has a 1-to-many > relationship, as in a single Hive session can contain multiple Spark > sessions, and each Spark session must belong to a Hive session. > Furthermore, we should include both the Hive session id and Spark session id > in the console logs + the Spark Web UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Status: Patch Available (was: Open) > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, replication > Attachments: HIVE-20316.01.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Attachment: HIVE-20316.01.patch > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, replication > Attachments: HIVE-20316.01.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569961#comment-16569961 ] Sankar Hariappan commented on HIVE-20316: - [~maheshk114], Could you please take a look at the patch? > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, replication > Attachments: HIVE-20316.01.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570007#comment-16570007 ] ASF GitHub Bot commented on HIVE-20316: --- GitHub user sankarh opened a pull request: https://github.com/apache/hive/pull/408 HIVE-20316: Skip external table file listing for create table event. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sankarh/hive HIVE-20316 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/408.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #408 commit 0c89ab0d5811beeb58800b9f8f66f6f2d6119116 Author: Sankar Hariappan Date: 2018-08-06T09:42:10Z HIVE-20316: Skip external table file listing for create table event. > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-20316: -- Labels: DR pull-request-available replication (was: DR replication) > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20316) Skip external table file listing for create table event.
[
https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570013#comment-16570013
]
Hive QA commented on HIVE-20316:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13057/dev-support/hive-personality.sh
|
| git revision | master / 7795c0a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13057/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13057/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: hcatalog/server-extensions itests/hive-unit
standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13057/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Skip external table file listing for create table event.
>
>
> Key: HIVE-20316
> URL: https://issues.apache.org/jira/browse/HIVE-20316
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, repl
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-20316.01.patch
>
>
> We are currently skipping external table replication. We shall also skip
> listing all the files in create table event generation for external tables
[jira] [Work started] (HIVE-20264) Bootstrap repl dump with concurrent write and drop of ACID table makes target inconsistent.
[ https://issues.apache.org/jira/browse/HIVE-20264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20264 started by Sankar Hariappan. --- > Bootstrap repl dump with concurrent write and drop of ACID table makes target > inconsistent. > --- > > Key: HIVE-20264 > URL: https://issues.apache.org/jira/browse/HIVE-20264 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 4.0.0, 3.2.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, replication > > During bootstrap dump of ACID tables, let's consider the below sequence. > - Get lastReplId = last event ID logged. > - Current session (Thread-1), REPL DUMP -> Open txn (Txn1) - Event-10 > - Another session (Thread-2), Open txn (Txn2) - Event-11 > - Thread-2 -> Insert data (T1.D1) to ACID table. - Event-12 > - Thread-2 -> Commit Txn (Txn2) - Event-13 > - Thread-2 -> Drop table (T1) - Event-14 > - Thread-1 -> Dump ACID tables based on validTxnList based on Txn1. --> This > step skips all the data written by txns > Txn1. So, T1 will be missing. > - Thread-1 -> Commit Txn (Txn1) > - REPL LOAD from bootstrap dump will skip T1. > - Incremental REPL DUMP will start from Event-10 and hence allocate write id > for table T1 and drop table(T1) is idempotent. So, at target, exist entries > in TXN_TO_WRITE_ID and NEXT_WRITE_ID metastore tables. > - Now, when we create another table at source with same name T1 and > replicate, then it may lead to incorrect data for readers at target on T1. > Couple of proposals: > 1. Make allocate write ID idempotent which is not possible as table doesn't > exist and MM table import may lead to allocate write id before creating > table. So, cannot differentiate these 2 cases. > 2. Make Drop table event to drop entries from TXN_TO_WRITE_ID and > NEXT_WRITE_ID tables irrespective of table exist or not at target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20316) Skip external table file listing for create table event.
[
https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570053#comment-16570053
]
Hive QA commented on HIVE-20316:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934453/HIVE-20316.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 14864 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.listener.TestDbNotificationListener.addPartition
(batchId=257)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterPartition
(batchId=257)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterTable
(batchId=257)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropPartition
(batchId=257)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable
(batchId=257)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.exchangePartition
(batchId=257)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.insertPartition
(batchId=257)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.insertTable
(batchId=257)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13057/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13057/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13057/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934453 - PreCommit-HIVE-Build
> Skip external table file listing for create table event.
>
>
> Key: HIVE-20316
> URL: https://issues.apache.org/jira/browse/HIVE-20316
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, repl
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-20316.01.patch
>
>
> We are currently skipping external table replication. We shall also skip
> listing all the files in create table event generation for external tables.
> External tables might have very large number of files, so it would take long
> time to list them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-11625) Map instances with null keys are not properly handled for Parquet tables
[
https://issues.apache.org/jira/browse/HIVE-11625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570112#comment-16570112
]
pranav agrawal commented on HIVE-11625:
---
I am reading data from avro file, processing it and writing to parquet and
getting this issue. My types are Array and Array. Please suggest a
workaround.
> Map instances with null keys are not properly handled for Parquet tables
>
>
> Key: HIVE-11625
> URL: https://issues.apache.org/jira/browse/HIVE-11625
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 0.14.0, 0.13.1, 1.0.1, 1.1.1, 1.2.1
>Reporter: Cheng Lian
>Priority: Major
>
> Hive allows maps with null keys:
> {code:sql}
> hive> select map(null, 'foo', 1, 'bar', null, 'baz');
> {null:"baz",1:"bar"}
> {code}
> However, when written into Parquet tables, map entries with null as keys are
> either dropped or cause exceptions. Below is the result of Hive 0.14.0 and
> 0.13.1:
> {code:sql}
> hive> CREATE TABLE map_test STORED AS PARQUET
> > AS SELECT MAP(null, 'foo', 1, 'bar', null, 'baz');
> ...
> hive> SELECT * from map_test;
> {1:"bar"}
> {code}
> And Hive 1.2.1 throws exception:
> {noformat}
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing writable (null)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing writable (null)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:516)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ... 8 more
> Caused by: java.lang.RuntimeException: Parquet record is malformed: empty
> fields are illegal, the field should be ommited completely instead
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:64)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:121)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:753)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)
> ... 9 more
> Caused by: parquet.io.ParquetEncodingException: empty fields are illegal, the
> field should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:244)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeMap(DataWritableWriter.java:228)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeValue(DataWritableWriter.java:116)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeGroupFields(DataWritableWriter.java:89)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:60)
> ... 23 more
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing writable (null)
> a
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Status: Open (was: Patch Available) > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Attachment: HIVE-20316.02.patch > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Status: Patch Available (was: Open) Attached 02.patch to fix test failures. > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20316) Skip external table file listing for create table event.
[
https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570189#comment-16570189
]
Hive QA commented on HIVE-20316:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13058/dev-support/hive-personality.sh
|
| git revision | master / 7795c0a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13058/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13058/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: hcatalog/server-extensions itests/hcatalog-unit itests/hive-unit
standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13058/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Skip external table file listing for create table event.
>
>
> Key: HIVE-20316
> URL: https://issues.apache.org/jira/browse/HIVE-20316
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, repl
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch
>
>
> We are currently skipping external table replication. We shall also skip
> listing all the files in create
[jira] [Commented] (HIVE-20316) Skip external table file listing for create table event.
[
https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570234#comment-16570234
]
Hive QA commented on HIVE-20316:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934472/HIVE-20316.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14859 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=322)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13058/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13058/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13058/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934472 - PreCommit-HIVE-Build
> Skip external table file listing for create table event.
>
>
> Key: HIVE-20316
> URL: https://issues.apache.org/jira/browse/HIVE-20316
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, repl
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch
>
>
> We are currently skipping external table replication. We shall also skip
> listing all the files in create table event generation for external tables.
> External tables might have very large number of files, so it would take long
> time to list them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Attachment: HIVE-20316.02.patch > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Status: Open (was: Patch Available) > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Attachment: (was: HIVE-20316.02.patch) > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Status: Patch Available (was: Open) Failed tests seems to be flaky and so re-attaching same patch to re-run the ptest. > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[ https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20315: Status: In Progress (was: Patch Available) > Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary > casts/conversions > --- > > Key: HIVE-20315 > URL: https://issues.apache.org/jira/browse/HIVE-20315 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, > HIVE-20315.03.patch, HIVE-20315.04.patch > > > Generate multi-byte Unicode characters in addition to regular single byte > characters for random data. > Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in > vectorization without padding. > Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL > constants. > Fix NULL / Wrong Results issues in VectorElt. > Change performance Q files to generate non-user EXPLAIN with VECTORIZATION > display so unnecesary CAST / DECIMAL_64 conversions are visible. > Add test for vectorized STRUCT field reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[ https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20315: Status: Patch Available (was: In Progress) > Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary > casts/conversions > --- > > Key: HIVE-20315 > URL: https://issues.apache.org/jira/browse/HIVE-20315 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, > HIVE-20315.03.patch, HIVE-20315.04.patch > > > Generate multi-byte Unicode characters in addition to regular single byte > characters for random data. > Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in > vectorization without padding. > Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL > constants. > Fix NULL / Wrong Results issues in VectorElt. > Change performance Q files to generate non-user EXPLAIN with VECTORIZATION > display so unnecesary CAST / DECIMAL_64 conversions are visible. > Add test for vectorized STRUCT field reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[ https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20315: Attachment: HIVE-20315.05.patch > Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary > casts/conversions > --- > > Key: HIVE-20315 > URL: https://issues.apache.org/jira/browse/HIVE-20315 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, > HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch > > > Generate multi-byte Unicode characters in addition to regular single byte > characters for random data. > Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in > vectorization without padding. > Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL > constants. > Fix NULL / Wrong Results issues in VectorElt. > Change performance Q files to generate non-user EXPLAIN with VECTORIZATION > display so unnecesary CAST / DECIMAL_64 conversions are visible. > Add test for vectorized STRUCT field reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20316) Skip external table file listing for create table event.
[
https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570332#comment-16570332
]
Hive QA commented on HIVE-20316:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13059/dev-support/hive-personality.sh
|
| git revision | master / 7795c0a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13059/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13059/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: hcatalog/server-extensions itests/hcatalog-unit itests/hive-unit
standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13059/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Skip external table file listing for create table event.
>
>
> Key: HIVE-20316
> URL: https://issues.apache.org/jira/browse/HIVE-20316
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, repl
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch
>
>
> We are currently skipping external table replication. We shall also skip
> listing all the files in create
[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view
[
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Yan updated HIVE-20318:
Description:
The following sql throws NullPointerException.
This sql is not table/data specific and can be run directly.
{code:sql}
WITH t1 AS ( SELECT 0 AS c1 ),
t2 AS ( SELECT 0 AS c1 FROM ( SELECT COLLECT_SET('line') AS c2
) t3 lateral VIEW explode(ARRAY("a")) er AS c3
)SELECT c1FROM t1UNION ALLSELECT c1FROM t2
{code}
This is the exception.
{code}
2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException: Hive Runtime Error while
closing operators
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
... 8 more
{code}
was:
The following sql throws NullPointerException.
This sql is not table/data specific and can be run directly.
{code:sql}
WITH t1 AS ( SELECT 0 AS c1 ),
t2 AS ( SELECT 0 AS c1 FROM ( SELECT COLLECT_SET('line') AS c2
) t3 lateral VIEW explode(ARRAY("a")) er AS c3
)SELECT c1FROM t1UNION ALLSELECT c1FROM t2
{code}
This is the exception.
2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException: Hive Runtime Error while
closing operators
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
... 8 more
> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.3.2
> Environment: Run on MR, hadoop 2.7.3
>Reporter: Wang Yan
>Priority: Blocker
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH t1 AS ( SELECT 0 AS c1 ),
> t2 AS ( SELECT 0 AS c1 FROM ( SELECT COLLECT_SET('line') AS c2
> ) t3 lateral VIEW explode(ARRAY("a")) er AS c3
> )SELECT c1FROM t1UNION ALLSELECT c1FROM t2
> {code}
> This is the exception.
> {code}
> 2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> a
[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view
[
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Yan updated HIVE-20318:
Description:
The following sql throws NullPointerException.
This sql is not table/data specific and can be run directly.
{code:sql}
WITH t1 AS ( SELECT 0 AS c1 ),
t2 AS ( SELECT 0 AS c1 FROM ( SELECT COLLECT_SET('line') AS c2
) t3 lateral VIEW explode(ARRAY("a")) er AS c3
)SELECT c1FROM t1UNION ALLSELECT c1FROM t2
{code}
This is the exception.
2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException: Hive Runtime Error while
closing operators
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
... 8 more
was:
The following sql throws NullPointerException.
This sql is not table/data specific and can be run directly.
WITH t1 AS ( SELECT 0 AS c1 ),
t2 AS ( SELECT 0 AS c1 FROM ( SELECT COLLECT_SET('line') AS c2
) t3 lateral VIEW explode(ARRAY("a")) er AS c3
)SELECT c1FROM t1UNION ALLSELECT c1FROM t2
This is the exception.
2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException: Hive Runtime Error while
closing operators
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
... 8 more
> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.3.2
> Environment: Run on MR, hadoop 2.7.3
>Reporter: Wang Yan
>Priority: Blocker
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH t1 AS ( SELECT 0 AS c1 ),
> t2 AS ( SELECT 0 AS c1 FROM ( SELECT COLLECT_SET('line') AS c2
> ) t3 lateral VIEW explode(ARRAY("a")) er AS c3
> )SELECT c1FROM t1UNION ALLSELECT c1FROM t2
> {code}
> This is the exception.
> 2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
>
[jira] [Updated] (HIVE-20318) NullPointerException when union with lateral view
[
https://issues.apache.org/jira/browse/HIVE-20318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Yan updated HIVE-20318:
Description:
The following sql throws NullPointerException.
This sql is not table/data specific and can be run directly.
{code:sql}
WITH
t1 AS (SELECT 0 AS c1),
t2 AS (
SELECT 0 AS c1
FROM (
SELECT COLLECT_SET('line') AS c2
) t3 lateral VIEW explode(ARRAY("a")) er AS c3
)
SELECT c1
FROM t1
UNION ALL
SELECT c1
FROM t2
{code}
This is the exception.
{code:java}
2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException: Hive Runtime Error while
closing operators
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
... 8 more
{code}
was:
The following sql throws NullPointerException.
This sql is not table/data specific and can be run directly.
{code:sql}
WITH t1 AS ( SELECT 0 AS c1 ),
t2 AS ( SELECT 0 AS c1 FROM ( SELECT COLLECT_SET('line') AS c2
) t3 lateral VIEW explode(ARRAY("a")) er AS c3
)SELECT c1FROM t1UNION ALLSELECT c1FROM t2
{code}
This is the exception.
{code}
2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException: Hive Runtime Error while
closing operators
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:3901)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1020)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
... 8 more
{code}
> NullPointerException when union with lateral view
> -
>
> Key: HIVE-20318
> URL: https://issues.apache.org/jira/browse/HIVE-20318
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.3.2
> Environment: Run on MR, hadoop 2.7.3
>Reporter: Wang Yan
>Priority: Blocker
>
> The following sql throws NullPointerException.
> This sql is not table/data specific and can be run directly.
> {code:sql}
> WITH
> t1 AS (SELECT 0 AS c1),
> t2 AS (
> SELECT 0 AS c1
> FROM (
> SELECT COLLECT_SET('line') AS c2
>) t3 lateral VIEW explode(ARRAY("a")) er AS c3
> )
> SELECT c1
> FROM t1
> UNION ALL
> SELECT c1
> FROM t2
> {code}
> This is the exception.
> {code:java}
> 2018-04-20 01:53:50,845 WARN [Thread-5] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>
[jira] [Commented] (HIVE-20292) Bad join ordering in tpcds query93 with primary constraint defined
[ https://issues.apache.org/jira/browse/HIVE-20292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570360#comment-16570360 ] Ashutosh Chauhan commented on HIVE-20292: - +1 Lets also revisit sub_count_check in a follow-up. > Bad join ordering in tpcds query93 with primary constraint defined > -- > > Key: HIVE-20292 > URL: https://issues.apache.org/jira/browse/HIVE-20292 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20292.1.patch, HIVE-20292.2.patch, > HIVE-20292.3.patch, HIVE-20292.4.patch, HIVE-20292.5.patch, > HIVE-20292.6.patch, HIVE-20292.7.patch > > > Query 93 has join (including outer) b/w store_sales, store_return and reason. > Without constraints store_return is joined with reason and then with > store_sales. > But if a primary key is added on store_return (alter table store_returns add > constraint tpcds_pk_sr primary key (sr_item_sk, sr_ticket_number) disable > novalidate rely) join order becomes ((store_sales, store_return), reason) > which is very inefficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20316) Skip external table file listing for create table event.
[
https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570385#comment-16570385
]
Hive QA commented on HIVE-20316:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934486/HIVE-20316.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14864 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.common.metrics.metrics2.TestCodahaleMetrics.testFileReporting
(batchId=273)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13059/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13059/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13059/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934486 - PreCommit-HIVE-Build
> Skip external table file listing for create table event.
>
>
> Key: HIVE-20316
> URL: https://issues.apache.org/jira/browse/HIVE-20316
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, repl
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch
>
>
> We are currently skipping external table replication. We shall also skip
> listing all the files in create table event generation for external tables.
> External tables might have very large number of files, so it would take long
> time to list them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570387#comment-16570387
]
Hive QA commented on HIVE-20315:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934490/HIVE-20315.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13060/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13060/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13060/
Messages:
{noformat}
This message was trimmed, see log for full details
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query71.q.out:97
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query71.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query73.q.out:68
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query73.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query74.q.out:148
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query74.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query79.q.out:108
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query79.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query82.q.out:75
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query82.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query83.q.out:166
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query83.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query85.q.out:164
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query85.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query89.q.out:99
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query89.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query91.q.out:132
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query91.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query98.q.out:138
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query98.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query10.q.out:114
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query10.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query12.q.out:62
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query12.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query13.q.out:98
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query13.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query15.q.out:36
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query15.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query16.q.out:58
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query16.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query17.q.out:86
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query17.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query18.q.out:64
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query18.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query20.q.out:54
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query20.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query23.q.out:102
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query23.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query27.q.out:42
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientposi
[jira] [Assigned] (HIVE-18620) Update materialized views when dropping its source tables
[ https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-18620: -- Assignee: Miklos Gergely (was: Jesus Camacho Rodriguez) > Update materialized views when dropping its source tables > - > > Key: HIVE-18620 > URL: https://issues.apache.org/jira/browse/HIVE-18620 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Critical > > When we want to drop a table used by a materialized view, we should prevent > dropping that table unless explicitly specified that materialized views based > on that table can be dropped too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20319) group by and union all always generate empty query result
[
https://issues.apache.org/jira/browse/HIVE-20319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Yan updated HIVE-20319:
Description:
The following query always generates empty results which is wrong.
{code:sql}
create table if not exists test_table(column1 string, column2 int);
insert into test_table values('a',1),('b',2);
set hive.optimize.union.remove=true;
select column1 from test_table group by column1
union all
select column1 from test_table group by column1;
{code}
Actual result : empty
Expected result:
{code:java}
a
b
a
b
{code}
Note that correct result is generated when set hive.optimize.union.remove=false.
It seems like the fix in https://issues.apache.org/jira/browse/HIVE-12788 is
insufficient.
was:
The following query always generates empty results which is wrong.
{code:sql}
create table if not exists test_table(column1 string, column2 int);
insert into test_table values('a',1),('b',2);
set hive.optimize.union.remove=true;
select column1 from test_table group by column1
union all
select column1 from test_table group by column1;
{code}
Actual result : empty
Expected result:
{code}
a
b
a
b
{code}
Note that when correct result is generated when set
hive.optimize.union.remove=false.
It seems like the fix in https://issues.apache.org/jira/browse/HIVE-12788 is
insufficient.
> group by and union all always generate empty query result
> -
>
> Key: HIVE-20319
> URL: https://issues.apache.org/jira/browse/HIVE-20319
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.3.2
>Reporter: Wang Yan
>Priority: Blocker
>
> The following query always generates empty results which is wrong.
> {code:sql}
> create table if not exists test_table(column1 string, column2 int);
> insert into test_table values('a',1),('b',2);
> set hive.optimize.union.remove=true;
> select column1 from test_table group by column1
> union all
> select column1 from test_table group by column1;
> {code}
> Actual result : empty
> Expected result:
> {code:java}
> a
> b
> a
> b
> {code}
> Note that correct result is generated when set
> hive.optimize.union.remove=false.
> It seems like the fix in https://issues.apache.org/jira/browse/HIVE-12788 is
> insufficient.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20319) group by and union all always generate empty query result
[
https://issues.apache.org/jira/browse/HIVE-20319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Yan updated HIVE-20319:
Description:
The following query always generates empty results which is wrong.
{code:sql}
create table if not exists test_table(column1 string, column2 int);
insert into test_table values('a',1),('b',2);
set hive.optimize.union.remove=true;
select column1 from test_table group by column1
union all
select column1 from test_table group by column1;
{code}
Actual result : empty
Expected result:
{code:java}
a
b
a
b
{code}
Note that correct result is generated when set hive.optimize.union.remove=false.
It seems like the fix in HIVE-12788 is insufficient.
was:
The following query always generates empty results which is wrong.
{code:sql}
create table if not exists test_table(column1 string, column2 int);
insert into test_table values('a',1),('b',2);
set hive.optimize.union.remove=true;
select column1 from test_table group by column1
union all
select column1 from test_table group by column1;
{code}
Actual result : empty
Expected result:
{code:java}
a
b
a
b
{code}
Note that correct result is generated when set hive.optimize.union.remove=false.
It seems like the fix in https://issues.apache.org/jira/browse/HIVE-12788 is
insufficient.
> group by and union all always generate empty query result
> -
>
> Key: HIVE-20319
> URL: https://issues.apache.org/jira/browse/HIVE-20319
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.3.2
>Reporter: Wang Yan
>Priority: Blocker
>
> The following query always generates empty results which is wrong.
> {code:sql}
> create table if not exists test_table(column1 string, column2 int);
> insert into test_table values('a',1),('b',2);
> set hive.optimize.union.remove=true;
> select column1 from test_table group by column1
> union all
> select column1 from test_table group by column1;
> {code}
> Actual result : empty
> Expected result:
> {code:java}
> a
> b
> a
> b
> {code}
> Note that correct result is generated when set
> hive.optimize.union.remove=false.
> It seems like the fix in HIVE-12788 is insufficient.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16480) ORC file with empty array and array fails to read
[
https://issues.apache.org/jira/browse/HIVE-16480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570408#comment-16570408
]
Elliot West commented on HIVE-16480:
Is there a work-around for this issue? We've tried the turning off
vectorization in Hive 2.3.2 but it still seems to take the vectorized path,
hitting this bug.
> ORC file with empty array and array fails to read
>
>
> Key: HIVE-16480
> URL: https://issues.apache.org/jira/browse/HIVE-16480
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0
>Reporter: David Capwell
>Assignee: Owen O'Malley
>Priority: Major
> Labels: pull-request-available
>
> We have a schema that has a array in it. We were unable to read this
> file and digging into ORC it seems that the issue is when the array is empty.
> Here is the stack trace
> {code:title=EmptyList.log|borderStyle=solid}
> ERROR 2017-04-19 09:29:17,075 [main] [EmptyList] [line 56] Failed to work
> with type float
> java.io.IOException: Error reading file:
> /var/folders/t8/t5x1031d7mn17f6xpwnkkv_4gn/T/1492619355819-0/file-float.orc
> at
> org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1052)
> ~[hive-orc-2.1.1.jar:2.1.1]
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.nextBatch(RecordReaderImpl.java:135)
> ~[hive-exec-2.1.1.jar:2.1.1]
> at EmptyList.emptyList(EmptyList.java:49) ~[test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_121]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_121]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> [junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> [junit-4.12.jar:4.12]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> [junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> [junit-4.12.jar:4.12]
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> [junit-4.12.jar:4.12]
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> [junit-4.12.jar:4.12]
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137) [junit-4.12.jar:4.12]
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> [junit-rt.jar:na]
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> [junit-rt.jar:na]
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
> [junit-rt.jar:na]
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> [junit-rt.jar:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_121]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_121]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> [idea_rt.jar:na]
> Caused by: java.io.EOFException: Read past EOF for compressed stream Stream
> for column 1 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0
> at
> org.apache.orc.impl.SerializationUtils.readFully(SerializationUtils.java:118)
> ~[hive-orc-2.1.1.jar:2.1.1]
> at
> org.apache.orc.impl.SerializationUtils.readFloat(SerializationUtils.java:78)
> ~[hive-orc-2.1.1.jar:2.1.1]
> at
> org.apache.orc.impl.TreeReaderFactory$FloatTreeReader.nextVector(TreeReaderFactory.java:619)
> ~[hive-orc-2.1.1.jar:2.1.1]
> at
> org.apache.orc.impl.TreeReaderFactory$L
[jira] [Updated] (HIVE-20319) group by and union all always generate empty query result
[
https://issues.apache.org/jira/browse/HIVE-20319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Yan updated HIVE-20319:
Environment: Run on MR, hadoop 2.7.3
> group by and union all always generate empty query result
> -
>
> Key: HIVE-20319
> URL: https://issues.apache.org/jira/browse/HIVE-20319
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.3.2
> Environment: Run on MR, hadoop 2.7.3
>Reporter: Wang Yan
>Priority: Blocker
>
> The following query always generates empty results which is wrong.
> {code:sql}
> create table if not exists test_table(column1 string, column2 int);
> insert into test_table values('a',1),('b',2);
> set hive.optimize.union.remove=true;
> select column1 from test_table group by column1
> union all
> select column1 from test_table group by column1;
> {code}
> Actual result : empty
> Expected result:
> {code:java}
> a
> b
> a
> b
> {code}
> Note that correct result is generated when set
> hive.optimize.union.remove=false.
> It seems like the fix in HIVE-12788 is insufficient.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Status: Open (was: Patch Available) > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Attachment: (was: HIVE-20316.02.patch) > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Status: Patch Available (was: Open) Failed test is flaky. Attaching same patch again for re-run. > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20316) Skip external table file listing for create table event.
[ https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20316: Attachment: HIVE-20316.02.patch > Skip external table file listing for create table event. > > > Key: HIVE-20316 > URL: https://issues.apache.org/jira/browse/HIVE-20316 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch > > > We are currently skipping external table replication. We shall also skip > listing all the files in create table event generation for external tables. > External tables might have very large number of files, so it would take long > time to list them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[ https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20315: Status: In Progress (was: Patch Available) > Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary > casts/conversions > --- > > Key: HIVE-20315 > URL: https://issues.apache.org/jira/browse/HIVE-20315 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, > HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch > > > Generate multi-byte Unicode characters in addition to regular single byte > characters for random data. > Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in > vectorization without padding. > Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL > constants. > Fix NULL / Wrong Results issues in VectorElt. > Change performance Q files to generate non-user EXPLAIN with VECTORIZATION > display so unnecesary CAST / DECIMAL_64 conversions are visible. > Add test for vectorized STRUCT field reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[ https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20315: Status: Patch Available (was: In Progress) > Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary > casts/conversions > --- > > Key: HIVE-20315 > URL: https://issues.apache.org/jira/browse/HIVE-20315 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, > HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch > > > Generate multi-byte Unicode characters in addition to regular single byte > characters for random data. > Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in > vectorization without padding. > Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL > constants. > Fix NULL / Wrong Results issues in VectorElt. > Change performance Q files to generate non-user EXPLAIN with VECTORIZATION > display so unnecesary CAST / DECIMAL_64 conversions are visible. > Add test for vectorized STRUCT field reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[ https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20315: Attachment: HIVE-20315.06.patch > Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary > casts/conversions > --- > > Key: HIVE-20315 > URL: https://issues.apache.org/jira/browse/HIVE-20315 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, > HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, > HIVE-20315.06.patch > > > Generate multi-byte Unicode characters in addition to regular single byte > characters for random data. > Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in > vectorization without padding. > Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL > constants. > Fix NULL / Wrong Results issues in VectorElt. > Change performance Q files to generate non-user EXPLAIN with VECTORIZATION > display so unnecesary CAST / DECIMAL_64 conversions are visible. > Add test for vectorized STRUCT field reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20278) Druid Scan Query avoid copying from List -> Map -> List
[ https://issues.apache.org/jira/browse/HIVE-20278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20278: Attachment: HIVE-20278.patch > Druid Scan Query avoid copying from List -> Map -> List > --- > > Key: HIVE-20278 > URL: https://issues.apache.org/jira/browse/HIVE-20278 > Project: Hive > Issue Type: Improvement >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: PERFORMANCE > Attachments: HIVE-20278.patch > > > DruidScanQueryRecordReader gets a compacted List from druid. It then > converts that list into a Map as DruidWritable where key is > the column name. > At the second stage DruidSerde takes this DruidWritable and creates a List > out out of the map again. We can avoid the map creation part by reading the > list sent by druid directly in the DruidSerde.deserialize() method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20278) Druid Scan Query avoid copying from List -> Map -> List
[ https://issues.apache.org/jira/browse/HIVE-20278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20278: Status: Patch Available (was: Open) > Druid Scan Query avoid copying from List -> Map -> List > --- > > Key: HIVE-20278 > URL: https://issues.apache.org/jira/browse/HIVE-20278 > Project: Hive > Issue Type: Improvement >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: PERFORMANCE > Attachments: HIVE-20278.patch > > > DruidScanQueryRecordReader gets a compacted List from druid. It then > converts that list into a Map as DruidWritable where key is > the column name. > At the second stage DruidSerde takes this DruidWritable and creates a List > out out of the map again. We can avoid the map creation part by reading the > list sent by druid directly in the DruidSerde.deserialize() method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20278) Druid Scan Query avoid copying from List -> Map -> List
[ https://issues.apache.org/jira/browse/HIVE-20278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20278: Attachment: (was: HIVE-20278.patch) > Druid Scan Query avoid copying from List -> Map -> List > --- > > Key: HIVE-20278 > URL: https://issues.apache.org/jira/browse/HIVE-20278 > Project: Hive > Issue Type: Improvement >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: PERFORMANCE > Attachments: HIVE-20278.patch > > > DruidScanQueryRecordReader gets a compacted List from druid. It then > converts that list into a Map as DruidWritable where key is > the column name. > At the second stage DruidSerde takes this DruidWritable and creates a List > out out of the map again. We can avoid the map creation part by reading the > list sent by druid directly in the DruidSerde.deserialize() method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20278) Druid Scan Query avoid copying from List -> Map -> List
[ https://issues.apache.org/jira/browse/HIVE-20278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20278: Attachment: HIVE-20278.patch > Druid Scan Query avoid copying from List -> Map -> List > --- > > Key: HIVE-20278 > URL: https://issues.apache.org/jira/browse/HIVE-20278 > Project: Hive > Issue Type: Improvement >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: PERFORMANCE > Attachments: HIVE-20278.patch > > > DruidScanQueryRecordReader gets a compacted List from druid. It then > converts that list into a Map as DruidWritable where key is > the column name. > At the second stage DruidSerde takes this DruidWritable and creates a List > out out of the map again. We can avoid the map creation part by reading the > list sent by druid directly in the DruidSerde.deserialize() method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20278) Druid Scan Query avoid copying from List -> Map -> List
[ https://issues.apache.org/jira/browse/HIVE-20278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570500#comment-16570500 ] Nishant Bangarwa commented on HIVE-20278: - +cc [~ashutoshc] Please review. https://reviews.apache.org/r/68240 > Druid Scan Query avoid copying from List -> Map -> List > --- > > Key: HIVE-20278 > URL: https://issues.apache.org/jira/browse/HIVE-20278 > Project: Hive > Issue Type: Improvement >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: PERFORMANCE > Attachments: HIVE-20278.patch > > > DruidScanQueryRecordReader gets a compacted List from druid. It then > converts that list into a Map as DruidWritable where key is > the column name. > At the second stage DruidSerde takes this DruidWritable and creates a List > out out of the map again. We can avoid the map creation part by reading the > list sent by druid directly in the DruidSerde.deserialize() method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20316) Skip external table file listing for create table event.
[
https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570504#comment-16570504
]
Hive QA commented on HIVE-20316:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13061/dev-support/hive-personality.sh
|
| git revision | master / 7795c0a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13061/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13061/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: hcatalog/server-extensions itests/hcatalog-unit itests/hive-unit
standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13061/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Skip external table file listing for create table event.
>
>
> Key: HIVE-20316
> URL: https://issues.apache.org/jira/browse/HIVE-20316
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, repl
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch
>
>
> We are currently skipping external table replication. We shall also skip
> listing all the files in create
[jira] [Assigned] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag
[ https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-20320: -- > Turn on hive.optimize.remove.sq_count_check flag > > > Key: HIVE-20320 > URL: https://issues.apache.org/jira/browse/HIVE-20320 > Project: Hive > Issue Type: Task > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20292) Bad join ordering in tpcds query93 with primary constraint defined
[ https://issues.apache.org/jira/browse/HIVE-20292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20292: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed this to master. Thanks for looking at it [~ashutoshc]. I have created HIVE-20320 to turn on sq_count_check optimization by default. > Bad join ordering in tpcds query93 with primary constraint defined > -- > > Key: HIVE-20292 > URL: https://issues.apache.org/jira/browse/HIVE-20292 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20292.1.patch, HIVE-20292.2.patch, > HIVE-20292.3.patch, HIVE-20292.4.patch, HIVE-20292.5.patch, > HIVE-20292.6.patch, HIVE-20292.7.patch > > > Query 93 has join (including outer) b/w store_sales, store_return and reason. > Without constraints store_return is joined with reason and then with > store_sales. > But if a primary key is added on store_return (alter table store_returns add > constraint tpcds_pk_sr primary key (sr_item_sk, sr_ticket_number) disable > novalidate rely) join order becomes ((store_sales, store_return), reason) > which is very inefficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20277) Vectorization: Case expressions that return BOOLEAN are not supported for FILTER
[
https://issues.apache.org/jira/browse/HIVE-20277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570526#comment-16570526
]
Gopal V commented on HIVE-20277:
LGTM - +1
> Vectorization: Case expressions that return BOOLEAN are not supported for
> FILTER
>
>
> Key: HIVE-20277
> URL: https://issues.apache.org/jira/browse/HIVE-20277
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Labels: Branch3Candidate
> Attachments: HIVE-20277.02.patch, HIVE-20277.03.patch,
> HIVE-20277.04.patch, HIVE-20277.WIP.01.patch
>
>
> In cases like Query89, the vertex with the filter is not vectorized.
> {code}
>Filter Operator
> predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN
> (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (null) END
> (type: boolean)
> {code}
> {code}
> Reducer 3
> Execution mode: llap
> Reduce Vectorization:
> enabled: true
> enableConditionsMet: hive.vectorized.execution.reduce.enabled
> IS true, hive.execution.engine tez IN [tez, spark] IS true
> notVectorizedReason: FILTER operator: Unexpected hive type
> name void
> vectorized: false
> {code}
> The query specifically has
> {code}
> where case when (avg_monthly_sales <> 0) then (abs(sum_sales -
> avg_monthly_sales) / avg_monthly_sales) else null end > 0.1
> {code}
> while rewriting it to
> {code}
> where case when (avg_monthly_sales <> 0) then (abs(sum_sales -
> avg_monthly_sales) / avg_monthly_sales) > 0.1 else false end
> {code}
> does vectorize into
> {code}
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression: SelectColumnIsTrue(col
> 12:boolean)(children: VectorUDFAdaptor(CASE WHEN ((avg_window_0 <> 0.0D))
> THEN (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false)
> END)(children: DoubleColNotEqualDoubleScalar(col 7:double, val 0.0) ->
> 8:boolean, DoubleColGreaterDoubleScalar(col 9:double, val 0.1)(children:
> DoubleColDivideDoubleColumn(col 10:double, col 7:double)(children:
> FuncAbsDoubleToDouble(col 9:double)(children:
> DoubleColSubtractDoubleColumn(col 6:double, col 7:double) -> 9:double) ->
> 10:double) -> 9:double) -> 11:boolean) -> 12:boolean)
> predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN
> (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false) END
> (type: boolean)
> Statistics: Num rows: 11 Data size: 5291 Basic stats:
> COMPLETE Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20277) Vectorization: Case expressions that return BOOLEAN are not supported for FILTER
[
https://issues.apache.org/jira/browse/HIVE-20277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-20277:
---
Labels: Branch3Candidate (was: )
> Vectorization: Case expressions that return BOOLEAN are not supported for
> FILTER
>
>
> Key: HIVE-20277
> URL: https://issues.apache.org/jira/browse/HIVE-20277
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Labels: Branch3Candidate
> Attachments: HIVE-20277.02.patch, HIVE-20277.03.patch,
> HIVE-20277.04.patch, HIVE-20277.WIP.01.patch
>
>
> In cases like Query89, the vertex with the filter is not vectorized.
> {code}
>Filter Operator
> predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN
> (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (null) END
> (type: boolean)
> {code}
> {code}
> Reducer 3
> Execution mode: llap
> Reduce Vectorization:
> enabled: true
> enableConditionsMet: hive.vectorized.execution.reduce.enabled
> IS true, hive.execution.engine tez IN [tez, spark] IS true
> notVectorizedReason: FILTER operator: Unexpected hive type
> name void
> vectorized: false
> {code}
> The query specifically has
> {code}
> where case when (avg_monthly_sales <> 0) then (abs(sum_sales -
> avg_monthly_sales) / avg_monthly_sales) else null end > 0.1
> {code}
> while rewriting it to
> {code}
> where case when (avg_monthly_sales <> 0) then (abs(sum_sales -
> avg_monthly_sales) / avg_monthly_sales) > 0.1 else false end
> {code}
> does vectorize into
> {code}
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression: SelectColumnIsTrue(col
> 12:boolean)(children: VectorUDFAdaptor(CASE WHEN ((avg_window_0 <> 0.0D))
> THEN (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false)
> END)(children: DoubleColNotEqualDoubleScalar(col 7:double, val 0.0) ->
> 8:boolean, DoubleColGreaterDoubleScalar(col 9:double, val 0.1)(children:
> DoubleColDivideDoubleColumn(col 10:double, col 7:double)(children:
> FuncAbsDoubleToDouble(col 9:double)(children:
> DoubleColSubtractDoubleColumn(col 6:double, col 7:double) -> 9:double) ->
> 10:double) -> 9:double) -> 11:boolean) -> 12:boolean)
> predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN
> (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false) END
> (type: boolean)
> Statistics: Num rows: 11 Data size: 5291 Basic stats:
> COMPLETE Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20316) Skip external table file listing for create table event.
[
https://issues.apache.org/jira/browse/HIVE-20316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570543#comment-16570543
]
Hive QA commented on HIVE-20316:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934505/HIVE-20316.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14864 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=322)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13061/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13061/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13061/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934505 - PreCommit-HIVE-Build
> Skip external table file listing for create table event.
>
>
> Key: HIVE-20316
> URL: https://issues.apache.org/jira/browse/HIVE-20316
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, repl
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-20316.01.patch, HIVE-20316.02.patch
>
>
> We are currently skipping external table replication. We shall also skip
> listing all the files in create table event generation for external tables.
> External tables might have very large number of files, so it would take long
> time to list them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570544#comment-16570544
]
Hive QA commented on HIVE-20315:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934506/HIVE-20315.06.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13062/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13062/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13062/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-08-06 17:49:23.340
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-13062/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-08-06 17:49:23.343
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
7795c0a..30c1791 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 7795c0a HIVE-19008: Improve Spark session id logging (Sahil
Takiar, reviewed by Aihua Xu)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 30c1791 HIVE-20292: Bad join ordering in tpcds query93 with
primary constraint defined (Vineet Garg, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-08-06 17:49:25.098
+ rm -rf ../yetus_PreCommit-HIVE-Build-13062
+ mkdir ../yetus_PreCommit-HIVE-Build-13062
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-13062
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-13062/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query2.q.out:140
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query2.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query59.q.out:419
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query59.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/perf/spark/query95.q.out:170
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/spark/query95.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query2.q.out:116
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query2.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query59.q.out:84
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query59.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query95.q.out:60
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query95.q.out'
with conflicts.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:2063: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:60822: trailing whitespace.
Map 1
/data/hiveptest/working/scratch/build.patch:60890: trailing whitespace.
Map 10
/data/hiveptest/working/scratch/build.patch:60933: trailing whitespace.
Map 8
/data/hiveptest/working/scratch/build.patch:60984: trailing whitespace.
Map 9
error: patch failed:
[jira] [Commented] (HIVE-20277) Vectorization: Case expressions that return BOOLEAN are not supported for FILTER
[
https://issues.apache.org/jira/browse/HIVE-20277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570560#comment-16570560
]
Matt McCline commented on HIVE-20277:
-
Committed to master. [~gopalv] thank you for your review!
> Vectorization: Case expressions that return BOOLEAN are not supported for
> FILTER
>
>
> Key: HIVE-20277
> URL: https://issues.apache.org/jira/browse/HIVE-20277
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Labels: Branch3Candidate
> Fix For: 4.0.0
>
> Attachments: HIVE-20277.02.patch, HIVE-20277.03.patch,
> HIVE-20277.04.patch, HIVE-20277.WIP.01.patch
>
>
> In cases like Query89, the vertex with the filter is not vectorized.
> {code}
>Filter Operator
> predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN
> (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (null) END
> (type: boolean)
> {code}
> {code}
> Reducer 3
> Execution mode: llap
> Reduce Vectorization:
> enabled: true
> enableConditionsMet: hive.vectorized.execution.reduce.enabled
> IS true, hive.execution.engine tez IN [tez, spark] IS true
> notVectorizedReason: FILTER operator: Unexpected hive type
> name void
> vectorized: false
> {code}
> The query specifically has
> {code}
> where case when (avg_monthly_sales <> 0) then (abs(sum_sales -
> avg_monthly_sales) / avg_monthly_sales) else null end > 0.1
> {code}
> while rewriting it to
> {code}
> where case when (avg_monthly_sales <> 0) then (abs(sum_sales -
> avg_monthly_sales) / avg_monthly_sales) > 0.1 else false end
> {code}
> does vectorize into
> {code}
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression: SelectColumnIsTrue(col
> 12:boolean)(children: VectorUDFAdaptor(CASE WHEN ((avg_window_0 <> 0.0D))
> THEN (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false)
> END)(children: DoubleColNotEqualDoubleScalar(col 7:double, val 0.0) ->
> 8:boolean, DoubleColGreaterDoubleScalar(col 9:double, val 0.1)(children:
> DoubleColDivideDoubleColumn(col 10:double, col 7:double)(children:
> FuncAbsDoubleToDouble(col 9:double)(children:
> DoubleColSubtractDoubleColumn(col 6:double, col 7:double) -> 9:double) ->
> 10:double) -> 9:double) -> 11:boolean) -> 12:boolean)
> predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN
> (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false) END
> (type: boolean)
> Statistics: Num rows: 11 Data size: 5291 Basic stats:
> COMPLETE Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20277) Vectorization: Case expressions that return BOOLEAN are not supported for FILTER
[
https://issues.apache.org/jira/browse/HIVE-20277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20277:
Fix Version/s: 4.0.0
> Vectorization: Case expressions that return BOOLEAN are not supported for
> FILTER
>
>
> Key: HIVE-20277
> URL: https://issues.apache.org/jira/browse/HIVE-20277
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Labels: Branch3Candidate
> Fix For: 4.0.0
>
> Attachments: HIVE-20277.02.patch, HIVE-20277.03.patch,
> HIVE-20277.04.patch, HIVE-20277.WIP.01.patch
>
>
> In cases like Query89, the vertex with the filter is not vectorized.
> {code}
>Filter Operator
> predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN
> (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (null) END
> (type: boolean)
> {code}
> {code}
> Reducer 3
> Execution mode: llap
> Reduce Vectorization:
> enabled: true
> enableConditionsMet: hive.vectorized.execution.reduce.enabled
> IS true, hive.execution.engine tez IN [tez, spark] IS true
> notVectorizedReason: FILTER operator: Unexpected hive type
> name void
> vectorized: false
> {code}
> The query specifically has
> {code}
> where case when (avg_monthly_sales <> 0) then (abs(sum_sales -
> avg_monthly_sales) / avg_monthly_sales) else null end > 0.1
> {code}
> while rewriting it to
> {code}
> where case when (avg_monthly_sales <> 0) then (abs(sum_sales -
> avg_monthly_sales) / avg_monthly_sales) > 0.1 else false end
> {code}
> does vectorize into
> {code}
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression: SelectColumnIsTrue(col
> 12:boolean)(children: VectorUDFAdaptor(CASE WHEN ((avg_window_0 <> 0.0D))
> THEN (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false)
> END)(children: DoubleColNotEqualDoubleScalar(col 7:double, val 0.0) ->
> 8:boolean, DoubleColGreaterDoubleScalar(col 9:double, val 0.1)(children:
> DoubleColDivideDoubleColumn(col 10:double, col 7:double)(children:
> FuncAbsDoubleToDouble(col 9:double)(children:
> DoubleColSubtractDoubleColumn(col 6:double, col 7:double) -> 9:double) ->
> 10:double) -> 9:double) -> 11:boolean) -> 12:boolean)
> predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN
> (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false) END
> (type: boolean)
> Statistics: Num rows: 11 Data size: 5291 Basic stats:
> COMPLETE Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20277) Vectorization: Case expressions that return BOOLEAN are not supported for FILTER
[
https://issues.apache.org/jira/browse/HIVE-20277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20277:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Vectorization: Case expressions that return BOOLEAN are not supported for
> FILTER
>
>
> Key: HIVE-20277
> URL: https://issues.apache.org/jira/browse/HIVE-20277
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Labels: Branch3Candidate
> Fix For: 4.0.0
>
> Attachments: HIVE-20277.02.patch, HIVE-20277.03.patch,
> HIVE-20277.04.patch, HIVE-20277.WIP.01.patch
>
>
> In cases like Query89, the vertex with the filter is not vectorized.
> {code}
>Filter Operator
> predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN
> (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (null) END
> (type: boolean)
> {code}
> {code}
> Reducer 3
> Execution mode: llap
> Reduce Vectorization:
> enabled: true
> enableConditionsMet: hive.vectorized.execution.reduce.enabled
> IS true, hive.execution.engine tez IN [tez, spark] IS true
> notVectorizedReason: FILTER operator: Unexpected hive type
> name void
> vectorized: false
> {code}
> The query specifically has
> {code}
> where case when (avg_monthly_sales <> 0) then (abs(sum_sales -
> avg_monthly_sales) / avg_monthly_sales) else null end > 0.1
> {code}
> while rewriting it to
> {code}
> where case when (avg_monthly_sales <> 0) then (abs(sum_sales -
> avg_monthly_sales) / avg_monthly_sales) > 0.1 else false end
> {code}
> does vectorize into
> {code}
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression: SelectColumnIsTrue(col
> 12:boolean)(children: VectorUDFAdaptor(CASE WHEN ((avg_window_0 <> 0.0D))
> THEN (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false)
> END)(children: DoubleColNotEqualDoubleScalar(col 7:double, val 0.0) ->
> 8:boolean, DoubleColGreaterDoubleScalar(col 9:double, val 0.1)(children:
> DoubleColDivideDoubleColumn(col 10:double, col 7:double)(children:
> FuncAbsDoubleToDouble(col 9:double)(children:
> DoubleColSubtractDoubleColumn(col 6:double, col 7:double) -> 9:double) ->
> 10:double) -> 9:double) -> 11:boolean) -> 12:boolean)
> predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN
> (((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false) END
> (type: boolean)
> Statistics: Num rows: 11 Data size: 5291 Basic stats:
> COMPLETE Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId
[ https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570564#comment-16570564 ] Eugene Koifman commented on HIVE-20291: --- I think if there is way to allocate a unique statement id for each writer that is the best option. > Allow HiveStreamingConnection to receive a WriteId > -- > > Key: HIVE-20291 > URL: https://issues.apache.org/jira/browse/HIVE-20291 > Project: Hive > Issue Type: Improvement >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20291.1.patch, HIVE-20291.2.patch > > > If the writeId is received externally it won't need to open connections to > the metastore. It won't be able to the commit in this case as well so it must > be done by the entity passing the writeId. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20278) Druid Scan Query avoid copying from List -> Map -> List
[
https://issues.apache.org/jira/browse/HIVE-20278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570570#comment-16570570
]
Hive QA commented on HIVE-20278:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} druid-handler in master has 13 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} druid-handler: The patch generated 16 new + 165
unchanged - 2 fixed = 181 total (was 167) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13063/dev-support/hive-personality.sh
|
| git revision | master / fe825bd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13063/yetus/diff-checkstyle-druid-handler.txt
|
| modules | C: druid-handler U: druid-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13063/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Druid Scan Query avoid copying from List -> Map -> List
> ---
>
> Key: HIVE-20278
> URL: https://issues.apache.org/jira/browse/HIVE-20278
> Project: Hive
> Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: PERFORMANCE
> Attachments: HIVE-20278.patch
>
>
> DruidScanQueryRecordReader gets a compacted List from druid. It then
> converts that list into a Map as DruidWritable where key is
> the column name.
> At the second stage DruidSerde takes this DruidWritable and creates a List
> out out of the map again. We can avoid the map creation part by reading the
> list sent by druid directly in the DruidSerde.deserialize() method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20118) SessionStateUserAuthenticator.getGroupNames() is always empty
[ https://issues.apache.org/jira/browse/HIVE-20118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20118: --- Status: Patch Available (was: Reopened) > SessionStateUserAuthenticator.getGroupNames() is always empty > - > > Key: HIVE-20118 > URL: https://issues.apache.org/jira/browse/HIVE-20118 > Project: Hive > Issue Type: Improvement > Components: Authorization >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20118.1.patch, HIVE-20118.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: Patch Available (was: In Progress)
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: In Progress (was: Patch Available)
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: In Progress (was: Patch Available)
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: Patch Available (was: In Progress)
Resubmit the diff as HIVE-20225.4.patch to trigger ptest
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch,
> HIVE-20225.6.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Attachment: HIVE-20225.6.patch
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch,
> HIVE-20225.6.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570628#comment-16570628
]
Lu Li edited comment on HIVE-20225 at 8/6/18 6:48 PM:
--
Resubmit the diff of HIVE-20225.4.patch as HIVE-20225.6.patch to trigger ptest
was (Author: luli):
Resubmit the diff as HIVE-20225.4.patch to trigger ptest
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch,
> HIVE-20225.6.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[ https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20315: Status: In Progress (was: Patch Available) > Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary > casts/conversions > --- > > Key: HIVE-20315 > URL: https://issues.apache.org/jira/browse/HIVE-20315 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, > HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, > HIVE-20315.06.patch, HIVE-20315.07.patch > > > Generate multi-byte Unicode characters in addition to regular single byte > characters for random data. > Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in > vectorization without padding. > Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL > constants. > Fix NULL / Wrong Results issues in VectorElt. > Change performance Q files to generate non-user EXPLAIN with VECTORIZATION > display so unnecesary CAST / DECIMAL_64 conversions are visible. > Add test for vectorized STRUCT field reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[ https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20315: Attachment: HIVE-20315.07.patch > Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary > casts/conversions > --- > > Key: HIVE-20315 > URL: https://issues.apache.org/jira/browse/HIVE-20315 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, > HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, > HIVE-20315.06.patch, HIVE-20315.07.patch > > > Generate multi-byte Unicode characters in addition to regular single byte > characters for random data. > Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in > vectorization without padding. > Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL > constants. > Fix NULL / Wrong Results issues in VectorElt. > Change performance Q files to generate non-user EXPLAIN with VECTORIZATION > display so unnecesary CAST / DECIMAL_64 conversions are visible. > Add test for vectorized STRUCT field reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[ https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20315: Status: Patch Available (was: In Progress) > Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary > casts/conversions > --- > > Key: HIVE-20315 > URL: https://issues.apache.org/jira/browse/HIVE-20315 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, > HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, > HIVE-20315.06.patch, HIVE-20315.07.patch > > > Generate multi-byte Unicode characters in addition to regular single byte > characters for random data. > Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in > vectorization without padding. > Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL > constants. > Fix NULL / Wrong Results issues in VectorElt. > Change performance Q files to generate non-user EXPLAIN with VECTORIZATION > display so unnecesary CAST / DECIMAL_64 conversions are visible. > Add test for vectorized STRUCT field reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20278) Druid Scan Query avoid copying from List -> Map -> List
[
https://issues.apache.org/jira/browse/HIVE-20278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570642#comment-16570642
]
Hive QA commented on HIVE-20278:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934516/HIVE-20278.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14865 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13063/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13063/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13063/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934516 - PreCommit-HIVE-Build
> Druid Scan Query avoid copying from List -> Map -> List
> ---
>
> Key: HIVE-20278
> URL: https://issues.apache.org/jira/browse/HIVE-20278
> Project: Hive
> Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: PERFORMANCE
> Attachments: HIVE-20278.patch
>
>
> DruidScanQueryRecordReader gets a compacted List from druid. It then
> converts that list into a Map as DruidWritable where key is
> the column name.
> At the second stage DruidSerde takes this DruidWritable and creates a List
> out out of the map again. We can avoid the map creation part by reading the
> list sent by druid directly in the DruidSerde.deserialize() method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20300) VectorFileSinkArrowOperator
[ https://issues.apache.org/jira/browse/HIVE-20300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20300: Status: Patch Available (was: Open) > VectorFileSinkArrowOperator > --- > > Key: HIVE-20300 > URL: https://issues.apache.org/jira/browse/HIVE-20300 > Project: Hive > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-20300.1.patch, HIVE-20300.2.patch, > HIVE-20300.3.patch, HIVE-20300.4.patch > > > Bypass the row-mode FileSinkOperator for pushing Arrow format to the > LlapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20300) VectorFileSinkArrowOperator
[ https://issues.apache.org/jira/browse/HIVE-20300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20300: Status: Open (was: Patch Available) > VectorFileSinkArrowOperator > --- > > Key: HIVE-20300 > URL: https://issues.apache.org/jira/browse/HIVE-20300 > Project: Hive > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-20300.1.patch, HIVE-20300.2.patch, > HIVE-20300.3.patch, HIVE-20300.4.patch > > > Bypass the row-mode FileSinkOperator for pushing Arrow format to the > LlapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20300) VectorFileSinkArrowOperator
[ https://issues.apache.org/jira/browse/HIVE-20300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20300: Attachment: HIVE-20300.4.patch > VectorFileSinkArrowOperator > --- > > Key: HIVE-20300 > URL: https://issues.apache.org/jira/browse/HIVE-20300 > Project: Hive > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-20300.1.patch, HIVE-20300.2.patch, > HIVE-20300.3.patch, HIVE-20300.4.patch > > > Bypass the row-mode FileSinkOperator for pushing Arrow format to the > LlapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20118) SessionStateUserAuthenticator.getGroupNames() is always empty
[
https://issues.apache.org/jira/browse/HIVE-20118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570682#comment-16570682
]
Hive QA commented on HIVE-20118:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 2303 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13064/dev-support/hive-personality.sh
|
| git revision | master / fe825bd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13064/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SessionStateUserAuthenticator.getGroupNames() is always empty
> -
>
> Key: HIVE-20118
> URL: https://issues.apache.org/jira/browse/HIVE-20118
> Project: Hive
> Issue Type: Improvement
> Components: Authorization
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20118.1.patch, HIVE-20118.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20118) SessionStateUserAuthenticator.getGroupNames() is always empty
[
https://issues.apache.org/jira/browse/HIVE-20118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570701#comment-16570701
]
Hive QA commented on HIVE-20118:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934380/HIVE-20118.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14865 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_drop_table]
(batchId=183)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13064/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13064/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13064/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934380 - PreCommit-HIVE-Build
> SessionStateUserAuthenticator.getGroupNames() is always empty
> -
>
> Key: HIVE-20118
> URL: https://issues.apache.org/jira/browse/HIVE-20118
> Project: Hive
> Issue Type: Improvement
> Components: Authorization
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20118.1.patch, HIVE-20118.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20283) Logs may be directed to 2 files if --hiveconf hive.log.file is used (metastore)
[ https://issues.apache.org/jira/browse/HIVE-20283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-20283: -- Labels: pull-request-available (was: ) > Logs may be directed to 2 files if --hiveconf hive.log.file is used > (metastore) > --- > > Key: HIVE-20283 > URL: https://issues.apache.org/jira/browse/HIVE-20283 > Project: Hive > Issue Type: Bug >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > > Unfortunately when doing this : > https://issues.apache.org/jira/browse/HIVE-19886 I forgot to do it as well > for the metastore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20283) Logs may be directed to 2 files if --hiveconf hive.log.file is used (metastore)
[ https://issues.apache.org/jira/browse/HIVE-20283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570711#comment-16570711 ] ASF GitHub Bot commented on HIVE-20283: --- GitHub user beltran opened a pull request: https://github.com/apache/hive/pull/409 HIVE-20283: Logs may be directed to 2 files if --hiveconf hive.log.fi… …le is used (metastore) You can merge this pull request into a Git repository by running: $ git pull https://github.com/beltran/hive HIVE-20283 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #409 commit a60c2297aa0639cf2fe4c68aa9cc44a550dd3384 Author: Jaume Marhuenda Date: 2018-08-06T20:01:04Z HIVE-20283: Logs may be directed to 2 files if --hiveconf hive.log.file is used (metastore) > Logs may be directed to 2 files if --hiveconf hive.log.file is used > (metastore) > --- > > Key: HIVE-20283 > URL: https://issues.apache.org/jira/browse/HIVE-20283 > Project: Hive > Issue Type: Bug >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > > Unfortunately when doing this : > https://issues.apache.org/jira/browse/HIVE-19886 I forgot to do it as well > for the metastore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20278) Druid Scan Query avoid copying from List -> Map -> List
[ https://issues.apache.org/jira/browse/HIVE-20278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570713#comment-16570713 ] Ashutosh Chauhan commented on HIVE-20278: - +1 What will it take to have RecordReaders other than Scan to return rows in order. No reason to overhead in that case either. Can you please create a follow-up for that. > Druid Scan Query avoid copying from List -> Map -> List > --- > > Key: HIVE-20278 > URL: https://issues.apache.org/jira/browse/HIVE-20278 > Project: Hive > Issue Type: Improvement >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: PERFORMANCE > Attachments: HIVE-20278.patch > > > DruidScanQueryRecordReader gets a compacted List from druid. It then > converts that list into a Map as DruidWritable where key is > the column name. > At the second stage DruidSerde takes this DruidWritable and creates a List > out out of the map again. We can avoid the map creation part by reading the > list sent by druid directly in the DruidSerde.deserialize() method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20278) Druid Scan Query avoid copying from List -> Map -> List
[ https://issues.apache.org/jira/browse/HIVE-20278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20278: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Nishant! > Druid Scan Query avoid copying from List -> Map -> List > --- > > Key: HIVE-20278 > URL: https://issues.apache.org/jira/browse/HIVE-20278 > Project: Hive > Issue Type: Improvement >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: PERFORMANCE > Fix For: 4.0.0 > > Attachments: HIVE-20278.patch > > > DruidScanQueryRecordReader gets a compacted List from druid. It then > converts that list into a Map as DruidWritable where key is > the column name. > At the second stage DruidSerde takes this DruidWritable and creates a List > out out of the map again. We can avoid the map creation part by reading the > list sent by druid directly in the DruidSerde.deserialize() method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20283) Logs may be directed to 2 files if --hiveconf hive.log.file is used (metastore)
[ https://issues.apache.org/jira/browse/HIVE-20283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20283: --- Attachment: HIVE-20283.1.patch Status: Patch Available (was: Open) > Logs may be directed to 2 files if --hiveconf hive.log.file is used > (metastore) > --- > > Key: HIVE-20283 > URL: https://issues.apache.org/jira/browse/HIVE-20283 > Project: Hive > Issue Type: Bug >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20283.1.patch > > > Unfortunately when doing this : > https://issues.apache.org/jira/browse/HIVE-19886 I forgot to do it as well > for the metastore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20118) SessionStateUserAuthenticator.getGroupNames() is always empty
[
https://issues.apache.org/jira/browse/HIVE-20118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570719#comment-16570719
]
Hive QA commented on HIVE-20118:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934380/HIVE-20118.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13065/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13065/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13065/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12934380/HIVE-20118.2.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934380 - PreCommit-HIVE-Build
> SessionStateUserAuthenticator.getGroupNames() is always empty
> -
>
> Key: HIVE-20118
> URL: https://issues.apache.org/jira/browse/HIVE-20118
> Project: Hive
> Issue Type: Improvement
> Components: Authorization
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20118.1.patch, HIVE-20118.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20118) SessionStateUserAuthenticator.getGroupNames() is always empty
[ https://issues.apache.org/jira/browse/HIVE-20118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20118: -- Attachment: HIVE-20118.3.patch > SessionStateUserAuthenticator.getGroupNames() is always empty > - > > Key: HIVE-20118 > URL: https://issues.apache.org/jira/browse/HIVE-20118 > Project: Hive > Issue Type: Improvement > Components: Authorization >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20118.1.patch, HIVE-20118.2.patch, > HIVE-20118.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570743#comment-16570743
]
Hive QA commented on HIVE-20225:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} contrib in master has 13 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} contrib: The patch generated 2 new + 0 unchanged - 0
fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13066/dev-support/hive-personality.sh
|
| git revision | master / 632fa4f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13066/yetus/diff-checkstyle-contrib.txt
|
| modules | C: contrib U: contrib |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13066/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch,
> HIVE-20225.6.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate t
[jira] [Updated] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag
[ https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20320: --- Attachment: HIVE-20320.1.patch > Turn on hive.optimize.remove.sq_count_check flag > > > Key: HIVE-20320 > URL: https://issues.apache.org/jira/browse/HIVE-20320 > Project: Hive > Issue Type: Task > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20320.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag
[ https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20320: --- Status: Patch Available (was: Open) > Turn on hive.optimize.remove.sq_count_check flag > > > Key: HIVE-20320 > URL: https://issues.apache.org/jira/browse/HIVE-20320 > Project: Hive > Issue Type: Task > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20320.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20323) Update desc formatted/extended table to show if constraint is enabled or disabled and rely/norely
[
https://issues.apache.org/jira/browse/HIVE-20323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg reassigned HIVE-20323:
--
> Update desc formatted/extended table to show if constraint is enabled or
> disabled and rely/norely
> -
>
> Key: HIVE-20323
> URL: https://issues.apache.org/jira/browse/HIVE-20323
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> Currently {{desc formatted }} do not show if a constraint is
> enabled or disable (or Rely or no rely). It is hard to figure out if a
> constraint is enabled or disabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20290) Lazy initialize ArrowColumnarBatchSerDe so it doesn't allocate buffers during GetSplits
[
https://issues.apache.org/jira/browse/HIVE-20290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20290:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Committed to master
> Lazy initialize ArrowColumnarBatchSerDe so it doesn't allocate buffers during
> GetSplits
> ---
>
> Key: HIVE-20290
> URL: https://issues.apache.org/jira/browse/HIVE-20290
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Fix For: 4.0.0
>
> Attachments: HIVE-20290.1.patch, HIVE-20290.2.patch
>
>
> When using {{GenericUDTFGetSplits}} to create {{LlapInputSplit}} for
> submission to {{LlapOutputFormatService}}, the physical plan generation
> initializes whatever SerDe is being used.
> {{ArrowColumnarBatchSerDe}} initializes buffers for Arrow and
> {{VectorizedRowBatch}} at this point inside HS2 which are never used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570800#comment-16570800
]
Hive QA commented on HIVE-20225:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934527/HIVE-20225.6.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14883 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestCatalogNonDefaultClient.databases
(batchId=218)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13066/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13066/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13066/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934527 - PreCommit-HIVE-Build
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch,
> HIVE-20225.6.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: In Progress (was: Patch Available)
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch,
> HIVE-20225.6.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Attachment: HIVE-20225.7.patch
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch,
> HIVE-20225.6.patch, HIVE-20225.7.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: Patch Available (was: In Progress)
Resubmit the diff of HIVE-20225.4.patch as HIVE-20225.7.patch to trigger ptest
because random test failed
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch,
> HIVE-20225.6.patch, HIVE-20225.7.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
