[jira] [Commented] (HIVE-16654) Optimize a combination of avg(), sum(), count(distinct) etc
[
https://issues.apache.org/jira/browse/HIVE-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018700#comment-16018700
]
Hive QA commented on HIVE-16654:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869152/HIVE-16654.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 68 failed/errored test(s), 10743 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby3_map]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby3_map_skew]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_11]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_8]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nullgroup4] (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[quotedid_skew]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin10]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin11]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin1]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin2]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin3]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin4]
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin6]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin7]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin9]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_union_remove_1]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_union_remove_2]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt10]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt11]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt12]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt14]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt16]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt17]
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt19]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt1]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt20]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt21]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt3]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt4]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt5]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt6]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt7]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt8]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_count] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_empty_where]
(batchId=22)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[count_dist_rewrite]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_1]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadataonly1]
(batchId=156)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query16]
(batchId=231)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query28]
(batchId=231)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query94]
(batchId=231)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query95]
(batchId=231)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby3_map]
(batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby3_map_skew]
(batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[nullgroup4]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoin_union_remove_1]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoin_union_remove_2]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt10]
(batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt11]
(batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt12]
(batchId=103)
[jira] [Updated] (HIVE-16723) Enable configurable MetaStoreSchemaInfo
[
https://issues.apache.org/jira/browse/HIVE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-16723:
---
Attachment: HIVE-16723.03.patch
Added the missing apache header for the IMetastoreSchemaInfo interface
> Enable configurable MetaStoreSchemaInfo
>
>
> Key: HIVE-16723
> URL: https://issues.apache.org/jira/browse/HIVE-16723
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16723.01.patch, HIVE-16723.02.patch,
> HIVE-16723.03.patch
>
>
> {{MetaStoreSchemaInfo}} class is used by Schema tool to get the schema
> version information. In addition to that schema tool invokes its init and
> upgrade methods to initialize and upgrade the metastore schema. It is
> possible that there are minor schema changes in the metastore schema which
> users have to maintain manually. We can potentially enhance schema tool to
> use a custom MetaStoreSchemaInfo implementation which can be plugged in based
> on configuration and schematool could run these customizations while running
> the upgrade and init scripts.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16654) Optimize a combination of avg(), sum(), count(distinct) etc
[
https://issues.apache.org/jira/browse/HIVE-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-16654:
---
Status: Patch Available (was: Open)
> Optimize a combination of avg(), sum(), count(distinct) etc
> ---
>
> Key: HIVE-16654
> URL: https://issues.apache.org/jira/browse/HIVE-16654
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16654.01.patch
>
>
> an example rewrite for q28 of tpcds is
> {code}
> (select LP as B1_LP ,CNT as B1_CNT,CNTD as B1_CNTD
> from (select sum(xc0) / sum(xc1) as LP, sum(xc1) as CNT, count(1) as
> CNTD from (select sum(ss_list_price) as xc0, count(ss_list_price) as xc1 from
> store_sales where
> ss_list_price is not null and ss_quantity between 0 and 5
> and (ss_list_price between 11 and 11+10
> or ss_coupon_amt between 460 and 460+1000
> or ss_wholesale_cost between 14 and 14+20)
> group by ss_list_price) ss0) ss1) B1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16654) Optimize a combination of avg(), sum(), count(distinct) etc
[
https://issues.apache.org/jira/browse/HIVE-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-16654:
---
Attachment: HIVE-16654.01.patch
> Optimize a combination of avg(), sum(), count(distinct) etc
> ---
>
> Key: HIVE-16654
> URL: https://issues.apache.org/jira/browse/HIVE-16654
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16654.01.patch
>
>
> an example rewrite for q28 of tpcds is
> {code}
> (select LP as B1_LP ,CNT as B1_CNT,CNTD as B1_CNTD
> from (select sum(xc0) / sum(xc1) as LP, sum(xc1) as CNT, count(1) as
> CNTD from (select sum(ss_list_price) as xc0, count(ss_list_price) as xc1 from
> store_sales where
> ss_list_price is not null and ss_quantity between 0 and 5
> and (ss_list_price between 11 and 11+10
> or ss_coupon_amt between 460 and 460+1000
> or ss_wholesale_cost between 14 and 14+20)
> group by ss_list_price) ss0) ss1) B1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16721) Inconsistent behavior in dealing with Timestamp stats
[ https://issues.apache.org/jira/browse/HIVE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-16721: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Patch committed to master. Thanks [~vgumashta] [~vgarg] > Inconsistent behavior in dealing with Timestamp stats > - > > Key: HIVE-16721 > URL: https://issues.apache.org/jira/browse/HIVE-16721 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Fix For: 3.0.0 > > Attachments: HIVE-16721.1.patch > > > HIVE-15003 added support for additional types for col stats. However, it > treats timestamp as DateColumnStatsData whereas when we read the timestamp > stats, we read as LongColumnStatsData > (https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java#L229). > We should make it consistent with original hive behavior -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16207) Add support for Complex Types in Fast SerDe
[ https://issues.apache.org/jira/browse/HIVE-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018645#comment-16018645 ] Matt McCline commented on HIVE-16207: - No, I don't think it needs to be documented. Thanks > Add support for Complex Types in Fast SerDe > --- > > Key: HIVE-16207 > URL: https://issues.apache.org/jira/browse/HIVE-16207 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Teddy Choi >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-16207.1.patch, HIVE-16207.1.patch.zip, > HIVE-16207.2.patch, HIVE-16207.3.patch, HIVE-16207.4.patch, > HIVE-16207.5.patch, HIVE-16207.6.patch, partial.patch > > > Add complex type support to Fast SerDe classes. This is needed for fully > supporting complex types in Vectorization -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HIVE-16113:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Resolved via
https://git-wip-us.apache.org/repos/asf?p=hive.git;a=commit;h=5f4eaa9b13e7beec8bb16fea94fec386e2bc1e00
> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Fix For: 3.0.0
>
> Attachments: HIVE-16113.1.patch, HIVE-16113.2.patch,
> HIVE-16113.3.patch, HIVE-16113.4.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018591#comment-16018591
]
Hive QA commented on HIVE-16113:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869132/HIVE-16113.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10741 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[named_column_join]
(batchId=72)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5371/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5371/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5371/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869132 - PreCommit-HIVE-Build
> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch, HIVE-16113.2.patch,
> HIVE-16113.3.patch, HIVE-16113.4.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16711) Remove property_id column from metastore_db_properties table
[
https://issues.apache.org/jira/browse/HIVE-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018573#comment-16018573
]
Hive QA commented on HIVE-16711:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869127/HIVE-16711.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10740 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5370/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5370/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5370/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869127 - PreCommit-HIVE-Build
> Remove property_id column from metastore_db_properties table
>
>
> Key: HIVE-16711
> URL: https://issues.apache.org/jira/browse/HIVE-16711
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16711.01.patch, HIVE-16711.02.patch,
> HIVE-16711.03.patch
>
>
> The property id column of the table {{METASTORE_DB_PROPERTIES}} is not really
> needed. We could instead convert column {{PROPERTY_KEY}} to primary key and
> change the identity-type to "application" in package.jdo file.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HIVE-16113:
Attachment: HIVE-16113.4.patch
PAtch.4 add the explainuser_3/analyze_3 GF diff (altough they're gen 88).
> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch, HIVE-16113.2.patch,
> HIVE-16113.3.patch, HIVE-16113.4.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16711) Remove property_id column from metastore_db_properties table
[
https://issues.apache.org/jira/browse/HIVE-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-16711:
---
Attachment: HIVE-16711.03.patch
Fixed one of the test in TestSchemaTool used for metastore_db_properties
> Remove property_id column from metastore_db_properties table
>
>
> Key: HIVE-16711
> URL: https://issues.apache.org/jira/browse/HIVE-16711
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16711.01.patch, HIVE-16711.02.patch,
> HIVE-16711.03.patch
>
>
> The property id column of the table {{METASTORE_DB_PROPERTIES}} is not really
> needed. We could instead convert column {{PROPERTY_KEY}} to primary key and
> change the identity-type to "application" in package.jdo file.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16177) non Acid to acid conversion doesn't handle _copy_N files
[
https://issues.apache.org/jira/browse/HIVE-16177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018541#comment-16018541
]
Hive QA commented on HIVE-16177:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869126/HIVE-16177.11.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5369/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5369/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5369/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-20 16:43:10.293
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-5369/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-20 16:43:10.296
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 7429f5f HIVE-16717: Extend shared scan optimizer to handle
partitions (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/partition_pruning.q
Removing ql/src/test/results/clientpositive/llap/partition_pruning.q.out
Removing ql/src/test/results/clientpositive/partition_pruning.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 7429f5f HIVE-16717: Extend shared scan optimizer to handle
partitions (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-20 16:43:11.077
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
patching file ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java
patching file ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java
patching file ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
patching file ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java
patching file ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
patching file ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
patching file ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java
patching file ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
patching file ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java
patching file
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java
patching file
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java
patching file
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
patching file
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
patching file
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/metastore/target/generated-sources/antlr3/org/apache/hadoop/hive/metastore/parser/FilterParser.java
[jira] [Commented] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018538#comment-16018538
]
Hive QA commented on HIVE-16113:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869114/HIVE-16113.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10741 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5368/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5368/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5368/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869114 - PreCommit-HIVE-Build
> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch, HIVE-16113.2.patch,
> HIVE-16113.3.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16177) non Acid to acid conversion doesn't handle _copy_N files
[
https://issues.apache.org/jira/browse/HIVE-16177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-16177:
--
Attachment: HIVE-16177.11.patch
> non Acid to acid conversion doesn't handle _copy_N files
>
>
> Key: HIVE-16177
> URL: https://issues.apache.org/jira/browse/HIVE-16177
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-16177.01.patch, HIVE-16177.02.patch,
> HIVE-16177.04.patch, HIVE-16177.07.patch, HIVE-16177.08.patch,
> HIVE-16177.09.patch, HIVE-16177.10.patch, HIVE-16177.11.patch
>
>
> {noformat}
> create table T(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES('transactional'='false')
> insert into T(a,b) values(1,2)
> insert into T(a,b) values(1,3)
> alter table T SET TBLPROPERTIES ('transactional'='true')
> {noformat}
> //we should now have bucket files 01_0 and 01_0_copy_1
> but OrcRawRecordMerger.OriginalReaderPair.next() doesn't know that there can
> be copy_N files and numbers rows in each bucket from 0 thus generating
> duplicate IDs
> {noformat}
> select ROW__ID, INPUT__FILE__NAME, a, b from T
> {noformat}
> produces
> {noformat}
> {"transactionid":0,"bucketid":1,"rowid":0},file:/Users/ekoifman/dev/hiverwgit/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands.../warehouse/nonacidorctbl/01_0,1,2
> {"transactionid\":0,"bucketid":1,"rowid":0},file:/Users/ekoifman/dev/hiverwgit/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands.../warehouse/nonacidorctbl/01_0_copy_1,1,3
> {noformat}
> [~owen.omalley], do you have any thoughts on a good way to handle this?
> attached patch has a few changes to make Acid even recognize copy_N but this
> is just a pre-requisite. The new UT demonstrates the issue.
> Futhermore,
> {noformat}
> alter table T compact 'major'
> select ROW__ID, INPUT__FILE__NAME, a, b from T order by b
> {noformat}
> produces
> {noformat}
> {"transactionid":0,"bucketid":1,"rowid":0}
> file:/Users/ekoifman/dev/hiverwgit/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommandswarehouse/nonacidorctbl/base_-9223372036854775808/bucket_1
> 1 2
> {noformat}
> HIVE-16177.04.patch has TestTxnCommands.testNonAcidToAcidConversion0()
> demonstrating this
> This is because compactor doesn't handle copy_N files either (skips them)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16693) beeline "source" command freezes if you have a comment in it?
[
https://issues.apache.org/jira/browse/HIVE-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018515#comment-16018515
]
Carter Shanklin commented on HIVE-16693:
Looks good
> beeline "source" command freezes if you have a comment in it?
> -
>
> Key: HIVE-16693
> URL: https://issues.apache.org/jira/browse/HIVE-16693
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Ferdinand Xu
> Attachments: HIVE-16693.patch
>
>
> As far as I'm observing in my environment, which is HDP 2.6.1, version
> identified as 2.1.0.2.6.1.0-69 rc65de77a7cab18faf09a1f8e54c5ad5c44af8957
> Call this file temp.sql:
> {code}
> -- comment
> select 1;
> {code}
> Then in beeline run:
> {code}
> source temp.sql;
> {code}
> When I do this beeline freezes and must be killed.
> The jstacks indicate a tight loop in
> org.apache.hive.beeline.Commands.sourceFileInternal:
> main,org.apache.hive.beeline.Commands.sourceFileInternal(877)
> main,org.apache.hive.beeline.Commands.sourceFile(860)
> main,org.apache.hive.beeline.Commands.executeInternal(933)
> main,org.apache.hive.beeline.Commands.execute(1161)
> main,org.apache.hive.beeline.Commands.sql(1076)
> main,org.apache.hive.beeline.BeeLine.dispatch(1145)
> There's never any other active thread.
> Looking at
> https://github.com/apache/hive/blob/455ffdd9125bdfe73b2c7f7ddebaeff138b77f53/beeline/src/java/org/apache/hive/beeline/Commands.java#L896
> Don't you need to read the next line before you continue?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr
[
https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018507#comment-16018507
]
Hive QA commented on HIVE-15104:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869112/TPC-H%20100G.xlsx
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5367/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5367/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5367/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-20 15:31:32.562
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-5367/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-20 15:31:32.564
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 7429f5f HIVE-16717: Extend shared scan optimizer to handle
partitions (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git clean -f -d
Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgDecimal.txt
Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgDecimalMerge.txt
Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgMerge.txt
Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgTimestamp.txt
Removing
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFSumTimestamp.java
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 7429f5f HIVE-16717: Extend shared scan optimizer to handle
partitions (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-20 15:31:33.510
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869112 - PreCommit-HIVE-Build
> Hive on Spark generate more shuffle data than hive on mr
>
>
> Key: HIVE-15104
> URL: https://issues.apache.org/jira/browse/HIVE-15104
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.1
>Reporter: wangwenli
>Assignee: Rui Li
> Attachments: HIVE-15104.1.patch, HIVE-15104.2.patch,
> HIVE-15104.3.patch, TPC-H 100G.xlsx
>
>
> the same sql, running on spark and mr engine, will generate different size
> of shuffle data.
> i think it is because of hive on mr just serialize part of HiveKey, but hive
> on spark which using kryo will serialize full of Hivekey object.
> what is your opionion?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HIVE-16113:
Attachment: HIVE-16113.3.patch
Patch.3 rebased to current master for new test run
> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch, HIVE-16113.2.patch,
> HIVE-16113.3.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr
[ https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15104: -- Attachment: TPC-H 100G.xlsx Attaching TPC-H benchmark result. It shows the improvement is more obvious for long queries when we need to shuffle a lot of data. And it's better to use with groupBy shuffle disabled. > Hive on Spark generate more shuffle data than hive on mr > > > Key: HIVE-15104 > URL: https://issues.apache.org/jira/browse/HIVE-15104 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.1 >Reporter: wangwenli >Assignee: Rui Li > Attachments: HIVE-15104.1.patch, HIVE-15104.2.patch, > HIVE-15104.3.patch, TPC-H 100G.xlsx > > > the same sql, running on spark and mr engine, will generate different size > of shuffle data. > i think it is because of hive on mr just serialize part of HiveKey, but hive > on spark which using kryo will serialize full of Hivekey object. > what is your opionion? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG
[
https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018410#comment-16018410
]
Hive QA commented on HIVE-16589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869101/HIVE-16589.07.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 10739 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_all_complex]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part_all_complex]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vectorization_limit]
(batchId=97)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_cast_constant]
(batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_aggregate]
(batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0]
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_13]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_14]
(batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_15]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_16]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_9]
(batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_pushdown]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_mapjoin]
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_shufflejoin]
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_timestamp_funcs]
(batchId=112)
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testAvgDecimal
(batchId=271)
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testAvgDecimalNegative
(batchId=271)
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testAvgLongNullKeyGroupBySingleBatch
(batchId=271)
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testAvgLongNulls
(batchId=271)
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testAvgLongRepeat
(batchId=271)
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testAvgLongRepeatConcatValues
(batchId=271)
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testAvgLongSimple
(batchId=271)
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testDoubleValueTypeAvg
(batchId=271)
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testDoubleValueTypeAvgOneKey
(batchId=271)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5366/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5366/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5366/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869101 - PreCommit-HIVE-Build
> Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and
> COMPLETE for AVG
> -
>
> Key: HIVE-16589
> URL: https://issues.apache.org/jira/browse/HIVE-16589
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch,
> HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch,
> HIVE-16589.06.patch, HIVE-16589.07.patch
>
>
> Allow Complex Types to be vectorized (since HIVE-16207: "Add support for
> Complex Types in Fast SerDe" was committed).
> Add more classes we vectorize AVG in preparation for fully supporting AVG
> GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in
> the AVG struct as input. And, add the COMPLETE mode that takes in the
> Original data and produces the Full Aggregation for completeness, so to speak.
[jira] [Updated] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16589: Attachment: HIVE-16589.07.patch > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG > - > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16589: Status: Patch Available (was: In Progress) > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG > - > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16589: Status: In Progress (was: Patch Available) > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG > - > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16717) Extend shared scan optimizer to handle partitions
[
https://issues.apache.org/jira/browse/HIVE-16717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-16717:
---
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master (added {{partition_shared_scan.q}} to local llap test files in
the commit).
Thanks for the review [~ashutoshc]!
> Extend shared scan optimizer to handle partitions
> -
>
> Key: HIVE-16717
> URL: https://issues.apache.org/jira/browse/HIVE-16717
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 3.0.0
>
> Attachments: HIVE-16717.01.patch, HIVE-16717.patch
>
>
> Extend shared scans optimizer rule introduced in HIVE-16602 to only merge
> scan operators on partitioned tables if the pruned partitions are equal.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16207) Add support for Complex Types in Fast SerDe
[ https://issues.apache.org/jira/browse/HIVE-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018344#comment-16018344 ] Lefty Leverenz commented on HIVE-16207: --- Should this be documented in the wiki? (I don't think Fast SerDe is documented anywhere.) > Add support for Complex Types in Fast SerDe > --- > > Key: HIVE-16207 > URL: https://issues.apache.org/jira/browse/HIVE-16207 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Teddy Choi >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-16207.1.patch, HIVE-16207.1.patch.zip, > HIVE-16207.2.patch, HIVE-16207.3.patch, HIVE-16207.4.patch, > HIVE-16207.5.patch, HIVE-16207.6.patch, partial.patch > > > Add complex type support to Fast SerDe classes. This is needed for fully > supporting complex types in Vectorization -- This message was sent by Atlassian JIRA (v6.3.15#6346)
