[jira] [Commented] (HIVE-10650) Improve sum() function over windowing to support additional range formats
[
https://issues.apache.org/jira/browse/HIVE-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541408#comment-14541408
]
Hive QA commented on HIVE-10650:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12732386/HIVE-10650.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 8922 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3869/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3869/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3869/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12732386 - PreCommit-HIVE-TRUNK-Build
> Improve sum() function over windowing to support additional range formats
> -
>
> Key: HIVE-10650
> URL: https://issues.apache.org/jira/browse/HIVE-10650
> Project: Hive
> Issue Type: Sub-task
> Components: PTF-Windowing
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-10650.patch
>
>
> Support the following windowing function {{x preceding and y preceding}} and
> {{x following and y following}}.
> e.g.
> {noformat}
> select sum(value) over (partition by key order by value rows between 2
> preceding and 1 preceding) from tbl1;
> select sum(value) over (partition by key order by value rows between
> unbounded preceding and 1 preceding) from tbl1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10582) variable typo in HiveOpConverter (714) and SemanticAnalyzer (7496)
[
https://issues.apache.org/jira/browse/HIVE-10582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541397#comment-14541397
]
Ashutosh Chauhan commented on HIVE-10582:
-
+1
> variable typo in HiveOpConverter (714) and SemanticAnalyzer (7496)
> --
>
> Key: HIVE-10582
> URL: https://issues.apache.org/jira/browse/HIVE-10582
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
>Priority: Minor
> Attachments: HIVE-10582.1.patch, rb33790.patch
>
>
> HiveOpConverter lines 703-717
> {code}
> int kindex = exprBack == null ? -1 :
> ExprNodeDescUtils.indexOf(exprBack, reduceKeysBack);
> if (kindex >= 0) {
> ColumnInfo newColInfo = new ColumnInfo(colInfo);
> newColInfo.setInternalName(Utilities.ReduceField.KEY +
> ".reducesinkkey" + kindex);
> newColInfo.setAlias(outputColName);
> newColInfo.setTabAlias(colInfo.getTabAlias());
> outputColumns.add(newColInfo);
> index[i] = kindex;
> continue;
> }
> int vindex = exprBack == null ? -1 :
> ExprNodeDescUtils.indexOf(exprBack, reduceValuesBack);
> if (kindex >= 0) { // looks like it should be vindex instead of kindex
> index[i] = -vindex - 1;
> continue;
> }
> {code}
> Most probably the second "if (kindex >= 0)" (line 714) should be replaces
> with "if (vindex >= 0)"
> The same situation in SemanticAnalyzer (7483-7499)
> {code}
> int kindex = exprBack == null ? -1 :
> ExprNodeDescUtils.indexOf(exprBack, reduceKeysBack);
> if (kindex >= 0) {
> ColumnInfo newColInfo = new ColumnInfo(colInfo);
> newColInfo.setInternalName(Utilities.ReduceField.KEY +
> ".reducesinkkey" + kindex);
> newColInfo.setTabAlias(nm[0]);
> outputRR.put(nm[0], nm[1], newColInfo);
> if (nm2 != null) {
> outputRR.addMappingOnly(nm2[0], nm2[1], newColInfo);
> }
> index[i] = kindex;
> continue;
> }
> int vindex = exprBack == null ? -1 :
> ExprNodeDescUtils.indexOf(exprBack, reduceValuesBack);
> if (kindex >= 0) { // looks like it should be vindex instead of kindex
> index[i] = -vindex - 1;
> continue;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10644) create SHA2 UDF
[ https://issues.apache.org/jira/browse/HIVE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-10644: --- Attachment: HIVE-10644.2.patch patch #2: use Text.getBytes() instead of toString() > create SHA2 UDF > --- > > Key: HIVE-10644 > URL: https://issues.apache.org/jira/browse/HIVE-10644 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Alexander Pivovarov >Assignee: Alexander Pivovarov > Attachments: HIVE-10644.1.patch, HIVE-10644.2.patch > > > Calculates the SHA-2 family of hash functions (SHA-224, SHA-256, SHA-384, and > SHA-512). The first argument is the cleartext string to be hashed. The second > argument indicates the desired bit length of the result, which must have a > value of 224, 256, 384, 512, or 0 (which is equivalent to 256). If either > argument is NULL or the hash length is not one of the permitted values, the > return value is NULL. > MySQL also has SHA2 function > https://dev.mysql.com/doc/refman/5.5/en/encryption-functions.html#function_sha2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10548) Remove dependency to s3 repository in root pom
[ https://issues.apache.org/jira/browse/HIVE-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541373#comment-14541373 ] Chengxiang Li commented on HIVE-10548: -- Committed to master, thanks Szehon for review. > Remove dependency to s3 repository in root pom > -- > > Key: HIVE-10548 > URL: https://issues.apache.org/jira/browse/HIVE-10548 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Szehon Ho >Assignee: Chengxiang Li > Attachments: HIVE-10548.2.patch, HIVE-10548.2.patch, HIVE-10548.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10636) CASE comparison operator rotation optimization
[ https://issues.apache.org/jira/browse/HIVE-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-10636: Attachment: HIVE-10636.3.patch With more tests. > CASE comparison operator rotation optimization > -- > > Key: HIVE-10636 > URL: https://issues.apache.org/jira/browse/HIVE-10636 > Project: Hive > Issue Type: New Feature > Components: Logical Optimizer >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-10636.1.patch, HIVE-10636.2.patch, > HIVE-10636.3.patch, HIVE-10636.patch > > > Step 1 as outlined in description of HIVE-9644 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10319) Hive CLI startup takes a long time with a large number of databases
[ https://issues.apache.org/jira/browse/HIVE-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541310#comment-14541310 ] Nezih Yigitbasi commented on HIVE-10319: [~jdere] Here is the RB link for the latest patch: https://reviews.apache.org/r/34147/ > Hive CLI startup takes a long time with a large number of databases > --- > > Key: HIVE-10319 > URL: https://issues.apache.org/jira/browse/HIVE-10319 > Project: Hive > Issue Type: Improvement > Components: CLI >Affects Versions: 1.0.0 >Reporter: Nezih Yigitbasi >Assignee: Nezih Yigitbasi > Attachments: HIVE-10319.1.patch, HIVE-10319.2.patch, HIVE-10319.patch > > > The Hive CLI takes a long time to start when there is a large number of > databases in the DW. I think the root cause is the way permanent UDFs are > loaded from the metastore. When I looked at the logs and the source code I > see that at startup Hive first gets all the databases from the metastore and > then for each database it makes a metastore call to get the permanent > functions for that database [see Hive.java | > https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L162-185]. > So the number of metastore calls made is in the order of the number of > databases. In production we have several hundreds of databases so Hive makes > several hundreds of RPC calls during startup, taking 30+ seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10319) Hive CLI startup takes a long time with a large number of databases
[ https://issues.apache.org/jira/browse/HIVE-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih Yigitbasi updated HIVE-10319: --- Attachment: HIVE-10319.2.patch > Hive CLI startup takes a long time with a large number of databases > --- > > Key: HIVE-10319 > URL: https://issues.apache.org/jira/browse/HIVE-10319 > Project: Hive > Issue Type: Improvement > Components: CLI >Affects Versions: 1.0.0 >Reporter: Nezih Yigitbasi >Assignee: Nezih Yigitbasi > Attachments: HIVE-10319.1.patch, HIVE-10319.2.patch, HIVE-10319.patch > > > The Hive CLI takes a long time to start when there is a large number of > databases in the DW. I think the root cause is the way permanent UDFs are > loaded from the metastore. When I looked at the logs and the source code I > see that at startup Hive first gets all the databases from the metastore and > then for each database it makes a metastore call to get the permanent > functions for that database [see Hive.java | > https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L162-185]. > So the number of metastore calls made is in the order of the number of > databases. In production we have several hundreds of databases so Hive makes > several hundreds of RPC calls during startup, taking 30+ seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10319) Hive CLI startup takes a long time with a large number of databases
[ https://issues.apache.org/jira/browse/HIVE-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih Yigitbasi updated HIVE-10319: --- Attachment: (was: HIVE-10319-no-gen.patch) > Hive CLI startup takes a long time with a large number of databases > --- > > Key: HIVE-10319 > URL: https://issues.apache.org/jira/browse/HIVE-10319 > Project: Hive > Issue Type: Improvement > Components: CLI >Affects Versions: 1.0.0 >Reporter: Nezih Yigitbasi >Assignee: Nezih Yigitbasi > Attachments: HIVE-10319.1.patch, HIVE-10319.patch > > > The Hive CLI takes a long time to start when there is a large number of > databases in the DW. I think the root cause is the way permanent UDFs are > loaded from the metastore. When I looked at the logs and the source code I > see that at startup Hive first gets all the databases from the metastore and > then for each database it makes a metastore call to get the permanent > functions for that database [see Hive.java | > https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L162-185]. > So the number of metastore calls made is in the order of the number of > databases. In production we have several hundreds of databases so Hive makes > several hundreds of RPC calls during startup, taking 30+ seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10563) MiniTezCliDriver tests ordering issues
[
https://issues.apache.org/jira/browse/HIVE-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541296#comment-14541296
]
Hive QA commented on HIVE-10563:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12732401/HIVE-10563.5.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 8921 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_distinct_gby
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3868/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3868/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3868/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12732401 - PreCommit-HIVE-TRUNK-Build
> MiniTezCliDriver tests ordering issues
> --
>
> Key: HIVE-10563
> URL: https://issues.apache.org/jira/browse/HIVE-10563
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-10563.1.patch, HIVE-10563.2.patch,
> HIVE-10563.3.patch, HIVE-10563.4.patch, HIVE-10563.5.patch
>
>
> There are a bunch of tests related to TestMiniTezCliDriver which gives
> ordering issues when run on Centos/Windows/OSX
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5733) Publish hive-exec artifact without all the dependencies
[
https://issues.apache.org/jira/browse/HIVE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541285#comment-14541285
]
Amareshwari Sriramadasu commented on HIVE-5733:
---
bq. what problems did you see with this patch?
I think the problem was - "removing shaded dependencies from hive-exec caused
failures in other modules, as other modules are expecting all dependencies come
with hive-exec". Other solution we did on our end is to make protobuf version
provided and not bundle with hive-exec jar.
> Publish hive-exec artifact without all the dependencies
> ---
>
> Key: HIVE-5733
> URL: https://issues.apache.org/jira/browse/HIVE-5733
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
>Assignee: Amareshwari Sriramadasu
> Attachments: HIVE-5733.1.patch
>
>
> Currently the artifact {{hive-exec}} that is available in
> [maven|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-exec/0.12.0/hive-exec-0.12.0.jar]
> is shading all the dependencies (= the jar contains all Hive's
> dependencies). As other projects that are depending on Hive might be use
> slightly different version of the dependencies, it can easily happens that
> Hive's shaded version will be used instead which leads to very time consuming
> debugging of what is happening (for example SQOOP-1198).
> Would it be feasible publish {{hive-exec}} jar that will be build without
> shading any dependency? For example
> [avro-tools|http://search.maven.org/#artifactdetails%7Corg.apache.avro%7Cavro-tools%7C1.7.5%7Cjar]
> is having classifier "nodeps" that represents artifact without any
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-10685) Alter table concatenate oparetor will cause duplicate data
[
https://issues.apache.org/jira/browse/HIVE-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guoliming reassigned HIVE-10685:
Assignee: guoliming
> Alter table concatenate oparetor will cause duplicate data
> --
>
> Key: HIVE-10685
> URL: https://issues.apache.org/jira/browse/HIVE-10685
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: guoliming
>Assignee: guoliming
>
> "Orders" table has 15 rows and stored as ORC.
> {noformat}
> hive> select count(*) from orders;
> OK
> 15
> Time taken: 37.692 seconds, Fetched: 1 row(s)
> {noformat}
> The table contain 14 files,the size of each file is about 2.1 ~ 3.2 GB.
> After executing command : ALTER TABLE orders CONCATENATE;
> The table is already 1530115000 rows.
> My hive version is 1.1.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10685) Alter table concatenate oparetor will cause duplicate data
[
https://issues.apache.org/jira/browse/HIVE-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guoliming updated HIVE-10685:
-
Assignee: (was: guoliming)
> Alter table concatenate oparetor will cause duplicate data
> --
>
> Key: HIVE-10685
> URL: https://issues.apache.org/jira/browse/HIVE-10685
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: guoliming
>
> "Orders" table has 15 rows and stored as ORC.
> {noformat}
> hive> select count(*) from orders;
> OK
> 15
> Time taken: 37.692 seconds, Fetched: 1 row(s)
> {noformat}
> The table contain 14 files,the size of each file is about 2.1 ~ 3.2 GB.
> After executing command : ALTER TABLE orders CONCATENATE;
> The table is already 1530115000 rows.
> My hive version is 1.1.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10686) java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
[
https://issues.apache.org/jira/browse/HIVE-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541238#comment-14541238
]
Hive QA commented on HIVE-10686:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12732367/HIVE-10686.03.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 8921 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3867/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3867/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3867/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12732367 - PreCommit-HIVE-TRUNK-Build
> java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
> -
>
> Key: HIVE-10686
> URL: https://issues.apache.org/jira/browse/HIVE-10686
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10686.01.patch, HIVE-10686.02.patch,
> HIVE-10686.03.patch, HIVE-10686.patch
>
>
> CBO throws Index out of bound exception for TPC-DS Q70.
> Query
> {code}
> explain
> select
> sum(ss_net_profit) as total_sum
>,s_state
>,s_county
>,grouping__id as lochierarchy
>, rank() over(partition by grouping__id, case when grouping__id == 2 then
> s_state end order by sum(ss_net_profit)) as rank_within_parent
> from
> store_sales ss join date_dim d1 on d1.d_date_sk = ss.ss_sold_date_sk
> join store s on s.s_store_sk = ss.ss_store_sk
> where
> d1.d_month_seq between 1193 and 1193+11
> and s.s_state in
> ( select s_state
>from (select s_state as s_state, sum(ss_net_profit),
> rank() over ( partition by s_state order by
> sum(ss_net_profit) desc) as ranking
> from store_sales, store, date_dim
> where d_month_seq between 1193 and 1193+11
> and date_dim.d_date_sk =
> store_sales.ss_sold_date_sk
> and store.s_store_sk = store_sales.ss_store_sk
> group by s_state
> ) tmp1
>where ranking <= 5
> )
> group by s_state,s_county with rollup
> order by
>lochierarchy desc
> ,case when lochierarchy = 0 then s_state end
> ,rank_within_parent
> limit 100
> {code}
> Original plan (correct)
> {code}
> HiveSort(fetch=[100])
> HiveSort(sort0=[$3], sort1=[$5], sort2=[$4], dir0=[DESC], dir1=[ASC],
> dir2=[ASC])
> HiveProject(total_sum=[$4], s_state=[$0], s_county=[$1],
> lochierarchy=[$5], rank_within_parent=[rank() OVER (PARTITION BY $5,
> when(==($5, 2), $0) ORDER BY $4 ROWS BETWEEN 2147483647 FOLLOWING AND
> 2147483647 PRECEDING)], (tok_function when (= (tok_table_or_col lochierarchy)
> 0) (tok_table_or_col s_state))=[when(=($5, 0), $0)])
> HiveAggregate(group=[{0, 1}], groups=[[{0, 1}, {0}, {}]],
> indicator=[true], agg#0=[sum($2)], GROUPING__ID=[GROUPING__ID()])
> HiveProject($f0=[$7], $f1=[$6], $f2=[$1])
> HiveJoin(condition=[=($5, $2)], joinType=[inner], algorithm=[none],
> cost=[{1177.2086187101072 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($3, $0)], joinType=[inner],
> algorithm=[none], cost=[{2880430.428726483 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_net_profit=[$21],
> ss_store_sk=[$22])
> HiveTableScan(table=[[tpcds.store_sales]])
> HiveProject(d_date_sk=[$0], d_month_seq=[$3])
> HiveFilter(condition=[between(false, $3, 1193, +(1193, 11))])
> HiveTableScan(table=[[tpcds.date_dim]])
> HiveProject(s_store_sk=[$0], s_county=[$1], s_state=[$2])
> SemiJoin(condition=[=($2, $3)], joinType=[inner])
> HiveProject(s_store_sk=[$0], s_county=[$23], s_state=[$24])
> HiveTableScan(table=[[tpcds.store]])
> HiveProject(s_state=[$0])
> HiveFilter(condition=[<=($1, 5)])
>
[jira] [Commented] (HIVE-9152) Dynamic Partition Pruning [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541190#comment-14541190
]
Hive QA commented on HIVE-9152:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12732438/HIVE-9152.8-spark.patch
{color:red}ERROR:{color} -1 due to 40 failed/errored test(s), 8709 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-update_orig_table.q-sort_merge_join_desc_1.q-order.q-and-12-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-bucket6.q-scriptfile1_win.q-quotedid_smb.q-and-1-more - did
not produce a TEST-*.xml file
TestMinimrCliDriver-bucketizedhiveinputformat.q-empty_dir_in_table.q - did not
produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-infer_bucket_sort_map_operators.q-load_hdfs_file_with_space_in_the_name.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-import_exported_table.q-truncate_column_buckets.q-bucket_num_reducers2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-infer_bucket_sort_num_buckets.q-parallel_orderby.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-join1.q-infer_bucket_sort_bucketed_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-bucket5.q-infer_bucket_sort_merge.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-input16_cc.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-bucket_num_reducers.q-scriptfile1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx_cbo_2.q-bucketmapjoin6.q-bucket4.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-reduce_deduplicate.q-infer_bucket_sort_dyn_part.q-udf_using.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-uber_reduce.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-stats_counter_partitioned.q-external_table_with_space_in_location_path.q-disable_merge_for_bucketing.q-and-1-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_partition_metadataonly
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_pcr
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_example_add
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_in_file
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_view
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_elt
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_string_concat
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_decimal_date
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_div0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_case
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_math_funcs
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_string_funcs
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/851/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/851/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-851/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 40 tests failed
{noformat}
This message is automatically generated.
A
[jira] [Commented] (HIVE-10679) JsonSerde ignores varchar and char size limit specified during table creation
[
https://issues.apache.org/jira/browse/HIVE-10679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541166#comment-14541166
]
Hive QA commented on HIVE-10679:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12732359/HIVE-10679.2.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 8922 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3866/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3866/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3866/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12732359 - PreCommit-HIVE-TRUNK-Build
> JsonSerde ignores varchar and char size limit specified during table creation
> -
>
> Key: HIVE-10679
> URL: https://issues.apache.org/jira/browse/HIVE-10679
> Project: Hive
> Issue Type: Bug
>Reporter: Jagruti Varia
>Assignee: Jason Dere
> Attachments: HIVE-10679.1.patch, HIVE-10679.2.patch
>
>
> JsonSerde ignores varchar and char size limit specified during table creation
> and always creates varchar or char column with max length.
> steps to reproduce the issue:
> {noformat}
> create table jsonserde_1 (v varchar(50), c char(50)) row format serde
> 'org.apache.hive.hcatalog.data.JsonSerDe';
> desc jsonserde_1;
> OK
> v varchar(65535) from deserializer
> c char(255) from deserializer
> Time taken: 0.468 seconds, Fetched: 2 row(s)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10319) Hive CLI startup takes a long time with a large number of databases
[ https://issues.apache.org/jira/browse/HIVE-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541160#comment-14541160 ] Jason Dere commented on HIVE-10319: --- An annoying thing about RB is that it hangs if you create your diff with --no-prefix (I know the Hive guidelines say to use that), it actually works if you omit that option when creating the diff. > Hive CLI startup takes a long time with a large number of databases > --- > > Key: HIVE-10319 > URL: https://issues.apache.org/jira/browse/HIVE-10319 > Project: Hive > Issue Type: Improvement > Components: CLI >Affects Versions: 1.0.0 >Reporter: Nezih Yigitbasi >Assignee: Nezih Yigitbasi > Attachments: HIVE-10319-no-gen.patch, HIVE-10319.1.patch, > HIVE-10319.patch > > > The Hive CLI takes a long time to start when there is a large number of > databases in the DW. I think the root cause is the way permanent UDFs are > loaded from the metastore. When I looked at the logs and the source code I > see that at startup Hive first gets all the databases from the metastore and > then for each database it makes a metastore call to get the permanent > functions for that database [see Hive.java | > https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L162-185]. > So the number of metastore calls made is in the order of the number of > databases. In production we have several hundreds of databases so Hive makes > several hundreds of RPC calls during startup, taking 30+ seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10675) Provide option to skip Accumulo related Hive tests in itests directory
[ https://issues.apache.org/jira/browse/HIVE-10675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541112#comment-14541112 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-10675: -- The test failures are unrelated to the fix. Thanks Hari > Provide option to skip Accumulo related Hive tests in itests directory > -- > > Key: HIVE-10675 > URL: https://issues.apache.org/jira/browse/HIVE-10675 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-10675.1.patch, HIVE-10675.2.patch > > > We shouldn’t be running Accumulo+Hive tests in Windows as Accumulo is > currently unsupported on Windows. Hence we should provide an option to > disable these tests; that being mentioned, the default behavior would be to > enable these tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9152) Dynamic Partition Pruning [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-9152: --- Attachment: HIVE-9152.8-spark.patch > Dynamic Partition Pruning [Spark Branch] > > > Key: HIVE-9152 > URL: https://issues.apache.org/jira/browse/HIVE-9152 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Brock Noland >Assignee: Chao Sun > Attachments: HIVE-9152.1-spark.patch, HIVE-9152.2-spark.patch, > HIVE-9152.3-spark.patch, HIVE-9152.4-spark.patch, HIVE-9152.5-spark.patch, > HIVE-9152.6-spark.patch, HIVE-9152.8-spark.patch > > > Tez implemented dynamic partition pruning in HIVE-7826. This is a nice > optimization and we should implement the same in HOS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10697) ObjecInspectorConvertors#UnionConvertor does a faulty conversion
[
https://issues.apache.org/jira/browse/HIVE-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swarnim Kulkarni updated HIVE-10697:
Description:
Currently the UnionConvertor in the ObjectInspectorConvertors class has an
issue with the convert method where it attempts to convert the objectinspector
itself instead of converting the field.[1]. This should be changed to convert
the field itself. This could result in a ClassCastException as shown below:
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyUnionObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.lazy.LazyString
at
org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector.getPrimitiveWritableObject(LazyStringObjectInspector.java:51)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:391)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:338)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$UnionConverter.convert(ObjectInspectorConverters.java:456)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:395)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$MapConverter.convert(ObjectInspectorConverters.java:539)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:395)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:154)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:127)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518)
... 9 more
{code}
[1]
https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java#L466
was:
Currently the UnionConvertor in the ObjectInspectorConvertors class has an
issue with the convert method where it attempts to convert the objectinspector
itself instead of converting the field.[1]. This should be changed to convert
the field itself. This could result in a ClassCastException as shown below:
{quote}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyUnionObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.lazy.LazyString
at
org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector.getPrimitiveWritableObject(LazyStringObjectInspector.java:51)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:391)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:338)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$UnionConverter.convert(ObjectInspectorConverters.java:456)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:395)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$MapConverter.convert(ObjectInspectorConverters.java:539)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:395)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:154)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:127)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518)
... 9 more
{quote}
[1]
https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java#L466
> ObjecInspectorConvertors#UnionConvertor does a faulty conversion
>
>
> Key: HIVE-10697
> URL: https://issues.apache.org/jira/browse/HIVE-10697
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
>
> Currently the UnionConvertor in the ObjectInspectorConvertors class has an
> issue with the convert method where it attempts to convert the
> objectinspector itself instead of converting the field.[1]. This should be
> changed to convert the field itself. This could result in a
> ClassCastExcep
[jira] [Updated] (HIVE-10636) CASE comparison operator rotation optimization
[ https://issues.apache.org/jira/browse/HIVE-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-10636: Attachment: HIVE-10636.2.patch Rebased patch after HIVE-10325 commit, includes HIVE-10327 > CASE comparison operator rotation optimization > -- > > Key: HIVE-10636 > URL: https://issues.apache.org/jira/browse/HIVE-10636 > Project: Hive > Issue Type: New Feature > Components: Logical Optimizer >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-10636.1.patch, HIVE-10636.2.patch, HIVE-10636.patch > > > Step 1 as outlined in description of HIVE-9644 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10697) ObjecInspectorConvertors#UnionConvertor does a faulty conversion
[
https://issues.apache.org/jira/browse/HIVE-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swarnim Kulkarni updated HIVE-10697:
Description:
Currently the UnionConvertor in the ObjectInspectorConvertors class has an
issue with the convert method where it attempts to convert the objectinspector
itself instead of converting the field.[1]. This should be changed to convert
the field itself. This could result in a ClassCastException as shown below:
{quote}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyUnionObjectInspector
cannot be cast to org.apache.hadoop.hive.serde2.lazy.LazyString
at
org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector.getPrimitiveWritableObject(LazyStringObjectInspector.java:51)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:391)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:338)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$UnionConverter.convert(ObjectInspectorConverters.java:456)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:395)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$MapConverter.convert(ObjectInspectorConverters.java:539)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:395)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:154)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:127)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518)
... 9 more
{quote}
[1]
https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java#L466
was:
Currently the UnionConvertor in the ObjectInspectorConvertors class has an
issue with the convert method where it attempts to convert the objectinspector
itself instead of converting the field.[1]. This should be changed to convert
the field itself. This could result in a ClassCastException as shown below:
{quote}
{quote}
[1]
https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java#L466
> ObjecInspectorConvertors#UnionConvertor does a faulty conversion
>
>
> Key: HIVE-10697
> URL: https://issues.apache.org/jira/browse/HIVE-10697
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
>
> Currently the UnionConvertor in the ObjectInspectorConvertors class has an
> issue with the convert method where it attempts to convert the
> objectinspector itself instead of converting the field.[1]. This should be
> changed to convert the field itself. This could result in a
> ClassCastException as shown below:
> {quote}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyUnionObjectInspector
> cannot be cast to org.apache.hadoop.hive.serde2.lazy.LazyString
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector.getPrimitiveWritableObject(LazyStringObjectInspector.java:51)
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:391)
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:338)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$UnionConverter.convert(ObjectInspectorConverters.java:456)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:395)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$MapConverter.convert(ObjectInspectorConverters.java:539)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:395)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:154)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:127)
> at
> org.apache.hadoop.hive.ql.exec.M
[jira] [Updated] (HIVE-8769) Physical optimizer : Incorrect CE results in a shuffle join instead of a Map join (PK/FK pattern not detected)
[
https://issues.apache.org/jira/browse/HIVE-8769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-8769:
--
Attachment: HIVE-8769.01.patch
> Physical optimizer : Incorrect CE results in a shuffle join instead of a Map
> join (PK/FK pattern not detected)
> --
>
> Key: HIVE-8769
> URL: https://issues.apache.org/jira/browse/HIVE-8769
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Pengcheng Xiong
> Attachments: HIVE-8769.01.patch
>
>
> TPC-DS Q82 is running slower than hive 13 because the join type is not
> correct.
> The estimate for item x inventory x date_dim is 227 Million rows while the
> actual is 3K rows.
> Hive 13 finishes in 753 seconds.
> Hive 14 finishes in 1,267 seconds.
> Hive 14 + force map join finished in 431 seconds.
> Query
> {code}
> select i_item_id
>,i_item_desc
>,i_current_price
> from item, inventory, date_dim, store_sales
> where i_current_price between 30 and 30+30
> and inv_item_sk = i_item_sk
> and d_date_sk=inv_date_sk
> and d_date between '2002-05-30' and '2002-07-30'
> and i_manufact_id in (437,129,727,663)
> and inv_quantity_on_hand between 100 and 500
> and ss_item_sk = i_item_sk
> group by i_item_id,i_item_desc,i_current_price
> order by i_item_id
> limit 100
> {code}
> Plan
> {code}
> STAGE PLANS:
> Stage: Stage-1
> Tez
> Edges:
> Map 7 <- Map 1 (BROADCAST_EDGE), Map 2 (BROADCAST_EDGE)
> Reducer 4 <- Map 3 (SIMPLE_EDGE), Map 7 (SIMPLE_EDGE)
> Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
> Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
> DagName: mmokhtar_20141106005353_7a2eb8df-12ff-4fe9-89b4-30f1e4e3fb90:1
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: item
> filterExpr: ((i_current_price BETWEEN 30 AND 60 and
> (i_manufact_id) IN (437, 129, 727, 663)) and i_item_sk is not null) (type:
> boolean)
> Statistics: Num rows: 462000 Data size: 663862160 Basic
> stats: COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: ((i_current_price BETWEEN 30 AND 60 and
> (i_manufact_id) IN (437, 129, 727, 663)) and i_item_sk is not null) (type:
> boolean)
> Statistics: Num rows: 115500 Data size: 34185680 Basic
> stats: COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: i_item_sk (type: int), i_item_id (type:
> string), i_item_desc (type: string), i_current_price (type: float)
> outputColumnNames: _col0, _col1, _col2, _col3
> Statistics: Num rows: 115500 Data size: 33724832 Basic
> stats: COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: int)
> sort order: +
> Map-reduce partition columns: _col0 (type: int)
> Statistics: Num rows: 115500 Data size: 33724832
> Basic stats: COMPLETE Column stats: COMPLETE
> value expressions: _col1 (type: string), _col2 (type:
> string), _col3 (type: float)
> Execution mode: vectorized
> Map 2
> Map Operator Tree:
> TableScan
> alias: date_dim
> filterExpr: (d_date BETWEEN '2002-05-30' AND '2002-07-30'
> and d_date_sk is not null) (type: boolean)
> Statistics: Num rows: 73049 Data size: 81741831 Basic
> stats: COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: (d_date BETWEEN '2002-05-30' AND '2002-07-30'
> and d_date_sk is not null) (type: boolean)
> Statistics: Num rows: 36524 Data size: 3579352 Basic
> stats: COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: d_date_sk (type: int)
> outputColumnNames: _col0
> Statistics: Num rows: 36524 Data size: 146096 Basic
> stats: COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: int)
> sort order: +
> Map-reduce partition columns: _col0 (type: int)
> Statistics: Num rows: 36524 Data size: 146096 Basic
> stats: COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: _col0 (type: int)
> outputColumnNames: _co
[jira] [Updated] (HIVE-10697) ObjecInspectorConvertors#UnionConvertor does a faulty conversion
[
https://issues.apache.org/jira/browse/HIVE-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swarnim Kulkarni updated HIVE-10697:
Description:
Currently the UnionConvertor in the ObjectInspectorConvertors class has an
issue with the convert method where it attempts to convert the objectinspector
itself instead of converting the field.[1]. This should be changed to convert
the field itself. This could result in a ClassCastException as shown below:
{quote}
{quote}
[1]
https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java#L466
was:
Currently the UnionConvertor in the ObjectInspectorConvertors class has an
issue with the convert method where it attempts to convert the objectinspector
itself instead of converting the field.[1] This should be changed to convert
the field itself.
[1]
https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java#L466
> ObjecInspectorConvertors#UnionConvertor does a faulty conversion
>
>
> Key: HIVE-10697
> URL: https://issues.apache.org/jira/browse/HIVE-10697
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
>
> Currently the UnionConvertor in the ObjectInspectorConvertors class has an
> issue with the convert method where it attempts to convert the
> objectinspector itself instead of converting the field.[1]. This should be
> changed to convert the field itself. This could result in a
> ClassCastException as shown below:
> {quote}
> {quote}
> [1]
> https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java#L466
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10319) Hive CLI startup takes a long time with a large number of databases
[
https://issues.apache.org/jira/browse/HIVE-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nezih Yigitbasi updated HIVE-10319:
---
Attachment: HIVE-10319-no-gen.patch
[~jdere] I rebased the patch, can you please retry? Btw, I am having a hard
time creating a review on review board, it just hangs (even without the
generated sources) and doesn't create a review. Anyway, attaching this new file
("HIVE-10319-no-gen.patch") for review purposes.
> Hive CLI startup takes a long time with a large number of databases
> ---
>
> Key: HIVE-10319
> URL: https://issues.apache.org/jira/browse/HIVE-10319
> Project: Hive
> Issue Type: Improvement
> Components: CLI
>Affects Versions: 1.0.0
>Reporter: Nezih Yigitbasi
>Assignee: Nezih Yigitbasi
> Attachments: HIVE-10319-no-gen.patch, HIVE-10319.1.patch,
> HIVE-10319.patch
>
>
> The Hive CLI takes a long time to start when there is a large number of
> databases in the DW. I think the root cause is the way permanent UDFs are
> loaded from the metastore. When I looked at the logs and the source code I
> see that at startup Hive first gets all the databases from the metastore and
> then for each database it makes a metastore call to get the permanent
> functions for that database [see Hive.java |
> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L162-185].
> So the number of metastore calls made is in the order of the number of
> databases. In production we have several hundreds of databases so Hive makes
> several hundreds of RPC calls during startup, taking 30+ seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10565) LLAP: Native Vector Map Join doesn't handle filtering and matching on LEFT OUTER JOIN repeated key correctly
[ https://issues.apache.org/jira/browse/HIVE-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541078#comment-14541078 ] Gunther Hagleitner commented on HIVE-10565: --- +1 lgtm. can you comment on the test failures though? > LLAP: Native Vector Map Join doesn't handle filtering and matching on LEFT > OUTER JOIN repeated key correctly > > > Key: HIVE-10565 > URL: https://issues.apache.org/jira/browse/HIVE-10565 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 1.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 1.3.0 > > Attachments: HIVE-10565.01.patch, HIVE-10565.02.patch, > HIVE-10565.03.patch, HIVE-10565.04.patch, HIVE-10565.05.patch, > HIVE-10565.06.patch, HIVE-10565.07.patch, HIVE-10565.08.patch, > HIVE-10565.09.patch, HIVE-10565.091.patch, HIVE-10565.092.patch > > > Filtering can knock out some of the rows for a repeated key, but those > knocked out rows need to be included in the LEFT OUTER JOIN result and are > currently not when only some rows are filtered out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10696) TestAddResource tests are non-portable
[ https://issues.apache.org/jira/browse/HIVE-10696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-10696: - Attachment: HIVE-10696.1.patch [~thejas] can you please review the fix. Thanks Hari > TestAddResource tests are non-portable > -- > > Key: HIVE-10696 > URL: https://issues.apache.org/jira/browse/HIVE-10696 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-10696.1.patch > > > We need to make sure these tests work in windows as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10101) LLAP: enable yourkit profiling of tasks
[ https://issues.apache.org/jira/browse/HIVE-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541052#comment-14541052 ] Mostafa Mokhtar commented on HIVE-10101: [~sershe] I would rather use JMC which is already part of the JVM since Java 7. http://www.oracle.com/technetwork/java/javaseproducts/mission-control/java-mission-control-1998576.html > LLAP: enable yourkit profiling of tasks > --- > > Key: HIVE-10101 > URL: https://issues.apache.org/jira/browse/HIVE-10101 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-10101.02.patch, HIVE-10101.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10652) LLAP: AM task communication retry is too long
[ https://issues.apache.org/jira/browse/HIVE-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-10652: -- Attachment: HIVE-10652.1.addendum.txt Addendum patch to remove some unused imports. > LLAP: AM task communication retry is too long > - > > Key: HIVE-10652 > URL: https://issues.apache.org/jira/browse/HIVE-10652 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-10652.1.addendum.txt, HIVE-10652.1.txt > > > Mentioned by [~sseth] while discussing HIVE-10648. 45sec (or whatever) is a > bit too long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10675) Provide option to skip Accumulo related Hive tests in itests directory
[
https://issues.apache.org/jira/browse/HIVE-10675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541042#comment-14541042
]
Hive QA commented on HIVE-10675:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12732353/HIVE-10675.2.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 8919 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3865/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3865/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3865/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12732353 - PreCommit-HIVE-TRUNK-Build
> Provide option to skip Accumulo related Hive tests in itests directory
> --
>
> Key: HIVE-10675
> URL: https://issues.apache.org/jira/browse/HIVE-10675
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-10675.1.patch, HIVE-10675.2.patch
>
>
> We shouldn’t be running Accumulo+Hive tests in Windows as Accumulo is
> currently unsupported on Windows. Hence we should provide an option to
> disable these tests; that being mentioned, the default behavior would be to
> enable these tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10695) Hive Query Produces Wrong Result: PPD
[ https://issues.apache.org/jira/browse/HIVE-10695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541035#comment-14541035 ] Laljo John Pullokkaran commented on HIVE-10695: --- This is duplicate of HIVE-9613 > Hive Query Produces Wrong Result: PPD > - > > Key: HIVE-10695 > URL: https://issues.apache.org/jira/browse/HIVE-10695 > Project: Hive > Issue Type: Bug >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran >Priority: Critical > Fix For: 0.14.1 > > Attachments: HIVE-10695.patch > > > Following query produces wrong result: > select * from t1 s left outer join (select key, value from t1) f on > s.key=f.key and s.value=f.value left outer join (select key, value from > t1) c on s.key=c.key where f.key is null; > This is due to PPD getting confused between qualified col name & non > qualified. > In many places in code column info doesn't include table alias which leads to > PPD problem. > This is fixed in trunk as part of HIVE-9327 > https://issues.apache.org/jira/browse/HIVE-9327 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10649) LLAP: AM gets stuck completely if one node is dead
[
https://issues.apache.org/jira/browse/HIVE-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated HIVE-10649:
--
Assignee: (was: Siddharth Seth)
> LLAP: AM gets stuck completely if one node is dead
> --
>
> Key: HIVE-10649
> URL: https://issues.apache.org/jira/browse/HIVE-10649
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>
> See HIVE-10648.
> When AM cannot connect to a node, that appears to cause it to stall; example
> log, there are no other interleaving logs even though this is happening in
> the middle of Map 1 on TPCH q1, i.e. there are plenty of tasks scheduled.
> From "Assigning" messages I can also see tasks are scheduled to all the nodes
> before and after the pause, not just to the problematic node.
> LLAP daemons have corresponding gaps where between two fragments nothing is
> ran for a long time on any daemon.
> {noformat}
> 2015-05-07 12:13:46,679 INFO [Dispatcher thread: Central] impl.TaskImpl:
> task_1429683757595_0784_1_00_000276 Task Transitioned from SCHEDULED to
> RUNNING due to event T_ATTEMPT_LAUNCHED
> 2015-05-07 12:13:46,811 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 10 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:46,955 INFO [LlapSchedulerNodeEnabler]
> impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 1611673583
> 2015-05-07 12:13:47,811 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 11 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:48,812 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 12 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:49,813 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 13 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:50,813 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 14 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:51,814 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 15 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:52,814 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 16 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:53,815 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 17 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:54,816 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 18 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:55,816 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 19 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:56,817 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 20 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:56,971 INFO [LlapSchedulerNodeEnabler]
> impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 1611673583
> 2015-05-07 12:13:57,817 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 21 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:58,818 INFO [TaskCommunicator # 3] ipc.Client: Retrying
[jira] [Resolved] (HIVE-10649) LLAP: AM gets stuck completely if one node is dead
[
https://issues.apache.org/jira/browse/HIVE-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth resolved HIVE-10649.
---
Resolution: Duplicate
> LLAP: AM gets stuck completely if one node is dead
> --
>
> Key: HIVE-10649
> URL: https://issues.apache.org/jira/browse/HIVE-10649
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Siddharth Seth
>
> See HIVE-10648.
> When AM cannot connect to a node, that appears to cause it to stall; example
> log, there are no other interleaving logs even though this is happening in
> the middle of Map 1 on TPCH q1, i.e. there are plenty of tasks scheduled.
> From "Assigning" messages I can also see tasks are scheduled to all the nodes
> before and after the pause, not just to the problematic node.
> LLAP daemons have corresponding gaps where between two fragments nothing is
> ran for a long time on any daemon.
> {noformat}
> 2015-05-07 12:13:46,679 INFO [Dispatcher thread: Central] impl.TaskImpl:
> task_1429683757595_0784_1_00_000276 Task Transitioned from SCHEDULED to
> RUNNING due to event T_ATTEMPT_LAUNCHED
> 2015-05-07 12:13:46,811 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 10 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:46,955 INFO [LlapSchedulerNodeEnabler]
> impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 1611673583
> 2015-05-07 12:13:47,811 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 11 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:48,812 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 12 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:49,813 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 13 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:50,813 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 14 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:51,814 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 15 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:52,814 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 16 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:53,815 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 17 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:54,816 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 18 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:55,816 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 19 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:56,817 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 20 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:56,971 INFO [LlapSchedulerNodeEnabler]
> impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 1611673583
> 2015-05-07 12:13:57,817 INFO [TaskCommunicator # 3] ipc.Client: Retrying
> connect to server: cn059-10.l42scl.hortonworks.com/172.19.128.59:15001.
> Already tried 21 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)
> 2015-05-07 12:13:58,818 INFO [TaskCommunicator #
[jira] [Resolved] (HIVE-10652) LLAP: AM task communication retry is too long
[ https://issues.apache.org/jira/browse/HIVE-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-10652. --- Resolution: Fixed Fix Version/s: llap > LLAP: AM task communication retry is too long > - > > Key: HIVE-10652 > URL: https://issues.apache.org/jira/browse/HIVE-10652 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-10652.1.txt > > > Mentioned by [~sseth] while discussing HIVE-10648. 45sec (or whatever) is a > bit too long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10652) LLAP: AM task communication retry is too long
[ https://issues.apache.org/jira/browse/HIVE-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-10652: -- Attachment: HIVE-10652.1.txt Made configurable, and defaults to 16s. > LLAP: AM task communication retry is too long > - > > Key: HIVE-10652 > URL: https://issues.apache.org/jira/browse/HIVE-10652 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Siddharth Seth > Attachments: HIVE-10652.1.txt > > > Mentioned by [~sseth] while discussing HIVE-10648. 45sec (or whatever) is a > bit too long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10695) Hive Query Produces Wrong Result: PPD
[ https://issues.apache.org/jira/browse/HIVE-10695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-10695: -- Attachment: HIVE-10695.patch > Hive Query Produces Wrong Result: PPD > - > > Key: HIVE-10695 > URL: https://issues.apache.org/jira/browse/HIVE-10695 > Project: Hive > Issue Type: Bug >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran >Priority: Critical > Fix For: 0.14.1 > > Attachments: HIVE-10695.patch > > > Following query produces wrong result: > select * from t1 s left outer join (select key, value from t1) f on > s.key=f.key and s.value=f.value left outer join (select key, value from > t1) c on s.key=c.key where f.key is null; > This is due to PPD getting confused between qualified col name & non > qualified. > In many places in code column info doesn't include table alias which leads to > PPD problem. > This is fixed in trunk as part of HIVE-9327 > https://issues.apache.org/jira/browse/HIVE-9327 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10319) Hive CLI startup takes a long time with a large number of databases
[ https://issues.apache.org/jira/browse/HIVE-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih Yigitbasi updated HIVE-10319: --- Attachment: HIVE-10319.1.patch > Hive CLI startup takes a long time with a large number of databases > --- > > Key: HIVE-10319 > URL: https://issues.apache.org/jira/browse/HIVE-10319 > Project: Hive > Issue Type: Improvement > Components: CLI >Affects Versions: 1.0.0 >Reporter: Nezih Yigitbasi >Assignee: Nezih Yigitbasi > Attachments: HIVE-10319.1.patch, HIVE-10319.patch > > > The Hive CLI takes a long time to start when there is a large number of > databases in the DW. I think the root cause is the way permanent UDFs are > loaded from the metastore. When I looked at the logs and the source code I > see that at startup Hive first gets all the databases from the metastore and > then for each database it makes a metastore call to get the permanent > functions for that database [see Hive.java | > https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L162-185]. > So the number of metastore calls made is in the order of the number of > databases. In production we have several hundreds of databases so Hive makes > several hundreds of RPC calls during startup, taking 30+ seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10695) Hive Query Produces Wrong Result: PPD
[ https://issues.apache.org/jira/browse/HIVE-10695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-10695: -- Description: Following query produces wrong result: select * from t1 s left outer join (select key, value from t1) f on s.key=f.key and s.value=f.value left outer join (select key, value from t1) c on s.key=c.key where f.key is null; This is due to PPD getting confused between qualified col name & non qualified. In many places in code column info doesn't include table alias which leads to PPD problem. This is fixed in trunk as part of HIVE-9327 https://issues.apache.org/jira/browse/HIVE-9327 was: Following query produces wrong result: select * from t1 s left outer join (select key, value from t1) f on s.key=f.key and s.value=f.value left outer join (select key, value from t1) c on s.key=c.key where f.key is null; This is due to PPD gets confused between qualified col name & non qualified. In many places in code column info doesn't include table alias which leads to PPD problem. This is fixed in trunk as part of HIVE-9327 https://issues.apache.org/jira/browse/HIVE-9327 > Hive Query Produces Wrong Result: PPD > - > > Key: HIVE-10695 > URL: https://issues.apache.org/jira/browse/HIVE-10695 > Project: Hive > Issue Type: Bug >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran >Priority: Critical > Fix For: 0.14.1 > > > Following query produces wrong result: > select * from t1 s left outer join (select key, value from t1) f on > s.key=f.key and s.value=f.value left outer join (select key, value from > t1) c on s.key=c.key where f.key is null; > This is due to PPD getting confused between qualified col name & non > qualified. > In many places in code column info doesn't include table alias which leads to > PPD problem. > This is fixed in trunk as part of HIVE-9327 > https://issues.apache.org/jira/browse/HIVE-9327 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10563) MiniTezCliDriver tests ordering issues
[ https://issues.apache.org/jira/browse/HIVE-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-10563: - Attachment: HIVE-10563.5.patch > MiniTezCliDriver tests ordering issues > -- > > Key: HIVE-10563 > URL: https://issues.apache.org/jira/browse/HIVE-10563 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-10563.1.patch, HIVE-10563.2.patch, > HIVE-10563.3.patch, HIVE-10563.4.patch, HIVE-10563.5.patch > > > There are a bunch of tests related to TestMiniTezCliDriver which gives > ordering issues when run on Centos/Windows/OSX -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10563) MiniTezCliDriver tests ordering issues
[ https://issues.apache.org/jira/browse/HIVE-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-10563: - Attachment: (was: HIVE-10563.5.patch) > MiniTezCliDriver tests ordering issues > -- > > Key: HIVE-10563 > URL: https://issues.apache.org/jira/browse/HIVE-10563 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-10563.1.patch, HIVE-10563.2.patch, > HIVE-10563.3.patch, HIVE-10563.4.patch, HIVE-10563.5.patch > > > There are a bunch of tests related to TestMiniTezCliDriver which gives > ordering issues when run on Centos/Windows/OSX -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10559) IndexOutOfBoundsException with RemoveDynamicPruningBySize
[
https://issues.apache.org/jira/browse/HIVE-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540912#comment-14540912
]
Gunther Hagleitner commented on HIVE-10559:
---
+1
> IndexOutOfBoundsException with RemoveDynamicPruningBySize
> -
>
> Key: HIVE-10559
> URL: https://issues.apache.org/jira/browse/HIVE-10559
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-10559.01.patch, HIVE-10559.02.patch,
> HIVE-10559.03.patch, q85.q
>
>
> The problem can be reproduced by running the script attached.
> Backtrace
> {code}
> 2015-04-29 10:34:36,390 ERROR [main]: ql.Driver
> (SessionState.java:printError(956)) - FAILED: IndexOutOfBoundsException
> Index: 0, Size: 0
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.RemoveDynamicPruningBySize.process(RemoveDynamicPruningBySize.java:61)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
> at
> org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:77)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.runStatsDependentOptimizations(TezCompiler.java:281)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:123)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:102)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10092)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9932)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1026)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1000)
> at
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.runTest(TestMiniTezCliDriver.java:139)
> at
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_q85(TestMiniTezCliDriver.java:123)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4
[jira] [Commented] (HIVE-10690) ArrayIndexOutOfBounds exception in MetaStoreDirectSql.aggrColStatsForPartitions()
[
https://issues.apache.org/jira/browse/HIVE-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540897#comment-14540897
]
Vaibhav Gumashta commented on HIVE-10690:
-
Test failure not related.
> ArrayIndexOutOfBounds exception in
> MetaStoreDirectSql.aggrColStatsForPartitions()
> -
>
> Key: HIVE-10690
> URL: https://issues.apache.org/jira/browse/HIVE-10690
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.0
>Reporter: Jason Dere
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-10690.1.patch
>
>
> Noticed a bunch of these stack traces in hive.log while running some unit
> tests:
> {noformat}
> 2015-05-11 21:18:59,371 WARN [main]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(2420)) - Direct SQL failed
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1132)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6162)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6158)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2385)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy84.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:5662)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy86.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAggrColStatsFor(HiveMetaStoreClient.java:2064)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy87.getAggrColStatsFor(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAggrColStatsFor(Hive.java:3110)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:245)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:329)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:399)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:392)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:150)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:77)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:64)
> at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke(ReflectiveRelMetadataProvider.java:182)
> at com.sun.proxy.$Proxy108.getDistinctRowCount(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethod
[jira] [Commented] (HIVE-10690) ArrayIndexOutOfBounds exception in MetaStoreDirectSql.aggrColStatsForPartitions()
[
https://issues.apache.org/jira/browse/HIVE-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540895#comment-14540895
]
Hive QA commented on HIVE-10690:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12732341/HIVE-10690.1.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 8921 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.testReadDataFromEncryptedHiveTableByPig[1]
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3864/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3864/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3864/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12732341 - PreCommit-HIVE-TRUNK-Build
> ArrayIndexOutOfBounds exception in
> MetaStoreDirectSql.aggrColStatsForPartitions()
> -
>
> Key: HIVE-10690
> URL: https://issues.apache.org/jira/browse/HIVE-10690
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.0
>Reporter: Jason Dere
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-10690.1.patch
>
>
> Noticed a bunch of these stack traces in hive.log while running some unit
> tests:
> {noformat}
> 2015-05-11 21:18:59,371 WARN [main]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(2420)) - Direct SQL failed
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1132)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6162)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6158)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2385)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy84.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:5662)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy86.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAggrColStatsFor(HiveMetaStoreClient.java:2064)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy87.getAggrColStatsFor(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAggrColStatsFor(Hive.java:3110)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:245)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updat
[jira] [Updated] (HIVE-10641) create CRC32 UDF
[ https://issues.apache.org/jira/browse/HIVE-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-10641: --- Attachment: HIVE-10641.2.patch patch #2: - use Text.getBytes() instead of toString() > create CRC32 UDF > > > Key: HIVE-10641 > URL: https://issues.apache.org/jira/browse/HIVE-10641 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Alexander Pivovarov >Assignee: Alexander Pivovarov > Attachments: HIVE-10641.1.patch, HIVE-10641.2.patch > > > CRC32 computes a cyclic redundancy check value for string or binary argument > and returns bigint value. The result is NULL if the argument is NULL. > MySQL has similar function > https://dev.mysql.com/doc/refman/5.0/en/mathematical-functions.html#function_crc32 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10683) LLAP: Add a mechanism for daemons to inform the AM about killed tasks
[ https://issues.apache.org/jira/browse/HIVE-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-10683. --- Resolution: Fixed > LLAP: Add a mechanism for daemons to inform the AM about killed tasks > - > > Key: HIVE-10683 > URL: https://issues.apache.org/jira/browse/HIVE-10683 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-10683.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10683) LLAP: Add a mechanism for daemons to inform the AM about killed tasks
[ https://issues.apache.org/jira/browse/HIVE-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-10683: -- Attachment: HIVE-10683.1.txt > LLAP: Add a mechanism for daemons to inform the AM about killed tasks > - > > Key: HIVE-10683 > URL: https://issues.apache.org/jira/browse/HIVE-10683 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-10683.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10639) create SHA1 UDF
[ https://issues.apache.org/jira/browse/HIVE-10639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-10639: --- Attachment: HIVE-10639.3.patch patch#3: - use Text.getBytes() instead of toString() > create SHA1 UDF > --- > > Key: HIVE-10639 > URL: https://issues.apache.org/jira/browse/HIVE-10639 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Alexander Pivovarov >Assignee: Alexander Pivovarov > Attachments: HIVE-10639.1.patch, HIVE-10639.2.patch, > HIVE-10639.3.patch > > > Calculates an SHA-1 160-bit checksum for the string and binary, as described > in RFC 3174 (Secure Hash Algorithm). The value is returned as a string of 40 > hex digits, or NULL if the argument was NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10639) create SHA1 UDF
[ https://issues.apache.org/jira/browse/HIVE-10639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540869#comment-14540869 ] Jason Dere commented on HIVE-10639: --- I think this looks good - can you use Text.getBytes() rather than toString() like you did with HIVE-10657, assuming this doesn't change the test results? > create SHA1 UDF > --- > > Key: HIVE-10639 > URL: https://issues.apache.org/jira/browse/HIVE-10639 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Alexander Pivovarov >Assignee: Alexander Pivovarov > Attachments: HIVE-10639.1.patch, HIVE-10639.2.patch > > > Calculates an SHA-1 160-bit checksum for the string and binary, as described > in RFC 3174 (Secure Hash Algorithm). The value is returned as a string of 40 > hex digits, or NULL if the argument was NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10657) Remove copyBytes operation from MD5 UDF
[ https://issues.apache.org/jira/browse/HIVE-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-10657: --- Attachment: HIVE-10657.2.patch patch #2: - use Text.getBytes() instead of toString() > Remove copyBytes operation from MD5 UDF > --- > > Key: HIVE-10657 > URL: https://issues.apache.org/jira/browse/HIVE-10657 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Alexander Pivovarov >Assignee: Alexander Pivovarov >Priority: Minor > Attachments: HIVE-10657.1.patch, HIVE-10657.2.patch > > > Current MD5 UDF implementation uses Apache Commons DigestUtils.md5Hex method > to get md5 hex. > DigestUtils does not provide md5Hex method with signature (byte[], start, > length). This is why copyBytes method was added to UDFMd5 to get bytes[] from > BytesWritable. > To avoid copying bytes from BytesWritable to new byte array we can use java > MessageDigest API directly. > MessageDigest has method update(byte[], start, length) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10319) Hive CLI startup takes a long time with a large number of databases
[ https://issues.apache.org/jira/browse/HIVE-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540846#comment-14540846 ] Jason Dere commented on HIVE-10319: --- Last patch didn't appear to apply properly .. would you be able to update the patch? Also would you be able to create a RB entry for this? Wow, there are a lot of auto-generated files in the diff, probably because you are the first person to use thrift-0.9.2 to generate the files :) If you do create a RB can you exclude the src/gen/thrift/ files from the diff used in the RB? > Hive CLI startup takes a long time with a large number of databases > --- > > Key: HIVE-10319 > URL: https://issues.apache.org/jira/browse/HIVE-10319 > Project: Hive > Issue Type: Improvement > Components: CLI >Affects Versions: 1.0.0 >Reporter: Nezih Yigitbasi >Assignee: Nezih Yigitbasi > Attachments: HIVE-10319.patch > > > The Hive CLI takes a long time to start when there is a large number of > databases in the DW. I think the root cause is the way permanent UDFs are > loaded from the metastore. When I looked at the logs and the source code I > see that at startup Hive first gets all the databases from the metastore and > then for each database it makes a metastore call to get the permanent > functions for that database [see Hive.java | > https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L162-185]. > So the number of metastore calls made is in the order of the number of > databases. In production we have several hundreds of databases so Hive makes > several hundreds of RPC calls during startup, taking 30+ seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10650) Improve sum() function over windowing to support additional range formats
[
https://issues.apache.org/jira/browse/HIVE-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540839#comment-14540839
]
Aihua Xu commented on HIVE-10650:
-
Initial patch.
> Improve sum() function over windowing to support additional range formats
> -
>
> Key: HIVE-10650
> URL: https://issues.apache.org/jira/browse/HIVE-10650
> Project: Hive
> Issue Type: Sub-task
> Components: PTF-Windowing
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-10650.patch
>
>
> Support the following windowing function {{x preceding and y preceding}} and
> {{x following and y following}}.
> e.g.
> {noformat}
> select sum(value) over (partition by key order by value rows between 2
> preceding and 1 preceding) from tbl1;
> select sum(value) over (partition by key order by value rows between
> unbounded preceding and 1 preceding) from tbl1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10650) Improve sum() function over windowing to support additional range formats
[
https://issues.apache.org/jira/browse/HIVE-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-10650:
Description:
Support the following windowing function {{x preceding and y preceding}} and
{{x following and y following}}.
e.g.
{noformat}
select sum(value) over (partition by key order by value rows between 2
preceding and 1 preceding) from tbl1;
select sum(value) over (partition by key order by value rows between unbounded
preceding and 1 preceding) from tbl1;
{noformat}
was:
Support the following windowing function {{x preceding and y preceding}} and
{{x following and y following}}.
e.g.
{noformat}
select sum(value) over (partition by key order by value rows between 2
preceding and 1 preceding) from tbl1;
{noformat}
> Improve sum() function over windowing to support additional range formats
> -
>
> Key: HIVE-10650
> URL: https://issues.apache.org/jira/browse/HIVE-10650
> Project: Hive
> Issue Type: Sub-task
> Components: PTF-Windowing
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-10650.patch
>
>
> Support the following windowing function {{x preceding and y preceding}} and
> {{x following and y following}}.
> e.g.
> {noformat}
> select sum(value) over (partition by key order by value rows between 2
> preceding and 1 preceding) from tbl1;
> select sum(value) over (partition by key order by value rows between
> unbounded preceding and 1 preceding) from tbl1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10650) Improve sum() function over windowing to support additional range formats
[
https://issues.apache.org/jira/browse/HIVE-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-10650:
Description:
Support the following windowing function {{x preceding and y preceding}} and
{{x following and y following}}.
e.g.
{noformat}
select sum(value) over (partition by key order by value rows between 2
preceding and 1 preceding) from tbl1;
{noformat}
> Improve sum() function over windowing to support additional range formats
> -
>
> Key: HIVE-10650
> URL: https://issues.apache.org/jira/browse/HIVE-10650
> Project: Hive
> Issue Type: Sub-task
> Components: PTF-Windowing
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-10650.patch
>
>
> Support the following windowing function {{x preceding and y preceding}} and
> {{x following and y following}}.
> e.g.
> {noformat}
> select sum(value) over (partition by key order by value rows between 2
> preceding and 1 preceding) from tbl1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10687) AvroDeserializer fails to deserialize evolved union fields
[
https://issues.apache.org/jira/browse/HIVE-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540826#comment-14540826
]
Swarnim Kulkarni commented on HIVE-10687:
-
The test failure in unrelated.
> AvroDeserializer fails to deserialize evolved union fields
> --
>
> Key: HIVE-10687
> URL: https://issues.apache.org/jira/browse/HIVE-10687
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
> Attachments: HIVE-10687.1.patch
>
>
> Consider the union field:
> {noformat}
> union {int, string}
> {noformat}
> and now this field evolves to
> {noformat}
> union {null, int, string}.
> {noformat}
> Running it through the avro schema compatibility check[1], they are actually
> compatible which means that the latter could be used to deserialize the data
> written with former. However the avro deserializer fails to do that. Mainly
> because of the way it reads the tags from the reader schema and then reds the
> corresponding data from the writer schema. [2]
> [1] http://pastebin.cerner.corp/31078
> [2]
> https://github.com/cloudera/hive/blob/cdh5.4.0-release/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L354
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10650) Improve sum() function over windowing to support additional range formats
[ https://issues.apache.org/jira/browse/HIVE-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-10650: Attachment: HIVE-10650.patch > Improve sum() function over windowing to support additional range formats > - > > Key: HIVE-10650 > URL: https://issues.apache.org/jira/browse/HIVE-10650 > Project: Hive > Issue Type: Sub-task > Components: PTF-Windowing >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-10650.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10693) LLAP: DAG got stuck after reducer fetch failed
[
https://issues.apache.org/jira/browse/HIVE-10693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-10693:
Description:
Internal app ID application_1429683757595_0912, LLAP
application_1429683757595_0911. If someone without access wants to investigate
I'll get the logs.
I've ran into this only once. Feel free to close as not repro, I'll reopen if I
see again :) I want to make sure some debug info is preserved just in case.
Running Q1 - Map 1 w/1000 tasks (in this particular case), followed by Reducer
2 and Reducer 3, 1 task each, IIRC 3 is uber.
Fetch failed with I'd assume some random disturbance in the force:
{noformat}
2015-05-12 13:37:31,056 [fetcher [Map_1] #17()] WARN
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped:
Failed to verify reply after connecting to
cn047-10.l42scl.hortonworks.com:15551 with 1 inputs pending
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.$$YJP$$socketRead0(Native Method)
at java.net.SocketInputStream.socketRead0(SocketInputStream.java)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:703)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:787)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1534)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1439)
at
org.apache.tez.runtime.library.common.shuffle.HttpConnection.getInputStream(HttpConnection.java:256)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:339)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:257)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:167)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.run(FetcherOrderedGrouped.java:182)
{noformat}
AM registered this as Map 1 task failure
{noformat}
2015-05-12 13:37:31,156 INFO [Dispatcher thread: Central] impl.TaskAttemptImpl:
attempt_1429683757595_0912_1_00_000998_0 blamed for read error from
attempt_1429683757595_0912_1_01_00_0 at inputIndex 998
...
2015-05-12 13:37:31,174 INFO [Dispatcher thread: Central] impl.TaskImpl:
Scheduling new attempt for task: task_1429683757595_0912_1_00_000998,
currentFailedAttempts: 1, maxFailedAttempts: 4
{noformat}
Eventually Map 1 completed
{noformat}
2015-05-12 13:38:25,247 INFO [Dispatcher thread: Central]
history.HistoryEventHandler:
[HISTORY][DAG:dag_1429683757595_0912_1][Event:VERTEX_FINISHED]: vertexName=Map
1, vertexId=vertex_1429683757595_0912_1_00, initRequestedTime=1431462752913,
initedTime=1431462754818, startRequestedTime=1431462754819,
startedTime=1431462754819, finishTime=1431463105101, timeTaken=350282,
status=SUCCEEDED, diagnostics=, counters=Counters: 29,
org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=59,
RACK_LOCAL_TASKS=941, File System Counters, FILE_BYTES_READ=2160704,
FILE_BYTES_WRITTEN=20377550, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0,
FILE_WRITE_OPS=0, HDFS_BYTES_READ=9798097828287, HDFS_BYTES_WRITTEN=0,
HDFS_READ_OPS=406131, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0,
org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=4000,
GC_TIME_MILLIS=73309, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=-1000,
VIRTUAL_MEMORY_BYTES=-1000, COMMITTED_HEAP_BYTES=25769803776000,
INPUT_RECORDS_PROCESSED=5861038, OUTPUT_RECORDS=4000, OUTPUT_BYTES=376000,
OUTPUT_BYTES_WITH_OVERHEAD=0, OUTPUT_BYTES_PHYSICAL=0,
ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0,
ADDITIONAL_SPILL_COUNT=0, HIVE, DESERIALIZE_ERRORS=0,
RECORDS_IN_Map_1=589709, RECORDS_OUT_INTERMEDIATE_Map_1=4000,
vertexStats=firstTaskStartTime=1431462757804, firstTasksToStart=[
task_1429683757595_0912_1_00_00 ], lastTaskFinishTime=1431463105085,
lastTasksToFinish=[ task_1429683757595_0912_1_00_000999 ],
minTaskDuration=1743, maxTaskDuration=236653,
avgTaskDuration=6377.3342, numSuccessfulTasks=1000,
shortestDurationTasks=[ task_1429683757595_0912_1_00_000472 ],
longestDurationTasks=[ task_1429683757595_0912_1_00_000999 ],
vertexTaskStats={numFailedTaskAttempts=1, numKilledTaskAttempts=0
[jira] [Commented] (HIVE-10676) Update Hive's README to mention spark, and to remove jdk1.6
[ https://issues.apache.org/jira/browse/HIVE-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540792#comment-14540792 ] Sushanth Sowmyan commented on HIVE-10676: - I've updated those two pages - could you please verify to see if it's good? > Update Hive's README to mention spark, and to remove jdk1.6 > --- > > Key: HIVE-10676 > URL: https://issues.apache.org/jira/browse/HIVE-10676 > Project: Hive > Issue Type: Task >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan >Priority: Trivial > Labels: TODOC1.2 > Fix For: 1.2.0 > > Attachments: HIVE-10676.2.patch, HIVE-10676.patch > > > a) Hive's README file mentions only 2 execution frameworks, and does not > mention spark. We should add that in. > b) We should remove jdk1.6 from the README, since hive no longer supports or > even compiles under jdk1.6. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10692) LLAP: DAGs get stuck at start with no tasks executing
[
https://issues.apache.org/jira/browse/HIVE-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-10692:
Summary: LLAP: DAGs get stuck at start with no tasks executing (was: DAGs
get stuck at start with no tasks executing)
> LLAP: DAGs get stuck at start with no tasks executing
> -
>
> Key: HIVE-10692
> URL: https://issues.apache.org/jira/browse/HIVE-10692
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Siddharth Seth
>
> Internal app ID application_1429683757595_0914, LLAP
> application_1429683757595_0913. If someone without access wants to
> investigate I'll get the logs.
> 2nd dag failed to start executing:
> See syslog_dag_1429683757595_0914_2 log file.
> This happened to me a couple of times today, didn't see it before.
> After many S_TA_LAUNCH_REQUEST-s, the following is logged and after that
> there's no more logging aside from refreshes until I killed the DAG. LLAP
> daemons were idling meanwhile.
> I don't see any errors (aside from ATS) before this happened
> {noformat}
> 2015-05-12 13:52:08,997 INFO [TaskSchedulerEventHandlerThread]
> rm.TaskSchedulerEventHandler: Processing the event EventType:
> S_TA_LAUNCH_REQUEST
> 2015-05-12 13:52:18,507 INFO [LlapSchedulerNodeEnabler]
> impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
> 2015-05-12 13:52:25,315 INFO [HistoryEventHandlingThread]
> ats.ATSHistoryLoggingService: Event queue stats,
> eventsProcessedSinceLastUpdate=407, eventQueueSize=614
> 2015-05-12 13:52:28,507 INFO [LlapSchedulerNodeEnabler]
> impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10692) DAGs get stuck at start with no tasks executing
[
https://issues.apache.org/jira/browse/HIVE-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-10692:
Description:
Internal app ID application_1429683757595_0914, LLAP
application_1429683757595_0913. If someone without access wants to investigate
I'll get the logs.
2nd dag failed to start executing:
See syslog_dag_1429683757595_0914_2 log file.
This happened to me a couple of times today, didn't see it before.
After many S_TA_LAUNCH_REQUEST-s, the following is logged and after that
there's no more logging aside from refreshes until I killed the DAG. LLAP
daemons were idling meanwhile.
I don't see any errors (aside from ATS) before this happened
{noformat}
2015-05-12 13:52:08,997 INFO [TaskSchedulerEventHandlerThread]
rm.TaskSchedulerEventHandler: Processing the event EventType:
S_TA_LAUNCH_REQUEST
2015-05-12 13:52:18,507 INFO [LlapSchedulerNodeEnabler]
impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
2015-05-12 13:52:25,315 INFO [HistoryEventHandlingThread]
ats.ATSHistoryLoggingService: Event queue stats,
eventsProcessedSinceLastUpdate=407, eventQueueSize=614
2015-05-12 13:52:28,507 INFO [LlapSchedulerNodeEnabler]
impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
{noformat}
was:
Internal app ID application_1429683757595_0914, LLAP
application_1429683757595_0913. If someone without access wants to investigate
I'll get the logs.
2nd dag failed to start executing:
See syslog_dag_1429683757595_0914_2 log file.
This happened to me a couple of times today, didn't see it before.
After many S_TA_LAUNCH_REQUEST-s, the following is logged and after that
there's no more logging aside from refreshes until I killed the DAG. LLAP
daemons were idling meanwhile.
{noformat}
2015-05-12 13:52:08,997 INFO [TaskSchedulerEventHandlerThread]
rm.TaskSchedulerEventHandler: Processing the event EventType:
S_TA_LAUNCH_REQUEST
2015-05-12 13:52:18,507 INFO [LlapSchedulerNodeEnabler]
impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
2015-05-12 13:52:25,315 INFO [HistoryEventHandlingThread]
ats.ATSHistoryLoggingService: Event queue stats,
eventsProcessedSinceLastUpdate=407, eventQueueSize=614
2015-05-12 13:52:28,507 INFO [LlapSchedulerNodeEnabler]
impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
{noformat}
> DAGs get stuck at start with no tasks executing
> ---
>
> Key: HIVE-10692
> URL: https://issues.apache.org/jira/browse/HIVE-10692
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Siddharth Seth
>
> Internal app ID application_1429683757595_0914, LLAP
> application_1429683757595_0913. If someone without access wants to
> investigate I'll get the logs.
> 2nd dag failed to start executing:
> See syslog_dag_1429683757595_0914_2 log file.
> This happened to me a couple of times today, didn't see it before.
> After many S_TA_LAUNCH_REQUEST-s, the following is logged and after that
> there's no more logging aside from refreshes until I killed the DAG. LLAP
> daemons were idling meanwhile.
> I don't see any errors (aside from ATS) before this happened
> {noformat}
> 2015-05-12 13:52:08,997 INFO [TaskSchedulerEventHandlerThread]
> rm.TaskSchedulerEventHandler: Processing the event EventType:
> S_TA_LAUNCH_REQUEST
> 2015-05-12 13:52:18,507 INFO [LlapSchedulerNodeEnabler]
> impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
> 2015-05-12 13:52:25,315 INFO [HistoryEventHandlingThread]
> ats.ATSHistoryLoggingService: Event queue stats,
> eventsProcessedSinceLastUpdate=407, eventQueueSize=614
> 2015-05-12 13:52:28,507 INFO [LlapSchedulerNodeEnabler]
> impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10675) Provide option to skip Accumulo related Hive tests in itests directory
[ https://issues.apache.org/jira/browse/HIVE-10675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540758#comment-14540758 ] Sushanth Sowmyan commented on HIVE-10675: - +1. Thanks for the update! > Provide option to skip Accumulo related Hive tests in itests directory > -- > > Key: HIVE-10675 > URL: https://issues.apache.org/jira/browse/HIVE-10675 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-10675.1.patch, HIVE-10675.2.patch > > > We shouldn’t be running Accumulo+Hive tests in Windows as Accumulo is > currently unsupported on Windows. Hence we should provide an option to > disable these tests; that being mentioned, the default behavior would be to > enable these tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10657) Remove copyBytes operation from MD5 UDF
[ https://issues.apache.org/jira/browse/HIVE-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540751#comment-14540751 ] Jason Dere commented on HIVE-10657: --- Can you use Text.getBytes() rather than having to convert to String object? > Remove copyBytes operation from MD5 UDF > --- > > Key: HIVE-10657 > URL: https://issues.apache.org/jira/browse/HIVE-10657 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Alexander Pivovarov >Assignee: Alexander Pivovarov >Priority: Minor > Attachments: HIVE-10657.1.patch > > > Current MD5 UDF implementation uses Apache Commons DigestUtils.md5Hex method > to get md5 hex. > DigestUtils does not provide md5Hex method with signature (byte[], start, > length). This is why copyBytes method was added to UDFMd5 to get bytes[] from > BytesWritable. > To avoid copying bytes from BytesWritable to new byte array we can use java > MessageDigest API directly. > MessageDigest has method update(byte[], start, length) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10692) DAGs get stuck at start with no tasks executing
[
https://issues.apache.org/jira/browse/HIVE-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-10692:
Description:
Internal app ID application_1429683757595_0914, LLAP
application_1429683757595_0913. If someone without access wants to investigate
I'll get the logs.
2nd dag failed to start executing:
See syslog_dag_1429683757595_0914_2 log file.
This happened to me a couple of times today, didn't see it before.
After many S_TA_LAUNCH_REQUEST-s, the following is logged and after that
there's no more logging aside from refreshes until I killed the DAG. LLAP
daemons were idling meanwhile.
{noformat}
2015-05-12 13:52:08,997 INFO [TaskSchedulerEventHandlerThread]
rm.TaskSchedulerEventHandler: Processing the event EventType:
S_TA_LAUNCH_REQUEST
2015-05-12 13:52:18,507 INFO [LlapSchedulerNodeEnabler]
impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
2015-05-12 13:52:25,315 INFO [HistoryEventHandlingThread]
ats.ATSHistoryLoggingService: Event queue stats,
eventsProcessedSinceLastUpdate=407, eventQueueSize=614
2015-05-12 13:52:28,507 INFO [LlapSchedulerNodeEnabler]
impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
{noformat}
was:
Internal app ID application_1429683757595_0914, LLAP
application_1429683757595_0913. If someone without access wants to investigate
I'll get the logs.
2nd dag failed to start executing:
http://cn043-10.l42scl.hortonworks.com:8042/node/containerlogs/container_1429683757595_0914_01_01/sershe/syslog_dag_1429683757595_0914_2/?start=-65536
After many S_TA_LAUNCH_REQUEST-s, the following is logged and after that
there's no more logging aside from refreshes until I killed the DAG. LLAP
daemons were idling meanwhile.
{noformat}
2015-05-12 13:52:08,997 INFO [TaskSchedulerEventHandlerThread]
rm.TaskSchedulerEventHandler: Processing the event EventType:
S_TA_LAUNCH_REQUEST
2015-05-12 13:52:18,507 INFO [LlapSchedulerNodeEnabler]
impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
2015-05-12 13:52:25,315 INFO [HistoryEventHandlingThread]
ats.ATSHistoryLoggingService: Event queue stats,
eventsProcessedSinceLastUpdate=407, eventQueueSize=614
2015-05-12 13:52:28,507 INFO [LlapSchedulerNodeEnabler]
impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
{noformat}
> DAGs get stuck at start with no tasks executing
> ---
>
> Key: HIVE-10692
> URL: https://issues.apache.org/jira/browse/HIVE-10692
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Siddharth Seth
>
> Internal app ID application_1429683757595_0914, LLAP
> application_1429683757595_0913. If someone without access wants to
> investigate I'll get the logs.
> 2nd dag failed to start executing:
> See syslog_dag_1429683757595_0914_2 log file.
> This happened to me a couple of times today, didn't see it before.
> After many S_TA_LAUNCH_REQUEST-s, the following is logged and after that
> there's no more logging aside from refreshes until I killed the DAG. LLAP
> daemons were idling meanwhile.
> {noformat}
> 2015-05-12 13:52:08,997 INFO [TaskSchedulerEventHandlerThread]
> rm.TaskSchedulerEventHandler: Processing the event EventType:
> S_TA_LAUNCH_REQUEST
> 2015-05-12 13:52:18,507 INFO [LlapSchedulerNodeEnabler]
> impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
> 2015-05-12 13:52:25,315 INFO [HistoryEventHandlingThread]
> ats.ATSHistoryLoggingService: Event queue stats,
> eventsProcessedSinceLastUpdate=407, eventQueueSize=614
> 2015-05-12 13:52:28,507 INFO [LlapSchedulerNodeEnabler]
> impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10563) MiniTezCliDriver tests ordering issues
[ https://issues.apache.org/jira/browse/HIVE-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-10563: - Attachment: HIVE-10563.5.patch > MiniTezCliDriver tests ordering issues > -- > > Key: HIVE-10563 > URL: https://issues.apache.org/jira/browse/HIVE-10563 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-10563.1.patch, HIVE-10563.2.patch, > HIVE-10563.3.patch, HIVE-10563.4.patch, HIVE-10563.5.patch > > > There are a bunch of tests related to TestMiniTezCliDriver which gives > ordering issues when run on Centos/Windows/OSX -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10687) AvroDeserializer fails to deserialize evolved union fields
[
https://issues.apache.org/jira/browse/HIVE-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540678#comment-14540678
]
Hive QA commented on HIVE-10687:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12732330/HIVE-10687.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 8923 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3863/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3863/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3863/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12732330 - PreCommit-HIVE-TRUNK-Build
> AvroDeserializer fails to deserialize evolved union fields
> --
>
> Key: HIVE-10687
> URL: https://issues.apache.org/jira/browse/HIVE-10687
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
> Attachments: HIVE-10687.1.patch
>
>
> Consider the union field:
> {noformat}
> union {int, string}
> {noformat}
> and now this field evolves to
> {noformat}
> union {null, int, string}.
> {noformat}
> Running it through the avro schema compatibility check[1], they are actually
> compatible which means that the latter could be used to deserialize the data
> written with former. However the avro deserializer fails to do that. Mainly
> because of the way it reads the tags from the reader schema and then reds the
> corresponding data from the writer schema. [2]
> [1] http://pastebin.cerner.corp/31078
> [2]
> https://github.com/cloudera/hive/blob/cdh5.4.0-release/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L354
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10686) java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
[
https://issues.apache.org/jira/browse/HIVE-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-10686:
---
Attachment: HIVE-10686.03.patch
> java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
> -
>
> Key: HIVE-10686
> URL: https://issues.apache.org/jira/browse/HIVE-10686
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10686.01.patch, HIVE-10686.02.patch,
> HIVE-10686.03.patch, HIVE-10686.patch
>
>
> CBO throws Index out of bound exception for TPC-DS Q70.
> Query
> {code}
> explain
> select
> sum(ss_net_profit) as total_sum
>,s_state
>,s_county
>,grouping__id as lochierarchy
>, rank() over(partition by grouping__id, case when grouping__id == 2 then
> s_state end order by sum(ss_net_profit)) as rank_within_parent
> from
> store_sales ss join date_dim d1 on d1.d_date_sk = ss.ss_sold_date_sk
> join store s on s.s_store_sk = ss.ss_store_sk
> where
> d1.d_month_seq between 1193 and 1193+11
> and s.s_state in
> ( select s_state
>from (select s_state as s_state, sum(ss_net_profit),
> rank() over ( partition by s_state order by
> sum(ss_net_profit) desc) as ranking
> from store_sales, store, date_dim
> where d_month_seq between 1193 and 1193+11
> and date_dim.d_date_sk =
> store_sales.ss_sold_date_sk
> and store.s_store_sk = store_sales.ss_store_sk
> group by s_state
> ) tmp1
>where ranking <= 5
> )
> group by s_state,s_county with rollup
> order by
>lochierarchy desc
> ,case when lochierarchy = 0 then s_state end
> ,rank_within_parent
> limit 100
> {code}
> Original plan (correct)
> {code}
> HiveSort(fetch=[100])
> HiveSort(sort0=[$3], sort1=[$5], sort2=[$4], dir0=[DESC], dir1=[ASC],
> dir2=[ASC])
> HiveProject(total_sum=[$4], s_state=[$0], s_county=[$1],
> lochierarchy=[$5], rank_within_parent=[rank() OVER (PARTITION BY $5,
> when(==($5, 2), $0) ORDER BY $4 ROWS BETWEEN 2147483647 FOLLOWING AND
> 2147483647 PRECEDING)], (tok_function when (= (tok_table_or_col lochierarchy)
> 0) (tok_table_or_col s_state))=[when(=($5, 0), $0)])
> HiveAggregate(group=[{0, 1}], groups=[[{0, 1}, {0}, {}]],
> indicator=[true], agg#0=[sum($2)], GROUPING__ID=[GROUPING__ID()])
> HiveProject($f0=[$7], $f1=[$6], $f2=[$1])
> HiveJoin(condition=[=($5, $2)], joinType=[inner], algorithm=[none],
> cost=[{1177.2086187101072 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($3, $0)], joinType=[inner],
> algorithm=[none], cost=[{2880430.428726483 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_net_profit=[$21],
> ss_store_sk=[$22])
> HiveTableScan(table=[[tpcds.store_sales]])
> HiveProject(d_date_sk=[$0], d_month_seq=[$3])
> HiveFilter(condition=[between(false, $3, 1193, +(1193, 11))])
> HiveTableScan(table=[[tpcds.date_dim]])
> HiveProject(s_store_sk=[$0], s_county=[$1], s_state=[$2])
> SemiJoin(condition=[=($2, $3)], joinType=[inner])
> HiveProject(s_store_sk=[$0], s_county=[$23], s_state=[$24])
> HiveTableScan(table=[[tpcds.store]])
> HiveProject(s_state=[$0])
> HiveFilter(condition=[<=($1, 5)])
> HiveProject((tok_table_or_col s_state)=[$0],
> rank_window_0=[rank() OVER (PARTITION BY $0 ORDER BY $1 DESC ROWS BETWEEN
> 2147483647 FOLLOWING AND 2147483647 PRECEDING)])
> HiveAggregate(group=[{0}], agg#0=[sum($1)])
> HiveProject($f0=[$6], $f1=[$1])
> HiveJoin(condition=[=($5, $2)], joinType=[inner],
> algorithm=[none], cost=[{1177.2086187101072 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($3, $0)], joinType=[inner],
> algorithm=[none], cost=[{2880430.428726483 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0],
> ss_net_profit=[$21], ss_store_sk=[$22])
> HiveTableScan(table=[[tpcds.store_sales]])
> HiveProject(d_date_sk=[$0], d_month_seq=[$3])
> HiveFilter(condition=[between(false, $3,
> 1193, +(1193, 11))])
> HiveTableScan(table=[[tpcds.date_dim]])
> HiveProject(s_store_sk=[$0], s_state=[$24])
> HiveTableScan(table
[jira] [Updated] (HIVE-10686) java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
[
https://issues.apache.org/jira/browse/HIVE-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-10686:
---
Attachment: HIVE-10686.02.patch
> java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
> -
>
> Key: HIVE-10686
> URL: https://issues.apache.org/jira/browse/HIVE-10686
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10686.01.patch, HIVE-10686.02.patch,
> HIVE-10686.patch
>
>
> CBO throws Index out of bound exception for TPC-DS Q70.
> Query
> {code}
> explain
> select
> sum(ss_net_profit) as total_sum
>,s_state
>,s_county
>,grouping__id as lochierarchy
>, rank() over(partition by grouping__id, case when grouping__id == 2 then
> s_state end order by sum(ss_net_profit)) as rank_within_parent
> from
> store_sales ss join date_dim d1 on d1.d_date_sk = ss.ss_sold_date_sk
> join store s on s.s_store_sk = ss.ss_store_sk
> where
> d1.d_month_seq between 1193 and 1193+11
> and s.s_state in
> ( select s_state
>from (select s_state as s_state, sum(ss_net_profit),
> rank() over ( partition by s_state order by
> sum(ss_net_profit) desc) as ranking
> from store_sales, store, date_dim
> where d_month_seq between 1193 and 1193+11
> and date_dim.d_date_sk =
> store_sales.ss_sold_date_sk
> and store.s_store_sk = store_sales.ss_store_sk
> group by s_state
> ) tmp1
>where ranking <= 5
> )
> group by s_state,s_county with rollup
> order by
>lochierarchy desc
> ,case when lochierarchy = 0 then s_state end
> ,rank_within_parent
> limit 100
> {code}
> Original plan (correct)
> {code}
> HiveSort(fetch=[100])
> HiveSort(sort0=[$3], sort1=[$5], sort2=[$4], dir0=[DESC], dir1=[ASC],
> dir2=[ASC])
> HiveProject(total_sum=[$4], s_state=[$0], s_county=[$1],
> lochierarchy=[$5], rank_within_parent=[rank() OVER (PARTITION BY $5,
> when(==($5, 2), $0) ORDER BY $4 ROWS BETWEEN 2147483647 FOLLOWING AND
> 2147483647 PRECEDING)], (tok_function when (= (tok_table_or_col lochierarchy)
> 0) (tok_table_or_col s_state))=[when(=($5, 0), $0)])
> HiveAggregate(group=[{0, 1}], groups=[[{0, 1}, {0}, {}]],
> indicator=[true], agg#0=[sum($2)], GROUPING__ID=[GROUPING__ID()])
> HiveProject($f0=[$7], $f1=[$6], $f2=[$1])
> HiveJoin(condition=[=($5, $2)], joinType=[inner], algorithm=[none],
> cost=[{1177.2086187101072 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($3, $0)], joinType=[inner],
> algorithm=[none], cost=[{2880430.428726483 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_net_profit=[$21],
> ss_store_sk=[$22])
> HiveTableScan(table=[[tpcds.store_sales]])
> HiveProject(d_date_sk=[$0], d_month_seq=[$3])
> HiveFilter(condition=[between(false, $3, 1193, +(1193, 11))])
> HiveTableScan(table=[[tpcds.date_dim]])
> HiveProject(s_store_sk=[$0], s_county=[$1], s_state=[$2])
> SemiJoin(condition=[=($2, $3)], joinType=[inner])
> HiveProject(s_store_sk=[$0], s_county=[$23], s_state=[$24])
> HiveTableScan(table=[[tpcds.store]])
> HiveProject(s_state=[$0])
> HiveFilter(condition=[<=($1, 5)])
> HiveProject((tok_table_or_col s_state)=[$0],
> rank_window_0=[rank() OVER (PARTITION BY $0 ORDER BY $1 DESC ROWS BETWEEN
> 2147483647 FOLLOWING AND 2147483647 PRECEDING)])
> HiveAggregate(group=[{0}], agg#0=[sum($1)])
> HiveProject($f0=[$6], $f1=[$1])
> HiveJoin(condition=[=($5, $2)], joinType=[inner],
> algorithm=[none], cost=[{1177.2086187101072 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($3, $0)], joinType=[inner],
> algorithm=[none], cost=[{2880430.428726483 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0],
> ss_net_profit=[$21], ss_store_sk=[$22])
> HiveTableScan(table=[[tpcds.store_sales]])
> HiveProject(d_date_sk=[$0], d_month_seq=[$3])
> HiveFilter(condition=[between(false, $3,
> 1193, +(1193, 11))])
> HiveTableScan(table=[[tpcds.date_dim]])
> HiveProject(s_store_sk=[$0], s_state=[$24])
> HiveTableScan(table=[[tpcds.store]])
> {
[jira] [Commented] (HIVE-10679) JsonSerde ignores varchar and char size limit specified during table creation
[
https://issues.apache.org/jira/browse/HIVE-10679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540654#comment-14540654
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-10679:
--
+1
> JsonSerde ignores varchar and char size limit specified during table creation
> -
>
> Key: HIVE-10679
> URL: https://issues.apache.org/jira/browse/HIVE-10679
> Project: Hive
> Issue Type: Bug
>Reporter: Jagruti Varia
>Assignee: Jason Dere
> Attachments: HIVE-10679.1.patch, HIVE-10679.2.patch
>
>
> JsonSerde ignores varchar and char size limit specified during table creation
> and always creates varchar or char column with max length.
> steps to reproduce the issue:
> {noformat}
> create table jsonserde_1 (v varchar(50), c char(50)) row format serde
> 'org.apache.hive.hcatalog.data.JsonSerDe';
> desc jsonserde_1;
> OK
> v varchar(65535) from deserializer
> c char(255) from deserializer
> Time taken: 0.468 seconds, Fetched: 2 row(s)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10679) JsonSerde ignores varchar and char size limit specified during table creation
[
https://issues.apache.org/jira/browse/HIVE-10679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-10679:
--
Attachment: HIVE-10679.2.patch
Patch v2 also fixes the error message pointed out by Hari
> JsonSerde ignores varchar and char size limit specified during table creation
> -
>
> Key: HIVE-10679
> URL: https://issues.apache.org/jira/browse/HIVE-10679
> Project: Hive
> Issue Type: Bug
>Reporter: Jagruti Varia
>Assignee: Jason Dere
> Attachments: HIVE-10679.1.patch, HIVE-10679.2.patch
>
>
> JsonSerde ignores varchar and char size limit specified during table creation
> and always creates varchar or char column with max length.
> steps to reproduce the issue:
> {noformat}
> create table jsonserde_1 (v varchar(50), c char(50)) row format serde
> 'org.apache.hive.hcatalog.data.JsonSerDe';
> desc jsonserde_1;
> OK
> v varchar(65535) from deserializer
> c char(255) from deserializer
> Time taken: 0.468 seconds, Fetched: 2 row(s)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10675) Provide option to skip Accumulo related Hive tests in itests directory
[ https://issues.apache.org/jira/browse/HIVE-10675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540620#comment-14540620 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-10675: -- [~sushanth] Thanks for the review, attached the patch with comment. Thanks Hari > Provide option to skip Accumulo related Hive tests in itests directory > -- > > Key: HIVE-10675 > URL: https://issues.apache.org/jira/browse/HIVE-10675 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-10675.1.patch, HIVE-10675.2.patch > > > We shouldn’t be running Accumulo+Hive tests in Windows as Accumulo is > currently unsupported on Windows. Hence we should provide an option to > disable these tests; that being mentioned, the default behavior would be to > enable these tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10675) Provide option to skip Accumulo related Hive tests in itests directory
[ https://issues.apache.org/jira/browse/HIVE-10675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-10675: - Attachment: HIVE-10675.2.patch > Provide option to skip Accumulo related Hive tests in itests directory > -- > > Key: HIVE-10675 > URL: https://issues.apache.org/jira/browse/HIVE-10675 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-10675.1.patch, HIVE-10675.2.patch > > > We shouldn’t be running Accumulo+Hive tests in Windows as Accumulo is > currently unsupported on Windows. Hence we should provide an option to > disable these tests; that being mentioned, the default behavior would be to > enable these tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10679) JsonSerde ignores varchar and char size limit specified during table creation
[
https://issues.apache.org/jira/browse/HIVE-10679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540584#comment-14540584
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-10679:
--
+1
N.B:
We might need to improve this message in
PrimitiveObjectInspectorFactory.getPrimitiveJavaObjectInspector.
{code}
default:
throw new RuntimeException("Failed to create
JavaHiveVarcharObjectInspector for " + typeInfo );
{code}
Thanks
Hari
> JsonSerde ignores varchar and char size limit specified during table creation
> -
>
> Key: HIVE-10679
> URL: https://issues.apache.org/jira/browse/HIVE-10679
> Project: Hive
> Issue Type: Bug
>Reporter: Jagruti Varia
>Assignee: Jason Dere
> Attachments: HIVE-10679.1.patch
>
>
> JsonSerde ignores varchar and char size limit specified during table creation
> and always creates varchar or char column with max length.
> steps to reproduce the issue:
> {noformat}
> create table jsonserde_1 (v varchar(50), c char(50)) row format serde
> 'org.apache.hive.hcatalog.data.JsonSerDe';
> desc jsonserde_1;
> OK
> v varchar(65535) from deserializer
> c char(255) from deserializer
> Time taken: 0.468 seconds, Fetched: 2 row(s)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10675) Provide option to skip Accumulo related Hive tests in itests directory
[
https://issues.apache.org/jira/browse/HIVE-10675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540577#comment-14540577
]
Sushanth Sowmyan commented on HIVE-10675:
-
+1 to correctness, but confusing at first read when I see
{noformat}
true
{noformat}
Could you please add a comment there that indicates that the profile activation
clause will flip this to false as long as -DskipAccumuloTests is not specified?
Thanks!
> Provide option to skip Accumulo related Hive tests in itests directory
> --
>
> Key: HIVE-10675
> URL: https://issues.apache.org/jira/browse/HIVE-10675
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-10675.1.patch
>
>
> We shouldn’t be running Accumulo+Hive tests in Windows as Accumulo is
> currently unsupported on Windows. Hence we should provide an option to
> disable these tests; that being mentioned, the default behavior would be to
> enable these tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10690) ArrayIndexOutOfBounds exception in MetaStoreDirectSql.aggrColStatsForPartitions()
[
https://issues.apache.org/jira/browse/HIVE-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540569#comment-14540569
]
Jason Dere commented on HIVE-10690:
---
+1
> ArrayIndexOutOfBounds exception in
> MetaStoreDirectSql.aggrColStatsForPartitions()
> -
>
> Key: HIVE-10690
> URL: https://issues.apache.org/jira/browse/HIVE-10690
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.0
>Reporter: Jason Dere
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-10690.1.patch
>
>
> Noticed a bunch of these stack traces in hive.log while running some unit
> tests:
> {noformat}
> 2015-05-11 21:18:59,371 WARN [main]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(2420)) - Direct SQL failed
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1132)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6162)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6158)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2385)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy84.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:5662)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy86.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAggrColStatsFor(HiveMetaStoreClient.java:2064)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy87.getAggrColStatsFor(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAggrColStatsFor(Hive.java:3110)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:245)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:329)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:399)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:392)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:150)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:77)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:64)
> at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke(ReflectiveRelMetadataProvider.java:182)
> at com.sun.proxy.$Proxy108.getDistinctRowCount(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
[jira] [Updated] (HIVE-10672) Analyze command on a table using row format serde JsonSerDe fails with NoClassDefFoundError
[
https://issues.apache.org/jira/browse/HIVE-10672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10672:
Labels: TODOC1.2 (was: )
> Analyze command on a table using row format serde JsonSerDe fails with
> NoClassDefFoundError
> ---
>
> Key: HIVE-10672
> URL: https://issues.apache.org/jira/browse/HIVE-10672
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.2.0
>Reporter: Jason Dere
>Assignee: Jason Dere
> Labels: TODOC1.2
> Attachments: HIVE-10672.1.patch
>
>
> Found by [~deepesh].
> Running analyze command on a table created using the following DDL:
> {noformat}
> create external table all100kjson(
> s string,
> i int,
> d double,
> m map,
> bb array>)
> row format serde 'org.apache.hive.hcatalog.data.JsonSerDe'
> STORED AS TEXTFILE
> location '/user/hcat/tests/data/all100kjson';
> {noformat}
> analyze command
> {noformat}
> analyze table all100kjson compute statistics;
> {noformat}
> throws the following error:
> {noformat}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1431071702167_0006_1_00,
> diagnostics=[Task failed, taskId=task_1431071702167_0006_1_00_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:331)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/metastore/IMetaStoreClient
> at
> org.apache.hive.hcatalog.data.schema.HCatFieldSchema.(HCatFieldSchema.java:225)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:122)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165)
> at
> org.apache.hive.hcatalog.data.JsonSerDe.initialize(JsonSerDe.java:141)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:527)
> at
> org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:191)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.metastore.IMetaStoreClient
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
[jira] [Commented] (HIVE-10672) Analyze command on a table using row format serde JsonSerDe fails with NoClassDefFoundError
[
https://issues.apache.org/jira/browse/HIVE-10672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540566#comment-14540566
]
Sushanth Sowmyan commented on HIVE-10672:
-
Btw, I would suggest, as a general practice, in this case, in addition to this
patch, that if you add hcatalog-core to your aux jars, then you should add the
metastore jar to it as well. We should capture this in documentation, adding a
doc todo task for that.
> Analyze command on a table using row format serde JsonSerDe fails with
> NoClassDefFoundError
> ---
>
> Key: HIVE-10672
> URL: https://issues.apache.org/jira/browse/HIVE-10672
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.2.0
>Reporter: Jason Dere
>Assignee: Jason Dere
> Labels: TODOC1.2
> Attachments: HIVE-10672.1.patch
>
>
> Found by [~deepesh].
> Running analyze command on a table created using the following DDL:
> {noformat}
> create external table all100kjson(
> s string,
> i int,
> d double,
> m map,
> bb array>)
> row format serde 'org.apache.hive.hcatalog.data.JsonSerDe'
> STORED AS TEXTFILE
> location '/user/hcat/tests/data/all100kjson';
> {noformat}
> analyze command
> {noformat}
> analyze table all100kjson compute statistics;
> {noformat}
> throws the following error:
> {noformat}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1431071702167_0006_1_00,
> diagnostics=[Task failed, taskId=task_1431071702167_0006_1_00_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:331)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/metastore/IMetaStoreClient
> at
> org.apache.hive.hcatalog.data.schema.HCatFieldSchema.(HCatFieldSchema.java:225)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:122)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165)
> at
> org.apache.hive.hcatalog.data.JsonSerDe.initialize(JsonSerDe.java:141)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:527)
> at
> org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:191)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.metastore.IMetaStoreClient
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.j
[jira] [Updated] (HIVE-10690) ArrayIndexOutOfBounds exception in MetaStoreDirectSql.aggrColStatsForPartitions()
[
https://issues.apache.org/jira/browse/HIVE-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-10690:
Attachment: HIVE-10690.1.patch
> ArrayIndexOutOfBounds exception in
> MetaStoreDirectSql.aggrColStatsForPartitions()
> -
>
> Key: HIVE-10690
> URL: https://issues.apache.org/jira/browse/HIVE-10690
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.0
>Reporter: Jason Dere
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-10690.1.patch
>
>
> Noticed a bunch of these stack traces in hive.log while running some unit
> tests:
> {noformat}
> 2015-05-11 21:18:59,371 WARN [main]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(2420)) - Direct SQL failed
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1132)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6162)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6158)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2385)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy84.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:5662)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy86.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAggrColStatsFor(HiveMetaStoreClient.java:2064)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy87.getAggrColStatsFor(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAggrColStatsFor(Hive.java:3110)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:245)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:329)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:399)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:392)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:150)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:77)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:64)
> at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke(ReflectiveRelMetadataProvider.java:182)
> at com.sun.proxy.$Proxy108.getDistinctRowCount(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
[jira] [Commented] (HIVE-10690) ArrayIndexOutOfBounds exception in MetaStoreDirectSql.aggrColStatsForPartitions()
[
https://issues.apache.org/jira/browse/HIVE-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540559#comment-14540559
]
Vaibhav Gumashta commented on HIVE-10690:
-
[~jdere] Small patch, can you review please. Thanks.
> ArrayIndexOutOfBounds exception in
> MetaStoreDirectSql.aggrColStatsForPartitions()
> -
>
> Key: HIVE-10690
> URL: https://issues.apache.org/jira/browse/HIVE-10690
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.0
>Reporter: Jason Dere
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-10690.1.patch
>
>
> Noticed a bunch of these stack traces in hive.log while running some unit
> tests:
> {noformat}
> 2015-05-11 21:18:59,371 WARN [main]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(2420)) - Direct SQL failed
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1132)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6162)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6158)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2385)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy84.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:5662)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy86.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAggrColStatsFor(HiveMetaStoreClient.java:2064)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy87.getAggrColStatsFor(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAggrColStatsFor(Hive.java:3110)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:245)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:329)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:399)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:392)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:150)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:77)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:64)
> at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke(ReflectiveRelMetadataProvider.java:182)
> at com.sun.proxy.$Proxy108.getDistinctRowCount(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source)
> at
> su
[jira] [Updated] (HIVE-10690) ArrayIndexOutOfBounds exception in MetaStoreDirectSql.aggrColStatsForPartitions()
[
https://issues.apache.org/jira/browse/HIVE-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-10690:
Affects Version/s: 1.2.0
> ArrayIndexOutOfBounds exception in
> MetaStoreDirectSql.aggrColStatsForPartitions()
> -
>
> Key: HIVE-10690
> URL: https://issues.apache.org/jira/browse/HIVE-10690
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.0
>Reporter: Jason Dere
>Assignee: Vaibhav Gumashta
>
> Noticed a bunch of these stack traces in hive.log while running some unit
> tests:
> {noformat}
> 2015-05-11 21:18:59,371 WARN [main]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(2420)) - Direct SQL failed
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1132)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6162)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6158)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2385)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy84.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:5662)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy86.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAggrColStatsFor(HiveMetaStoreClient.java:2064)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy87.getAggrColStatsFor(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAggrColStatsFor(Hive.java:3110)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:245)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:329)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:399)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:392)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:150)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:77)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:64)
> at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke(ReflectiveRelMetadataProvider.java:182)
> at com.sun.proxy.$Proxy108.getDistinctRowCount(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invo
[jira] [Assigned] (HIVE-10690) ArrayIndexOutOfBounds exception in MetaStoreDirectSql.aggrColStatsForPartitions()
[
https://issues.apache.org/jira/browse/HIVE-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta reassigned HIVE-10690:
---
Assignee: Vaibhav Gumashta
> ArrayIndexOutOfBounds exception in
> MetaStoreDirectSql.aggrColStatsForPartitions()
> -
>
> Key: HIVE-10690
> URL: https://issues.apache.org/jira/browse/HIVE-10690
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Jason Dere
>Assignee: Vaibhav Gumashta
>
> Noticed a bunch of these stack traces in hive.log while running some unit
> tests:
> {noformat}
> 2015-05-11 21:18:59,371 WARN [main]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(2420)) - Direct SQL failed
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1132)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6162)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6158)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2385)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy84.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:5662)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy86.get_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAggrColStatsFor(HiveMetaStoreClient.java:2064)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy87.getAggrColStatsFor(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAggrColStatsFor(Hive.java:3110)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:245)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:329)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:399)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:392)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:150)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:77)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:64)
> at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider$1$1.invoke(ReflectiveRelMetadataProvider.java:182)
> at com.sun.proxy.$Proxy108.getDistinctRowCount(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
[jira] [Commented] (HIVE-10627) Queries fail with Failed to breakup Windowing invocations into Groups
[
https://issues.apache.org/jira/browse/HIVE-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540522#comment-14540522
]
Hive QA commented on HIVE-10627:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12732286/HIVE-10627.03.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 8921 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3862/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3862/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3862/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12732286 - PreCommit-HIVE-TRUNK-Build
> Queries fail with Failed to breakup Windowing invocations into Groups
> -
>
> Key: HIVE-10627
> URL: https://issues.apache.org/jira/browse/HIVE-10627
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10627.01.patch, HIVE-10627.01.patch,
> HIVE-10627.02.patch, HIVE-10627.03.patch, HIVE-10627.patch
>
>
> TPC-DS queries 51 fails with Failed to breakup Windowing invocations into
> Groups. At least 1 group must only depend on input columns. Also check for
> circular dependencies.
> {code}
> explain
> WITH web_v1 as (
> select
> ws_item_sk item_sk, d_date, sum(ws_sales_price),
> sum(sum(ws_sales_price))
> over (partition by ws_item_sk order by d_date rows between unbounded
> preceding and current row) cume_sales
> from web_sales
> ,date_dim
> where ws_sold_date_sk=d_date_sk
> and d_month_seq between 1193 and 1193+11
> and ws_item_sk is not NULL
> group by ws_item_sk, d_date),
> store_v1 as (
> select
> ss_item_sk item_sk, d_date, sum(ss_sales_price),
> sum(sum(ss_sales_price))
> over (partition by ss_item_sk order by d_date rows between unbounded
> preceding and current row) cume_sales
> from store_sales
> ,date_dim
> where ss_sold_date_sk=d_date_sk
> and d_month_seq between 1193 and 1193+11
> and ss_item_sk is not NULL
> group by ss_item_sk, d_date)
> select *
> from (select item_sk
> ,d_date
> ,web_sales
> ,store_sales
> ,max(web_sales)
> over (partition by item_sk order by d_date rows between unbounded
> preceding and current row) web_cumulative
> ,max(store_sales)
> over (partition by item_sk order by d_date rows between unbounded
> preceding and current row) store_cumulative
> from (select case when web.item_sk is not null then web.item_sk else
> store.item_sk end item_sk
> ,case when web.d_date is not null then web.d_date else
> store.d_date end d_date
> ,web.cume_sales web_sales
> ,store.cume_sales store_sales
>from web_v1 web full outer join store_v1 store on (web.item_sk =
> store.item_sk
> and web.d_date =
> store.d_date)
> )x )y
> where web_cumulative > store_cumulative
> order by item_sk
> ,d_date
> limit 100;
> {code}
> Exception
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup
> Windowing invocations into Groups. At least 1 group must only depend on input
> columns. Also check for circular dependencies.
> Underlying error: org.apache.hadoop.hive.ql.parse.SemanticException: Line
> 0:-1 Invalid column reference '$f2'
> at
> org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:94)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:11538)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8514)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8472)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9304)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9189)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9210)
> at
> org.apache.hadoop.hive.ql.parse.Se
[jira] [Updated] (HIVE-10689) HS2 metadata api calls should use HiveAuthorizer interface for authorization
[ https://issues.apache.org/jira/browse/HIVE-10689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-10689: - Summary: HS2 metadata api calls should use HiveAuthorizer interface for authorization (was: HS2 metadata api calls should be authorized via HiveAuthorizer) > HS2 metadata api calls should use HiveAuthorizer interface for authorization > > > Key: HIVE-10689 > URL: https://issues.apache.org/jira/browse/HIVE-10689 > Project: Hive > Issue Type: Bug > Components: Authorization, SQLStandardAuthorization >Reporter: Thejas M Nair >Assignee: Thejas M Nair > > java.sql.DataBaseMetadata apis in jdbc api result in calls to HS2 metadata > api's and their execution is via separate Hive Operation implementations, > that don't use the Hive Driver class. Invocation of these api's should also > be authorized using the HiveAuthorizer api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10687) AvroDeserializer fails to deserialize evolved union fields
[
https://issues.apache.org/jira/browse/HIVE-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540519#comment-14540519
]
Swarnim Kulkarni commented on HIVE-10687:
-
[~brocknoland] If you get a chance, mind taking a quick look since you had done
some work on HIVE-9462 which seems related.
> AvroDeserializer fails to deserialize evolved union fields
> --
>
> Key: HIVE-10687
> URL: https://issues.apache.org/jira/browse/HIVE-10687
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
> Attachments: HIVE-10687.1.patch
>
>
> Consider the union field:
> {noformat}
> union {int, string}
> {noformat}
> and now this field evolves to
> {noformat}
> union {null, int, string}.
> {noformat}
> Running it through the avro schema compatibility check[1], they are actually
> compatible which means that the latter could be used to deserialize the data
> written with former. However the avro deserializer fails to do that. Mainly
> because of the way it reads the tags from the reader schema and then reds the
> corresponding data from the writer schema. [2]
> [1] http://pastebin.cerner.corp/31078
> [2]
> https://github.com/cloudera/hive/blob/cdh5.4.0-release/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L354
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10688) constant folding is broken for case-when udf
[
https://issues.apache.org/jira/browse/HIVE-10688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540499#comment-14540499
]
Gopal V commented on HIVE-10688:
[~jvaria]: a fix for this exists in a patch I'm currently reviewing - HIVE-10327
We can backport that part immediately to 1.2.0.
> constant folding is broken for case-when udf
>
>
> Key: HIVE-10688
> URL: https://issues.apache.org/jira/browse/HIVE-10688
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jagruti Varia
>Assignee: Ashutosh Chauhan
> Fix For: 1.2.0
>
>
> In some cases, case-when udf throws IndexOutOfBoundsException as shown below:
> {noformat}
> FAILED: IndexOutOfBoundsException Index: 2, Size: 2
> java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.shortcutFunction(ConstantPropagateProcFactory.java:428)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:238)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.foldExpr(ConstantPropagateProcFactory.java:227)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory.access$000(C
[jira] [Updated] (HIVE-10687) AvroDeserializer fails to deserialize evolved union fields
[
https://issues.apache.org/jira/browse/HIVE-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swarnim Kulkarni updated HIVE-10687:
Attachment: HIVE-10687.1.patch
Patch attached.
> AvroDeserializer fails to deserialize evolved union fields
> --
>
> Key: HIVE-10687
> URL: https://issues.apache.org/jira/browse/HIVE-10687
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
> Attachments: HIVE-10687.1.patch
>
>
> Consider the union field:
> {noformat}
> union {int, string}
> {noformat}
> and now this field evolves to
> {noformat}
> union {null, int, string}.
> {noformat}
> Running it through the avro schema compatibility check[1], they are actually
> compatible which means that the latter could be used to deserialize the data
> written with former. However the avro deserializer fails to do that. Mainly
> because of the way it reads the tags from the reader schema and then reds the
> corresponding data from the writer schema. [2]
> [1] http://pastebin.cerner.corp/31078
> [2]
> https://github.com/cloudera/hive/blob/cdh5.4.0-release/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L354
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10630) Renaming tables across encryption zones renames table even though the operation throws error
[
https://issues.apache.org/jira/browse/HIVE-10630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540409#comment-14540409
]
Eugene Koifman commented on HIVE-10630:
---
raun udaf_corr.q locally - no issues
encryption_insert_partition_static.q fails in the same way in other bot runs
> Renaming tables across encryption zones renames table even though the
> operation throws error
>
>
> Key: HIVE-10630
> URL: https://issues.apache.org/jira/browse/HIVE-10630
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore, Security
>Reporter: Deepesh Khandelwal
>Assignee: Eugene Koifman
> Attachments: HIVE-10630.patch
>
>
> Create a table with data in an encrypted zone 1 and then rename it to
> encrypted zone 2.
> {noformat}
> hive> alter table encdb1.testtbl rename to encdb2.testtbl;
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. Unable to
> access old location
> hdfs://node-1.example.com:8020/apps/hive/warehouse/encdb1.db/testtbl for
> table encdb1.testtbl
> {noformat}
> Even though the command errors out the table is renamed. I think the right
> behavior should be to not rename the table at all including the metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10325) Remove ExprNodeNullEvaluator
[ https://issues.apache.org/jira/browse/HIVE-10325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540326#comment-14540326 ] Gopal V commented on HIVE-10325: [~ashutoshc]: LGTM +1 > Remove ExprNodeNullEvaluator > > > Key: HIVE-10325 > URL: https://issues.apache.org/jira/browse/HIVE-10325 > Project: Hive > Issue Type: Task > Components: Query Processor >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-10325.1.patch, HIVE-10325.2.patch, HIVE-10325.patch > > > since its purpose can instead be served by ExprNodeConstantEvaluator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10643) Refactoring Windowing for sum() to pass WindowFrameDef instead of two numbers (1 for number of preceding and 1 for number of following)
[
https://issues.apache.org/jira/browse/HIVE-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540265#comment-14540265
]
Aihua Xu commented on HIVE-10643:
-
Thanks.
> Refactoring Windowing for sum() to pass WindowFrameDef instead of two numbers
> (1 for number of preceding and 1 for number of following)
> ---
>
> Key: HIVE-10643
> URL: https://issues.apache.org/jira/browse/HIVE-10643
> Project: Hive
> Issue Type: Sub-task
> Components: PTF-Windowing
>Affects Versions: 1.3.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Minor
> Fix For: 1.3.0
>
> Attachments: HIVE-10643.patch
>
>
> The functionality should not be affected. Instead of passing 2 numbers (1 for
> # of preceding rows and 1 for # of following rows), we will pass
> WindowFrameDef object around. In the following subtasks, it will be used to
> support additional window like {{rows between x preceding and y preceding}}
> and {{rows between x following and y following}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10627) Queries fail with Failed to breakup Windowing invocations into Groups
[
https://issues.apache.org/jira/browse/HIVE-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-10627:
---
Attachment: HIVE-10627.03.patch
> Queries fail with Failed to breakup Windowing invocations into Groups
> -
>
> Key: HIVE-10627
> URL: https://issues.apache.org/jira/browse/HIVE-10627
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10627.01.patch, HIVE-10627.01.patch,
> HIVE-10627.02.patch, HIVE-10627.03.patch, HIVE-10627.patch
>
>
> TPC-DS queries 51 fails with Failed to breakup Windowing invocations into
> Groups. At least 1 group must only depend on input columns. Also check for
> circular dependencies.
> {code}
> explain
> WITH web_v1 as (
> select
> ws_item_sk item_sk, d_date, sum(ws_sales_price),
> sum(sum(ws_sales_price))
> over (partition by ws_item_sk order by d_date rows between unbounded
> preceding and current row) cume_sales
> from web_sales
> ,date_dim
> where ws_sold_date_sk=d_date_sk
> and d_month_seq between 1193 and 1193+11
> and ws_item_sk is not NULL
> group by ws_item_sk, d_date),
> store_v1 as (
> select
> ss_item_sk item_sk, d_date, sum(ss_sales_price),
> sum(sum(ss_sales_price))
> over (partition by ss_item_sk order by d_date rows between unbounded
> preceding and current row) cume_sales
> from store_sales
> ,date_dim
> where ss_sold_date_sk=d_date_sk
> and d_month_seq between 1193 and 1193+11
> and ss_item_sk is not NULL
> group by ss_item_sk, d_date)
> select *
> from (select item_sk
> ,d_date
> ,web_sales
> ,store_sales
> ,max(web_sales)
> over (partition by item_sk order by d_date rows between unbounded
> preceding and current row) web_cumulative
> ,max(store_sales)
> over (partition by item_sk order by d_date rows between unbounded
> preceding and current row) store_cumulative
> from (select case when web.item_sk is not null then web.item_sk else
> store.item_sk end item_sk
> ,case when web.d_date is not null then web.d_date else
> store.d_date end d_date
> ,web.cume_sales web_sales
> ,store.cume_sales store_sales
>from web_v1 web full outer join store_v1 store on (web.item_sk =
> store.item_sk
> and web.d_date =
> store.d_date)
> )x )y
> where web_cumulative > store_cumulative
> order by item_sk
> ,d_date
> limit 100;
> {code}
> Exception
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup
> Windowing invocations into Groups. At least 1 group must only depend on input
> columns. Also check for circular dependencies.
> Underlying error: org.apache.hadoop.hive.ql.parse.SemanticException: Line
> 0:-1 Invalid column reference '$f2'
> at
> org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:94)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:11538)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8514)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8472)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9304)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9189)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9210)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9189)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9210)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9189)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9210)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9189)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9210)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9189)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9210)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9592)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:208)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.ap
[jira] [Updated] (HIVE-10687) AvroDeserializer fails to deserialize evolved union fields
[
https://issues.apache.org/jira/browse/HIVE-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swarnim Kulkarni updated HIVE-10687:
Description:
Consider the union field:
{noformat}
union {int, string}
{noformat}
and now this field evolves to
{noformat}
union {null, int, string}.
{noformat}
Running it through the avro schema compatibility check[1], they are actually
compatible which means that the latter could be used to deserialize the data
written with former. However the avro deserializer fails to do that. Mainly
because of the way it reads the tags from the reader schema and then reds the
corresponding data from the writer schema. [2]
[1] http://pastebin.cerner.corp/31078
[2]
https://github.com/cloudera/hive/blob/cdh5.4.0-release/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L354
was:
Consider the union field:
union {int, string}
and now this field evolves to
union {null, int, string}.
Running it through the avro schema compatibility check[1], they are actually
compatible which means that the latter could be used to deserialize the data
written with former. However the avro deserializer fails to do that. Mainly
because of the way it reads the tags from the reader schema and then reds the
corresponding data from the writer schema. [2]
[1] http://pastebin.cerner.corp/31078
[2]
https://github.com/cloudera/hive/blob/cdh5.4.0-release/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L354
> AvroDeserializer fails to deserialize evolved union fields
> --
>
> Key: HIVE-10687
> URL: https://issues.apache.org/jira/browse/HIVE-10687
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
>
> Consider the union field:
> {noformat}
> union {int, string}
> {noformat}
> and now this field evolves to
> {noformat}
> union {null, int, string}.
> {noformat}
> Running it through the avro schema compatibility check[1], they are actually
> compatible which means that the latter could be used to deserialize the data
> written with former. However the avro deserializer fails to do that. Mainly
> because of the way it reads the tags from the reader schema and then reds the
> corresponding data from the writer schema. [2]
> [1] http://pastebin.cerner.corp/31078
> [2]
> https://github.com/cloudera/hive/blob/cdh5.4.0-release/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L354
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-10687) AvroDeserializer fails to deserialize evolved union fields
[
https://issues.apache.org/jira/browse/HIVE-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swarnim Kulkarni reassigned HIVE-10687:
---
Assignee: Swarnim Kulkarni
> AvroDeserializer fails to deserialize evolved union fields
> --
>
> Key: HIVE-10687
> URL: https://issues.apache.org/jira/browse/HIVE-10687
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
>
> Consider the union field:
> union {int, string}
> and now this field evolves to
> union {null, int, string}.
> Running it through the avro schema compatibility check[1], they are actually
> compatible which means that the latter could be used to deserialize the data
> written with former. However the avro deserializer fails to do that. Mainly
> because of the way it reads the tags from the reader schema and then reds the
> corresponding data from the writer schema. [2]
> [1] http://pastebin.cerner.corp/31078
> [2]
> https://github.com/cloudera/hive/blob/cdh5.4.0-release/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L354
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10254) Parquet PPD support DECIMAL
[ https://issues.apache.org/jira/browse/HIVE-10254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540127#comment-14540127 ] Ryan Blue commented on HIVE-10254: -- [~dongc], I think you're right that we need a comparator, but I think this should be at the column-level rather than associated with the Binary class. Could you open a Parquet issue to discuss this in the Parquet community? > Parquet PPD support DECIMAL > --- > > Key: HIVE-10254 > URL: https://issues.apache.org/jira/browse/HIVE-10254 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > > The Decimal data type has been supported for Parquet on Hive. But the > predicate push down does not work for Decimal type. This Jira will make PPD > support Decimal type for Parquet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10686) java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
[
https://issues.apache.org/jira/browse/HIVE-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539982#comment-14539982
]
Hive QA commented on HIVE-10686:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12732229/HIVE-10686.01.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 8919 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_global_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3861/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3861/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3861/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12732229 - PreCommit-HIVE-TRUNK-Build
> java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
> -
>
> Key: HIVE-10686
> URL: https://issues.apache.org/jira/browse/HIVE-10686
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10686.01.patch, HIVE-10686.patch
>
>
> CBO throws Index out of bound exception for TPC-DS Q70.
> Query
> {code}
> explain
> select
> sum(ss_net_profit) as total_sum
>,s_state
>,s_county
>,grouping__id as lochierarchy
>, rank() over(partition by grouping__id, case when grouping__id == 2 then
> s_state end order by sum(ss_net_profit)) as rank_within_parent
> from
> store_sales ss join date_dim d1 on d1.d_date_sk = ss.ss_sold_date_sk
> join store s on s.s_store_sk = ss.ss_store_sk
> where
> d1.d_month_seq between 1193 and 1193+11
> and s.s_state in
> ( select s_state
>from (select s_state as s_state, sum(ss_net_profit),
> rank() over ( partition by s_state order by
> sum(ss_net_profit) desc) as ranking
> from store_sales, store, date_dim
> where d_month_seq between 1193 and 1193+11
> and date_dim.d_date_sk =
> store_sales.ss_sold_date_sk
> and store.s_store_sk = store_sales.ss_store_sk
> group by s_state
> ) tmp1
>where ranking <= 5
> )
> group by s_state,s_county with rollup
> order by
>lochierarchy desc
> ,case when lochierarchy = 0 then s_state end
> ,rank_within_parent
> limit 100
> {code}
> Original plan (correct)
> {code}
> HiveSort(fetch=[100])
> HiveSort(sort0=[$3], sort1=[$5], sort2=[$4], dir0=[DESC], dir1=[ASC],
> dir2=[ASC])
> HiveProject(total_sum=[$4], s_state=[$0], s_county=[$1],
> lochierarchy=[$5], rank_within_parent=[rank() OVER (PARTITION BY $5,
> when(==($5, 2), $0) ORDER BY $4 ROWS BETWEEN 2147483647 FOLLOWING AND
> 2147483647 PRECEDING)], (tok_function when (= (tok_table_or_col lochierarchy)
> 0) (tok_table_or_col s_state))=[when(=($5, 0), $0)])
> HiveAggregate(group=[{0, 1}], groups=[[{0, 1}, {0}, {}]],
> indicator=[true], agg#0=[sum($2)], GROUPING__ID=[GROUPING__ID()])
> HiveProject($f0=[$7], $f1=[$6], $f2=[$1])
> HiveJoin(condition=[=($5, $2)], joinType=[inner], algorithm=[none],
> cost=[{1177.2086187101072 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($3, $0)], joinType=[inner],
> algorithm=[none], cost=[{2880430.428726483 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_net_profit=[$21],
> ss_store_sk=[$22])
> HiveTableScan(table=[[tpcds.store_sales]])
> HiveProject(d_date_sk=[$0], d_month_seq=[$3])
> HiveFilter(condition=[between(false, $3, 1193, +(1193, 11))])
> HiveTableScan(table=[[tpcds.date_dim]])
> HiveProject(s_store_sk=[$0], s_county=[$1], s_state=[$2])
> SemiJoin(condition=[=($2, $3)], joinType=[inner])
> Hiv
[jira] [Commented] (HIVE-10458) Enable parallel order by for spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539875#comment-14539875 ] Xuefu Zhang commented on HIVE-10458: Hi [~lirui], it doesn't seem making sense to have parallel order and then reduce again with one reducer. Thus, disabling parallel order for order by + limit seems better. As a side question, I remember that you mentioned that parallel order doesn't help much on performance. Could you quantify that? If so, maybe we shouldn't consider parallel order at all. > Enable parallel order by for spark [Spark Branch] > - > > Key: HIVE-10458 > URL: https://issues.apache.org/jira/browse/HIVE-10458 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-10458.1-spark.patch > > > We don't have to force reducer# to 1 as spark supports parallel sorting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10643) Refactoring Windowing for sum() to pass WindowFrameDef instead of two numbers (1 for number of preceding and 1 for number of following)
[
https://issues.apache.org/jira/browse/HIVE-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539844#comment-14539844
]
Aihua Xu commented on HIVE-10643:
-
Unrelated failures.
> Refactoring Windowing for sum() to pass WindowFrameDef instead of two numbers
> (1 for number of preceding and 1 for number of following)
> ---
>
> Key: HIVE-10643
> URL: https://issues.apache.org/jira/browse/HIVE-10643
> Project: Hive
> Issue Type: Sub-task
> Components: PTF-Windowing
>Affects Versions: 1.3.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Minor
> Attachments: HIVE-10643.patch
>
>
> The functionality should not be affected. Instead of passing 2 numbers (1 for
> # of preceding rows and 1 for # of following rows), we will pass
> WindowFrameDef object around. In the following subtasks, it will be used to
> support additional window like {{rows between x preceding and y preceding}}
> and {{rows between x following and y following}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10685) Alter table concatenate oparetor will cause duplicate data
[
https://issues.apache.org/jira/browse/HIVE-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-10685:
Affects Version/s: 1.1.0
> Alter table concatenate oparetor will cause duplicate data
> --
>
> Key: HIVE-10685
> URL: https://issues.apache.org/jira/browse/HIVE-10685
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: guoliming
>
> "Orders" table has 15 rows and stored as ORC.
> {noformat}
> hive> select count(*) from orders;
> OK
> 15
> Time taken: 37.692 seconds, Fetched: 1 row(s)
> {noformat}
> The table contain 14 files,the size of each file is about 2.1 ~ 3.2 GB.
> After executing command : ALTER TABLE orders CONCATENATE;
> The table is already 1530115000 rows.
> My hive version is 1.1.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10685) Alter table concatenate oparetor will cause duplicate data
[
https://issues.apache.org/jira/browse/HIVE-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-10685:
Description:
"Orders" table has 15 rows and stored as ORC.
{noformat}
hive> select count(*) from orders;
OK
15
Time taken: 37.692 seconds, Fetched: 1 row(s)
{noformat}
The table contain 14 files,the size of each file is about 2.1 ~ 3.2 GB.
After executing command : ALTER TABLE orders CONCATENATE;
The table is already 1530115000 rows.
My hive version is 1.1.0.
was:
"Orders" table has 15 rows and stored as ORC.
hive> select count(*) from orders;
OK
15
Time taken: 37.692 seconds, Fetched: 1 row(s)
The table contain 14 files,the size of each file is about 2.1 ~ 3.2 GB.
After executing command : ALTER TABLE orders CONCATENATE;
The table is already 1530115000 rows.
My hive version is 1.1.0.
> Alter table concatenate oparetor will cause duplicate data
> --
>
> Key: HIVE-10685
> URL: https://issues.apache.org/jira/browse/HIVE-10685
> Project: Hive
> Issue Type: Bug
>Reporter: guoliming
>
> "Orders" table has 15 rows and stored as ORC.
> {noformat}
> hive> select count(*) from orders;
> OK
> 15
> Time taken: 37.692 seconds, Fetched: 1 row(s)
> {noformat}
> The table contain 14 files,the size of each file is about 2.1 ~ 3.2 GB.
> After executing command : ALTER TABLE orders CONCATENATE;
> The table is already 1530115000 rows.
> My hive version is 1.1.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10686) java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
[
https://issues.apache.org/jira/browse/HIVE-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-10686:
---
Attachment: HIVE-10686.01.patch
> java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
> -
>
> Key: HIVE-10686
> URL: https://issues.apache.org/jira/browse/HIVE-10686
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10686.01.patch, HIVE-10686.patch
>
>
> CBO throws Index out of bound exception for TPC-DS Q70.
> Query
> {code}
> explain
> select
> sum(ss_net_profit) as total_sum
>,s_state
>,s_county
>,grouping__id as lochierarchy
>, rank() over(partition by grouping__id, case when grouping__id == 2 then
> s_state end order by sum(ss_net_profit)) as rank_within_parent
> from
> store_sales ss join date_dim d1 on d1.d_date_sk = ss.ss_sold_date_sk
> join store s on s.s_store_sk = ss.ss_store_sk
> where
> d1.d_month_seq between 1193 and 1193+11
> and s.s_state in
> ( select s_state
>from (select s_state as s_state, sum(ss_net_profit),
> rank() over ( partition by s_state order by
> sum(ss_net_profit) desc) as ranking
> from store_sales, store, date_dim
> where d_month_seq between 1193 and 1193+11
> and date_dim.d_date_sk =
> store_sales.ss_sold_date_sk
> and store.s_store_sk = store_sales.ss_store_sk
> group by s_state
> ) tmp1
>where ranking <= 5
> )
> group by s_state,s_county with rollup
> order by
>lochierarchy desc
> ,case when lochierarchy = 0 then s_state end
> ,rank_within_parent
> limit 100
> {code}
> Original plan (correct)
> {code}
> HiveSort(fetch=[100])
> HiveSort(sort0=[$3], sort1=[$5], sort2=[$4], dir0=[DESC], dir1=[ASC],
> dir2=[ASC])
> HiveProject(total_sum=[$4], s_state=[$0], s_county=[$1],
> lochierarchy=[$5], rank_within_parent=[rank() OVER (PARTITION BY $5,
> when(==($5, 2), $0) ORDER BY $4 ROWS BETWEEN 2147483647 FOLLOWING AND
> 2147483647 PRECEDING)], (tok_function when (= (tok_table_or_col lochierarchy)
> 0) (tok_table_or_col s_state))=[when(=($5, 0), $0)])
> HiveAggregate(group=[{0, 1}], groups=[[{0, 1}, {0}, {}]],
> indicator=[true], agg#0=[sum($2)], GROUPING__ID=[GROUPING__ID()])
> HiveProject($f0=[$7], $f1=[$6], $f2=[$1])
> HiveJoin(condition=[=($5, $2)], joinType=[inner], algorithm=[none],
> cost=[{1177.2086187101072 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($3, $0)], joinType=[inner],
> algorithm=[none], cost=[{2880430.428726483 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_net_profit=[$21],
> ss_store_sk=[$22])
> HiveTableScan(table=[[tpcds.store_sales]])
> HiveProject(d_date_sk=[$0], d_month_seq=[$3])
> HiveFilter(condition=[between(false, $3, 1193, +(1193, 11))])
> HiveTableScan(table=[[tpcds.date_dim]])
> HiveProject(s_store_sk=[$0], s_county=[$1], s_state=[$2])
> SemiJoin(condition=[=($2, $3)], joinType=[inner])
> HiveProject(s_store_sk=[$0], s_county=[$23], s_state=[$24])
> HiveTableScan(table=[[tpcds.store]])
> HiveProject(s_state=[$0])
> HiveFilter(condition=[<=($1, 5)])
> HiveProject((tok_table_or_col s_state)=[$0],
> rank_window_0=[rank() OVER (PARTITION BY $0 ORDER BY $1 DESC ROWS BETWEEN
> 2147483647 FOLLOWING AND 2147483647 PRECEDING)])
> HiveAggregate(group=[{0}], agg#0=[sum($1)])
> HiveProject($f0=[$6], $f1=[$1])
> HiveJoin(condition=[=($5, $2)], joinType=[inner],
> algorithm=[none], cost=[{1177.2086187101072 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($3, $0)], joinType=[inner],
> algorithm=[none], cost=[{2880430.428726483 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0],
> ss_net_profit=[$21], ss_store_sk=[$22])
> HiveTableScan(table=[[tpcds.store_sales]])
> HiveProject(d_date_sk=[$0], d_month_seq=[$3])
> HiveFilter(condition=[between(false, $3,
> 1193, +(1193, 11))])
> HiveTableScan(table=[[tpcds.date_dim]])
> HiveProject(s_store_sk=[$0], s_state=[$24])
> HiveTableScan(table=[[tpcds.store]])
> {code}
> Plan after fixTo
[jira] [Commented] (HIVE-5733) Publish hive-exec artifact without all the dependencies
[
https://issues.apache.org/jira/browse/HIVE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539789#comment-14539789
]
Steve Loughran commented on HIVE-5733:
--
[~amareshwari] : what problems did you see with this patch? And have you had a
go at doing another iteration to fix them?
> Publish hive-exec artifact without all the dependencies
> ---
>
> Key: HIVE-5733
> URL: https://issues.apache.org/jira/browse/HIVE-5733
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
>Assignee: Amareshwari Sriramadasu
> Attachments: HIVE-5733.1.patch
>
>
> Currently the artifact {{hive-exec}} that is available in
> [maven|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-exec/0.12.0/hive-exec-0.12.0.jar]
> is shading all the dependencies (= the jar contains all Hive's
> dependencies). As other projects that are depending on Hive might be use
> slightly different version of the dependencies, it can easily happens that
> Hive's shaded version will be used instead which leads to very time consuming
> debugging of what is happening (for example SQOOP-1198).
> Would it be feasible publish {{hive-exec}} jar that will be build without
> shading any dependency? For example
> [avro-tools|http://search.maven.org/#artifactdetails%7Corg.apache.avro%7Cavro-tools%7C1.7.5%7Cjar]
> is having classifier "nodeps" that represents artifact without any
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10686) java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
[
https://issues.apache.org/jira/browse/HIVE-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539767#comment-14539767
]
Hive QA commented on HIVE-10686:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12732211/HIVE-10686.patch
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 8921 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_gby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_global_limit
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_limit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_gby
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_limit
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3860/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3860/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3860/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12732211 - PreCommit-HIVE-TRUNK-Build
> java.lang.IndexOutOfBoundsException for query with rank() over(partition ...)
> -
>
> Key: HIVE-10686
> URL: https://issues.apache.org/jira/browse/HIVE-10686
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10686.patch
>
>
> CBO throws Index out of bound exception for TPC-DS Q70.
> Query
> {code}
> explain
> select
> sum(ss_net_profit) as total_sum
>,s_state
>,s_county
>,grouping__id as lochierarchy
>, rank() over(partition by grouping__id, case when grouping__id == 2 then
> s_state end order by sum(ss_net_profit)) as rank_within_parent
> from
> store_sales ss join date_dim d1 on d1.d_date_sk = ss.ss_sold_date_sk
> join store s on s.s_store_sk = ss.ss_store_sk
> where
> d1.d_month_seq between 1193 and 1193+11
> and s.s_state in
> ( select s_state
>from (select s_state as s_state, sum(ss_net_profit),
> rank() over ( partition by s_state order by
> sum(ss_net_profit) desc) as ranking
> from store_sales, store, date_dim
> where d_month_seq between 1193 and 1193+11
> and date_dim.d_date_sk =
> store_sales.ss_sold_date_sk
> and store.s_store_sk = store_sales.ss_store_sk
> group by s_state
> ) tmp1
>where ranking <= 5
> )
> group by s_state,s_county with rollup
> order by
>lochierarchy desc
> ,case when lochierarchy = 0 then s_state end
> ,rank_within_parent
> limit 100
> {code}
> Original plan (correct)
> {code}
> HiveSort(fetch=[100])
> HiveSort(sort0=[$3], sort1=[$5], sort2=[$4], dir0=[DESC], dir1=[ASC],
> dir2=[ASC])
> HiveProject(total_sum=[$4], s_state=[$0], s_county=[$1],
> lochierarchy=[$5], rank_within_parent=[rank() OVER (PARTITION BY $5,
> when(==($5, 2), $0) ORDER BY $4 ROWS BETWEEN 2147483647 FOLLOWING AND
> 2147483647 PRECEDING)], (tok_function when (= (tok_table_or_col lochierarchy)
> 0) (tok_table_or_col s_state))=[when(=($5, 0), $0)])
> HiveAggregate(group=[{0, 1}], groups=[[{0, 1}, {0}, {}]],
> indicator=[true], agg#0=[sum($2)], GROUPING__ID=[GROUPING__ID()])
> HiveProject($f0=[$7], $f1=[$6], $f2=[$1])
> HiveJoin(condition=[=($5, $2)], joinType=[inner], algorithm=[none],
> cost=[{1177.2086187101072 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($3, $0)], joinType=[inner],
> algorithm=[none], cost=[{2880430.428726483 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_net_profit=[$21],
> ss_store_sk=[$22])
> HiveTableScan(table=[[tpcds.store_sales]])
> HiveProject(d_date_sk=[$0], d_month_seq=[$3])
> HiveFilter(condit
[jira] [Commented] (HIVE-10676) Update Hive's README to mention spark, and to remove jdk1.6
[ https://issues.apache.org/jira/browse/HIVE-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539760#comment-14539760 ] Lefty Leverenz commented on HIVE-10676: --- Doc note: The wiki needs to mention that Java 1.6 isn't supported for Hive 1.2. This belongs in Getting Started and Installing Hive. * [Getting Started -- Requirements | https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-Requirements] * [Installing Hive | https://cwiki.apache.org/confluence/display/Hive/AdminManual+Installation#AdminManualInstallation-InstallingHive] > Update Hive's README to mention spark, and to remove jdk1.6 > --- > > Key: HIVE-10676 > URL: https://issues.apache.org/jira/browse/HIVE-10676 > Project: Hive > Issue Type: Task >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan >Priority: Trivial > Labels: TODOC1.2 > Fix For: 1.2.0 > > Attachments: HIVE-10676.2.patch, HIVE-10676.patch > > > a) Hive's README file mentions only 2 execution frameworks, and does not > mention spark. We should add that in. > b) We should remove jdk1.6 from the README, since hive no longer supports or > even compiles under jdk1.6. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10676) Update Hive's README to mention spark, and to remove jdk1.6
[ https://issues.apache.org/jira/browse/HIVE-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-10676: -- Labels: TODOC1.2 (was: ) > Update Hive's README to mention spark, and to remove jdk1.6 > --- > > Key: HIVE-10676 > URL: https://issues.apache.org/jira/browse/HIVE-10676 > Project: Hive > Issue Type: Task >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan >Priority: Trivial > Labels: TODOC1.2 > Fix For: 1.2.0 > > Attachments: HIVE-10676.2.patch, HIVE-10676.patch > > > a) Hive's README file mentions only 2 execution frameworks, and does not > mention spark. We should add that in. > b) We should remove jdk1.6 from the README, since hive no longer supports or > even compiles under jdk1.6. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
