[jira] [Commented] (HIVE-10209) FetchTask with VC may fail because ExecMapper.done is true
[
https://issues.apache.org/jira/browse/HIVE-10209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907678#comment-14907678
]
Lefty Leverenz commented on HIVE-10209:
---
Version note: This was also committed to branch-1.0 (for release 1.0.2) on
September 24th with commit 2801d2c4b1a61315ae7f28c0ea825580e30f411b.
> FetchTask with VC may fail because ExecMapper.done is true
> --
>
> Key: HIVE-10209
> URL: https://issues.apache.org/jira/browse/HIVE-10209
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.1.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 1.2.0
>
> Attachments: HIVE-10209.1-spark.patch, HIVE-10209.2-spark.patch
>
>
> ExecMapper.done is a static variable, and may cause issues in the following
> example:
> {code}
> set hive.fetch.task.conversion=minimal;
> select * from src where key < 10 limit 1;
> set hive.fetch.task.conversion=more;
> select *, BLOCK__OFFSET_INSIDE__FILE from src where key < 10;
> {code}
> The second select won't return any result, if running in local mode.
> The issue is, the first select query will be converted to a MapRedTask with
> only a mapper. And, when the task is done, because of the limit operator,
> ExecMapper.done will be set to true.
> Then, when the second select query begin to execute, it will call
> {{FetchOperator::getRecordReader()}}, and since here we have virtual column,
> an instance of {{HiveRecordReader}} will be returned. The problem is,
> {{HiveRecordReader::doNext()}} will check ExecMapper.done. In this case,
> since the value is true, it will quit immediately.
> In short, I think making ExecMapper.done static is a bad idea. The first
> query should in no way affect the second one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11943) Set old CLI as the default Client when using hive script
[ https://issues.apache.org/jira/browse/HIVE-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907677#comment-14907677 ] Ferdinand Xu commented on HIVE-11943: - Thanks [~leftylev] for figuring it out. I have updated the related section in the wiki. Please help me review it. Thank you! > Set old CLI as the default Client when using hive script > > > Key: HIVE-11943 > URL: https://issues.apache.org/jira/browse/HIVE-11943 > Project: Hive > Issue Type: Sub-task > Components: CLI >Affects Versions: beeline-cli-branch >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: beeline-cli-branch > > Attachments: HIVE-11943.1-beeline-cli.patch > > > Since we have some concerns about deprecating the current CLI, we will set > the old CLI as default. Once we resolve the problems, we will set the new CLI > as default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907667#comment-14907667
]
WangMeng commented on HIVE-11880:
-
[~xuefuz]I have rebased it and uploaded a new patch.
[~ashutoshc] I tried the patch of HIVE-11919 again after I rebased, it can not
fix this bug also.
Please check it again . Thanks.
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch,
> HIVE-11880.03.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangMeng updated HIVE-11880:
Attachment: HIVE-11880.03.patch
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch,
> HIVE-11880.03.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11943) Set old CLI as the default Client when using hive script
[ https://issues.apache.org/jira/browse/HIVE-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907654#comment-14907654 ] Lefty Leverenz commented on HIVE-11943: --- This should be documented in the wiki when it is merged to master (unless the default gets changed by then). * [Replacing the Implementation of Hive CLI Using Beeline | https://cwiki.apache.org/confluence/display/Hive/Replacing+the+Implementation+of+Hive+CLI+Using+Beeline] Now that the doc-tracking jira HIVE-10810 is closed, we don't have a way to flag doc issues for the beeline-cli branch. > Set old CLI as the default Client when using hive script > > > Key: HIVE-11943 > URL: https://issues.apache.org/jira/browse/HIVE-11943 > Project: Hive > Issue Type: Sub-task > Components: CLI >Affects Versions: beeline-cli-branch >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: beeline-cli-branch > > Attachments: HIVE-11943.1-beeline-cli.patch > > > Since we have some concerns about deprecating the current CLI, we will set > the old CLI as default. Once we resolve the problems, we will set the new CLI > as default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11945) ORC with non-local reads may not be reusing connection to DN
[ https://issues.apache.org/jira/browse/HIVE-11945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-11945: Attachment: HIVE-11945.3.patch HIVE-11945.2.patch Uploading the patch with zcr changes. [~prasanth_j] - S3AInputStream should actually have a lazy seek() implementation which can work well with readFully. This is followed in PrestoS3FileSystem as well. Will create a separate bug in HDFS to track this. > ORC with non-local reads may not be reusing connection to DN > > > Key: HIVE-11945 > URL: https://issues.apache.org/jira/browse/HIVE-11945 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: HIVE-11945.1.patch, HIVE-11945.2.patch, > HIVE-11945.3.patch > > > When “seek + readFully(buffer, offset, length)” is used, DFSInputStream ends > up going via “readWithStrategy()”. This sets up BlockReader with length > equivalent to that of the block size. So until this position is reached, > RemoteBlockReader2.peer would not be added to the PeerCache (Plz refer > RemoteBlockReader2.close() in HDFS). So eventually the next call to the same > DN would end opening a new socket. In ORC, when it is not a data local read, > this has a the possibility of opening/closing lots of connections with DN. > In random reads, it would be good to set this length to the amount of data > that is to be read (e.g pread call in DFSInputStream which sets up the > BlockReader’s length correctly & the code path returns the Peer back to peer > cache properly). “readFully(position, buffer, offset, length)” follows this > code path and ends up reusing the connections properly. Creating this JIRA to > fix this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11473) Upgrade Spark dependency to 1.5 [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907590#comment-14907590 ] Xuefu Zhang commented on HIVE-11473: The tarball is published here: http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz > Upgrade Spark dependency to 1.5 [Spark Branch] > -- > > Key: HIVE-11473 > URL: https://issues.apache.org/jira/browse/HIVE-11473 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Jimmy Xiang >Assignee: Rui Li > Attachments: HIVE-11473.1-spark.patch > > > In Spark 1.5, SparkListener interface is changed. So HoS may fail to create > the spark client if the un-implemented event callback method is invoked. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11642) LLAP: make sure tests pass #3
[
https://issues.apache.org/jira/browse/HIVE-11642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907577#comment-14907577
]
Hive QA commented on HIVE-11642:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12762284/HIVE-11642.10.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9634 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.initializationError
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5407/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5407/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5407/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12762284 - PreCommit-HIVE-TRUNK-Build
> LLAP: make sure tests pass #3
> -
>
> Key: HIVE-11642
> URL: https://issues.apache.org/jira/browse/HIVE-11642
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11642.01.patch, HIVE-11642.02.patch,
> HIVE-11642.03.patch, HIVE-11642.04.patch, HIVE-11642.05.patch,
> HIVE-11642.08.patch, HIVE-11642.09.patch, HIVE-11642.10.patch,
> HIVE-11642.patch
>
>
> Tests should pass against the most recent branch and Tez 0.8.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11791) Add unit test for HIVE-10122
[ https://issues.apache.org/jira/browse/HIVE-11791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907572#comment-14907572 ] Ashutosh Chauhan commented on HIVE-11791: - +1 > Add unit test for HIVE-10122 > > > Key: HIVE-11791 > URL: https://issues.apache.org/jira/browse/HIVE-11791 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 1.1.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy >Priority: Minor > Attachments: HIVE-11791.2.patch, HIVE-11791.3.patch, HIVE-11791.patch > > > Unit tests for PartitionPruner.compactExpr() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10571) HiveMetaStoreClient should close existing thrift connection before its reconnect
[ https://issues.apache.org/jira/browse/HIVE-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907567#comment-14907567 ] Lefty Leverenz commented on HIVE-10571: --- This has also been committed to branch-1.2 and branch-1.0 for upcoming releases 1.2.2 and 1.0.2. The commit to branch-1.2 is 37e1f833ea1aece3b1e744e6d6db55fdb1514573 (Sept. 1) and the commit to branch-1.0 is 7bc067a64802fbaa72996e8ef41e2134d71c5c14 (Sept 24). Fix Version/s currently says "1.3.0, 2.0.0" but I don't find a commit to branch-1 for 1.3.0 in email, searching by jira number or subject line. Is it in the branch? > HiveMetaStoreClient should close existing thrift connection before its > reconnect > > > Key: HIVE-10571 > URL: https://issues.apache.org/jira/browse/HIVE-10571 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-10571.patch, HIVE-10571.patch, HIVE-10571.patch > > > HiveMetaStoreClient should first close its existing thrift connection, no > matter it is already dead or still live, before its opening another > connection in its reconnect() method. Otherwise, it might lead to resource > huge accumulation or leak at HMS site when client keeps on retrying. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10785) Support aggregate push down through joins
[
https://issues.apache.org/jira/browse/HIVE-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907550#comment-14907550
]
Lefty Leverenz commented on HIVE-10785:
---
Doc note: This adds configuration parameter *hive.transpose.aggr.join* to
HiveConf.java, so it will need to be documented in the wiki for release 2.0.0.
* [Configuration Properties -- Query and DDL Execution (put at end of the
section) |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution]
> Support aggregate push down through joins
> -
>
> Key: HIVE-10785
> URL: https://issues.apache.org/jira/browse/HIVE-10785
> Project: Hive
> Issue Type: New Feature
> Components: CBO, Logical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Ashutosh Chauhan
> Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-10785.2.patch, HIVE-10785.3.patch,
> HIVE-10785.4.patch, HIVE-10785.5.patch, HIVE-10785.6.patch, HIVE-10785.patch
>
>
> Enable {{AggregateJoinTransposeRule}} in CBO that pushes Aggregate through
> Join operators. The rule has been extended in Calcite 1.4 to cover complex
> cases e.g. Aggregate operators comprising UDAF. The decision on whether to
> push the Aggregate through Join or not should be cost-driven.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10785) Support aggregate push down through joins
[
https://issues.apache.org/jira/browse/HIVE-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-10785:
--
Labels: TODOC2.0 (was: )
> Support aggregate push down through joins
> -
>
> Key: HIVE-10785
> URL: https://issues.apache.org/jira/browse/HIVE-10785
> Project: Hive
> Issue Type: New Feature
> Components: CBO, Logical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Ashutosh Chauhan
> Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-10785.2.patch, HIVE-10785.3.patch,
> HIVE-10785.4.patch, HIVE-10785.5.patch, HIVE-10785.6.patch, HIVE-10785.patch
>
>
> Enable {{AggregateJoinTransposeRule}} in CBO that pushes Aggregate through
> Join operators. The rule has been extended in Calcite 1.4 to cover complex
> cases e.g. Aggregate operators comprising UDAF. The decision on whether to
> push the Aggregate through Join or not should be cost-driven.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11473) Upgrade Spark dependency to 1.5 [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907539#comment-14907539 ] Xuefu Zhang commented on HIVE-11473: Thanks, Rui. I'm building it. However, publishing it might take a little bit longer as I don't have the credentials. I will update once done. > Upgrade Spark dependency to 1.5 [Spark Branch] > -- > > Key: HIVE-11473 > URL: https://issues.apache.org/jira/browse/HIVE-11473 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Jimmy Xiang >Assignee: Rui Li > Attachments: HIVE-11473.1-spark.patch > > > In Spark 1.5, SparkListener interface is changed. So HoS may fail to create > the spark client if the un-implemented event callback method is invoked. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11791) Add unit test for HIVE-10122
[ https://issues.apache.org/jira/browse/HIVE-11791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907538#comment-14907538 ] Illya Yalovyy commented on HIVE-11791: -- Those tests have been failing for a couple of builds already. They don't seem to be related. Can we proceed? > Add unit test for HIVE-10122 > > > Key: HIVE-11791 > URL: https://issues.apache.org/jira/browse/HIVE-11791 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 1.1.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy >Priority: Minor > Attachments: HIVE-11791.2.patch, HIVE-11791.3.patch, HIVE-11791.patch > > > Unit tests for PartitionPruner.compactExpr() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11930) how to prevent ppd the topN(a) udf predication in where clause?
[ https://issues.apache.org/jira/browse/HIVE-11930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feng Yuan updated HIVE-11930: - Priority: Minor (was: Blocker) > how to prevent ppd the topN(a) udf predication in where clause? > --- > > Key: HIVE-11930 > URL: https://issues.apache.org/jira/browse/HIVE-11930 > Project: Hive > Issue Type: New Feature > Components: Hive >Affects Versions: 0.14.0 >Reporter: Feng Yuan >Priority: Minor > > select > a.state_date,a.customer,a.taskid,a.step_id,a.exit_title,a.pv,top1000(a.only_id) > from > ( select > t1.state_date,t1.customer,t1.taskid,t1.step_id,t1.exit_title,t1.pv,t1.only_id > from > ( select t11.state_date, >t11.customer, >t11.taskid, >t11.step_id, >t11.exit_title, >t11.pv, >concat(t11.customer,t11.taskid,t11.step_id) as > only_id >from > ( select > state_date,customer,taskid,step_id,exit_title,count(*) as pv > from bdi_fact2.mid_url_step > where exit_url!='-1' > and exit_title !='-1' > and l_date='2015-08-31' > group by > state_date,customer,taskid,step_id,exit_title > )t11 >)t1 >order by t1.only_id,t1.pv desc > )a > where a.customer='Cdianyingwang' > and a.taskid='33' > and a.step_id='0' > and top1000(a.only_id)<=10; > in above example: > outer top1000(a.only_id)<=10;will ppd to: > stage 1: > ( select t11.state_date, >t11.customer, >t11.taskid, >t11.step_id, >t11.exit_title, >t11.pv, >concat(t11.customer,t11.taskid,t11.step_id) as > only_id >from > ( select > state_date,customer,taskid,step_id,exit_title,count(*) as pv > from bdi_fact2.mid_url_step > where exit_url!='-1' > and exit_title !='-1' > and l_date='2015-08-31' > group by > state_date,customer,taskid,step_id,exit_title > )t11 >)t1 > and this stage have 2 reduce,so you can see this will output 20 records, > upon to outer stage,the final results is exactly this 20 records. > so i want to know is there any way to hint this topN udf predication not to > ppd? > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11930) how to prevent ppd the topN(a) udf predication in where clause?
[ https://issues.apache.org/jira/browse/HIVE-11930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feng Yuan updated HIVE-11930: - Issue Type: New Feature (was: Bug) > how to prevent ppd the topN(a) udf predication in where clause? > --- > > Key: HIVE-11930 > URL: https://issues.apache.org/jira/browse/HIVE-11930 > Project: Hive > Issue Type: New Feature > Components: Hive >Affects Versions: 0.14.0 >Reporter: Feng Yuan >Priority: Blocker > > select > a.state_date,a.customer,a.taskid,a.step_id,a.exit_title,a.pv,top1000(a.only_id) > from > ( select > t1.state_date,t1.customer,t1.taskid,t1.step_id,t1.exit_title,t1.pv,t1.only_id > from > ( select t11.state_date, >t11.customer, >t11.taskid, >t11.step_id, >t11.exit_title, >t11.pv, >concat(t11.customer,t11.taskid,t11.step_id) as > only_id >from > ( select > state_date,customer,taskid,step_id,exit_title,count(*) as pv > from bdi_fact2.mid_url_step > where exit_url!='-1' > and exit_title !='-1' > and l_date='2015-08-31' > group by > state_date,customer,taskid,step_id,exit_title > )t11 >)t1 >order by t1.only_id,t1.pv desc > )a > where a.customer='Cdianyingwang' > and a.taskid='33' > and a.step_id='0' > and top1000(a.only_id)<=10; > in above example: > outer top1000(a.only_id)<=10;will ppd to: > stage 1: > ( select t11.state_date, >t11.customer, >t11.taskid, >t11.step_id, >t11.exit_title, >t11.pv, >concat(t11.customer,t11.taskid,t11.step_id) as > only_id >from > ( select > state_date,customer,taskid,step_id,exit_title,count(*) as pv > from bdi_fact2.mid_url_step > where exit_url!='-1' > and exit_title !='-1' > and l_date='2015-08-31' > group by > state_date,customer,taskid,step_id,exit_title > )t11 >)t1 > and this stage have 2 reduce,so you can see this will output 20 records, > upon to outer stage,the final results is exactly this 20 records. > so i want to know is there any way to hint this topN udf predication not to > ppd? > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11930) how to prevent ppd the topN(a) udf predication in where clause?
[ https://issues.apache.org/jira/browse/HIVE-11930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907523#comment-14907523 ] Feng Yuan commented on HIVE-11930: -- thanks your reply so much!! > how to prevent ppd the topN(a) udf predication in where clause? > --- > > Key: HIVE-11930 > URL: https://issues.apache.org/jira/browse/HIVE-11930 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.14.0 >Reporter: Feng Yuan >Priority: Blocker > > select > a.state_date,a.customer,a.taskid,a.step_id,a.exit_title,a.pv,top1000(a.only_id) > from > ( select > t1.state_date,t1.customer,t1.taskid,t1.step_id,t1.exit_title,t1.pv,t1.only_id > from > ( select t11.state_date, >t11.customer, >t11.taskid, >t11.step_id, >t11.exit_title, >t11.pv, >concat(t11.customer,t11.taskid,t11.step_id) as > only_id >from > ( select > state_date,customer,taskid,step_id,exit_title,count(*) as pv > from bdi_fact2.mid_url_step > where exit_url!='-1' > and exit_title !='-1' > and l_date='2015-08-31' > group by > state_date,customer,taskid,step_id,exit_title > )t11 >)t1 >order by t1.only_id,t1.pv desc > )a > where a.customer='Cdianyingwang' > and a.taskid='33' > and a.step_id='0' > and top1000(a.only_id)<=10; > in above example: > outer top1000(a.only_id)<=10;will ppd to: > stage 1: > ( select t11.state_date, >t11.customer, >t11.taskid, >t11.step_id, >t11.exit_title, >t11.pv, >concat(t11.customer,t11.taskid,t11.step_id) as > only_id >from > ( select > state_date,customer,taskid,step_id,exit_title,count(*) as pv > from bdi_fact2.mid_url_step > where exit_url!='-1' > and exit_title !='-1' > and l_date='2015-08-31' > group by > state_date,customer,taskid,step_id,exit_title > )t11 >)t1 > and this stage have 2 reduce,so you can see this will output 20 records, > upon to outer stage,the final results is exactly this 20 records. > so i want to know is there any way to hint this topN udf predication not to > ppd? > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11831) TXN tables in Oracle should be created with ROWDEPENDENCIES
[ https://issues.apache.org/jira/browse/HIVE-11831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-11831: -- Labels: TODOC2.0 (was: ) > TXN tables in Oracle should be created with ROWDEPENDENCIES > --- > > Key: HIVE-11831 > URL: https://issues.apache.org/jira/browse/HIVE-11831 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-11831.01.patch, HIVE-11831.patch > > > These frequently-updated tables may otherwise suffer from spurious deadlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11831) TXN tables in Oracle should be created with ROWDEPENDENCIES
[ https://issues.apache.org/jira/browse/HIVE-11831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907487#comment-14907487 ] Lefty Leverenz commented on HIVE-11831: --- I suggest a small section in the Transactions wikidoc, with a new Limitations bullet pointing to the section. * [Hive Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions] * [Hive Transactions -- Limitations | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Limitations] > TXN tables in Oracle should be created with ROWDEPENDENCIES > --- > > Key: HIVE-11831 > URL: https://issues.apache.org/jira/browse/HIVE-11831 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.0.0 > > Attachments: HIVE-11831.01.patch, HIVE-11831.patch > > > These frequently-updated tables may otherwise suffer from spurious deadlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11791) Add unit test for HIVE-10122
[
https://issues.apache.org/jira/browse/HIVE-11791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907482#comment-14907482
]
Hive QA commented on HIVE-11791:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12761944/HIVE-11791.3.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9619 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.streaming.TestStreaming.testAddPartition
org.apache.hive.hcatalog.streaming.TestStreaming.testRemainingTransactions
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5406/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5406/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5406/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12761944 - PreCommit-HIVE-TRUNK-Build
> Add unit test for HIVE-10122
>
>
> Key: HIVE-11791
> URL: https://issues.apache.org/jira/browse/HIVE-11791
> Project: Hive
> Issue Type: Test
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
>Priority: Minor
> Attachments: HIVE-11791.2.patch, HIVE-11791.3.patch, HIVE-11791.patch
>
>
> Unit tests for PartitionPruner.compactExpr()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine
[ https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907436#comment-14907436 ] Rui Li commented on HIVE-11887: --- Hi [~sershe], do you mean UDFExampleAdd.jar needs to be created under /tmp when building hive on spark? I just tried building on my side but there isn't such a jar under /tmp. > spark tests break the build on a shared machine > --- > > Key: HIVE-11887 > URL: https://issues.apache.org/jira/browse/HIVE-11887 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > Spark download creates UDFExampleAdd jar in /tmp; when building on a shared > machine, someone else's jar from a build prevents this jar from being created > (I have no permissions to this file because it was created by a different > user) and the build fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11941) Update committer list
[ https://issues.apache.org/jira/browse/HIVE-11941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907418#comment-14907418 ] Chaoyu Tang commented on HIVE-11941: [~lars_francke] Could you provide your information? > Update committer list > - > > Key: HIVE-11941 > URL: https://issues.apache.org/jira/browse/HIVE-11941 > Project: Hive > Issue Type: Bug >Reporter: Chaoyu Tang >Priority: Minor > Attachments: HIVE-11941.patch > > > Please update the committer list in http://hive.apache.org/people.html: > --- > Name: Chaoyu Tang > Apache ID: ctang > Organization: Cloudera (www.cloudera.com) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11934) Transaction lock retry logic results in infinite loop
[
https://issues.apache.org/jira/browse/HIVE-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-11934:
--
Attachment: HIVE-11934.patch
> Transaction lock retry logic results in infinite loop
> -
>
> Key: HIVE-11934
> URL: https://issues.apache.org/jira/browse/HIVE-11934
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Transactions
>Affects Versions: 1.2.1
>Reporter: Steve Howard
>Assignee: Eugene Koifman
>Priority: Minor
> Attachments: HIVE-11934.patch
>
>
> We reset the deadlock count to 0 every time the lock() method is called in
> org.apache.hadoop.hive.metastore.txn.TxnHandler, so the ten count is never
> reached in checkRetryable().
> We should let checkRetryable handle the deadlock count.
> public LockResponse lock(LockRequest rqst)
> throws NoSuchTxnException, TxnAbortedException, MetaException
> {
> >>>this.deadlockCnt = 0; <<<
> try
> {
> Connection dbConn = null;
> try
> {
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11819) HiveServer2 catches OOMs on request threads
[ https://issues.apache.org/jira/browse/HIVE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907385#comment-14907385 ] Sergey Shelukhin commented on HIVE-11819: - I'd rather do it when needed > HiveServer2 catches OOMs on request threads > --- > > Key: HIVE-11819 > URL: https://issues.apache.org/jira/browse/HIVE-11819 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11819.01.patch, HIVE-11819.02.patch, > HIVE-11819.patch > > > ThriftCLIService methods such as ExecuteStatement are apparently capable of > catching OOMs because they get wrapped in RTE by HiveSessionProxy. > This shouldn't happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11960) parentheses in join conditions are not supported
[ https://issues.apache.org/jira/browse/HIVE-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907383#comment-14907383 ] Sergey Shelukhin commented on HIVE-11960: - [~ashutoshc] do you want to take a look? > parentheses in join conditions are not supported > > > Key: HIVE-11960 > URL: https://issues.apache.org/jira/browse/HIVE-11960 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11960.patch > > > These should be supported; they are ANSI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11831) TXN tables in Oracle should be created with ROWDEPENDENCIES
[ https://issues.apache.org/jira/browse/HIVE-11831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11831: -- Component/s: (was: Hive) Transactions > TXN tables in Oracle should be created with ROWDEPENDENCIES > --- > > Key: HIVE-11831 > URL: https://issues.apache.org/jira/browse/HIVE-11831 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.0.0 > > Attachments: HIVE-11831.01.patch, HIVE-11831.patch > > > These frequently-updated tables may otherwise suffer from spurious deadlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11833) TxnHandler heartbeat txn doesn't need serializable DB txn level
[ https://issues.apache.org/jira/browse/HIVE-11833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11833: -- Component/s: Transactions > TxnHandler heartbeat txn doesn't need serializable DB txn level > --- > > Key: HIVE-11833 > URL: https://issues.apache.org/jira/browse/HIVE-11833 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.0.0 > > Attachments: HIVE-11833.patch > > > What it does is: > 1) Update lock heartbeat time, fails if not found. > 2) Get txn state. > 3) If not found, look for txn in completed, fails regardless of result. > 4) Update txn heartbeat time if not (3) and not aborted. > All this can run the same under repeatable-reads. > Now if it runs under read-committed, someone could > 1) update txn state after we read it > 2) delete txn state (moving to completed) after we read it > 3) same for completed state > In case of 1 we will update heartbeat for e.g. aborted txn without detecting > it. UPD: We can change queries to detect it > In case of 2 the update will produce 0 rows so we will detect that and can > check completed as we already do. > The 3 case seems like it doesn't matter. > I don't know if (1) matters. These heartbeats happen often and can cause > contention on the db -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11831) TXN tables in Oracle should be created with ROWDEPENDENCIES
[ https://issues.apache.org/jira/browse/HIVE-11831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11831: -- Labels: Tra (was: ) > TXN tables in Oracle should be created with ROWDEPENDENCIES > --- > > Key: HIVE-11831 > URL: https://issues.apache.org/jira/browse/HIVE-11831 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: Tra > Fix For: 2.0.0 > > Attachments: HIVE-11831.01.patch, HIVE-11831.patch > > > These frequently-updated tables may otherwise suffer from spurious deadlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11831) TXN tables in Oracle should be created with ROWDEPENDENCIES
[ https://issues.apache.org/jira/browse/HIVE-11831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11831: -- Labels: (was: Tra) > TXN tables in Oracle should be created with ROWDEPENDENCIES > --- > > Key: HIVE-11831 > URL: https://issues.apache.org/jira/browse/HIVE-11831 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.0.0 > > Attachments: HIVE-11831.01.patch, HIVE-11831.patch > > > These frequently-updated tables may otherwise suffer from spurious deadlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11960) parentheses in join conditions are not supported
[ https://issues.apache.org/jira/browse/HIVE-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11960: Attachment: HIVE-11960.patch Simple patch, let's see what it breaks > parentheses in join conditions are not supported > > > Key: HIVE-11960 > URL: https://issues.apache.org/jira/browse/HIVE-11960 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11960.patch > > > These should be supported; they are ANSI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11831) TXN tables in Oracle should be created with ROWDEPENDENCIES
[ https://issues.apache.org/jira/browse/HIVE-11831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11831: -- Component/s: Hive > TXN tables in Oracle should be created with ROWDEPENDENCIES > --- > > Key: HIVE-11831 > URL: https://issues.apache.org/jira/browse/HIVE-11831 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: Tra > Fix For: 2.0.0 > > Attachments: HIVE-11831.01.patch, HIVE-11831.patch > > > These frequently-updated tables may otherwise suffer from spurious deadlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11960) parentheses in join conditions are not supported
[ https://issues.apache.org/jira/browse/HIVE-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-11960: --- Assignee: Sergey Shelukhin > parentheses in join conditions are not supported > > > Key: HIVE-11960 > URL: https://issues.apache.org/jira/browse/HIVE-11960 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > These should be supported; they are ANSI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11132) Queries using join and group by produce incorrect output when hive.auto.convert.join=false and hive.optimize.reducededuplication=true
[ https://issues.apache.org/jira/browse/HIVE-11132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907369#comment-14907369 ] Gopal V commented on HIVE-11132: LGTM - +1 > Queries using join and group by produce incorrect output when > hive.auto.convert.join=false and hive.optimize.reducededuplication=true > - > > Key: HIVE-11132 > URL: https://issues.apache.org/jira/browse/HIVE-11132 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Rich Haase >Assignee: Ashutosh Chauhan > Attachments: HIVE-11132.2.patch, HIVE-11132.patch > > > Queries using join and group by produce multiple output rows with the same > key when hive.auto.convert.join=false and > hive.optimize.reducededuplication=true. This interaction between > configuration parameters is unexpected and should be well documented at the > very least and should likely be considered a bug. > e.g. > hive> set hive.auto.convert.join = false; > hive> set hive.optimize.reducededuplication = true; > hive> SELECT foo.id, count(*) as factor > > FROM foo > > JOIN bar ON (foo.id = bar.id and foo.line_id = bar.line_id) > > JOIN split ON (foo.id = split.id and foo.line_id = split.line_id) > > JOIN forecast ON (foo.id = forecast.id AND foo.line_id = > forecast.line_id) > > WHERE foo.order != ‘blah’ AND foo.id = ‘XYZ' > > GROUP BY foo.id; > XYZ 79 > XYZ 74 > XYZ 297 > XYZ 66 > hive> set hive.auto.convert.join = true; > hive> set hive.optimize.reducededuplication = true; > hive> SELECT foo.id, count(*) as factor > > FROM foo > > JOIN bar ON (foo.id = bar.id and foo.line_id = bar.line_id) > > JOIN split ON (foo.id = split.id and foo.line_id = split.line_id) > > JOIN forecast ON (foo.id = forecast.id AND foo.line_id = > forecast.line_id) > > WHERE foo.order != ‘blah’ AND foo.id = ‘XYZ' > > GROUP BY foo.id; > XYZ 516 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11778) Merge beeline-cli branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-11778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-11778: Attachment: HIVE-11778.1.patch > Merge beeline-cli branch to trunk > - > > Key: HIVE-11778 > URL: https://issues.apache.org/jira/browse/HIVE-11778 > Project: Hive > Issue Type: Sub-task > Components: CLI >Affects Versions: 2.0.0 >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-11778.1.patch, HIVE-11778.patch > > > The team working on the beeline-cli branch would like to merge their work to > trunk. This jira will track that effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-11958) Merge master to beeline-cli branch 09/25/2015
[ https://issues.apache.org/jira/browse/HIVE-11958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu resolved HIVE-11958. - Resolution: Fixed commit 6b3e82d39c159f11829007fbbeff2fdf1216bb78 Merge: 046c5eb 41a12cb Author: Ferdinand Xu Date: Thu Sep 24 21:29:17 2015 -0400 HIVE-11958: Merge branch 'master' into beeline-cli > Merge master to beeline-cli branch 09/25/2015 > - > > Key: HIVE-11958 > URL: https://issues.apache.org/jira/browse/HIVE-11958 > Project: Hive > Issue Type: Sub-task > Components: CLI >Affects Versions: beeline-cli-branch >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: beeline-cli-branch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11959) add simple test case for TestTableIterable
[ https://issues.apache.org/jira/browse/HIVE-11959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907364#comment-14907364 ] Thejas M Nair commented on HIVE-11959: -- [~sushanth] can you please review ? > add simple test case for TestTableIterable > -- > > Key: HIVE-11959 > URL: https://issues.apache.org/jira/browse/HIVE-11959 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-11959.1.patch > > > Adding a test case to TableIterable which was introduced in HIVE-11407 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11959) add simple test case for TestTableIterable
[ https://issues.apache.org/jira/browse/HIVE-11959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-11959: - Attachment: (was: HIVE-11959.1.patch) > add simple test case for TestTableIterable > -- > > Key: HIVE-11959 > URL: https://issues.apache.org/jira/browse/HIVE-11959 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-11959.1.patch > > > Adding a test case to TableIterable which was introduced in HIVE-11407 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11959) add simple test case for TestTableIterable
[ https://issues.apache.org/jira/browse/HIVE-11959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-11959: - Attachment: HIVE-11959.1.patch > add simple test case for TestTableIterable > -- > > Key: HIVE-11959 > URL: https://issues.apache.org/jira/browse/HIVE-11959 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-11959.1.patch > > > Adding a test case to TableIterable which was introduced in HIVE-11407 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11132) Queries using join and group by produce incorrect output when hive.auto.convert.join=false and hive.optimize.reducededuplication=true
[ https://issues.apache.org/jira/browse/HIVE-11132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-11132: Attachment: HIVE-11132.2.patch > Queries using join and group by produce incorrect output when > hive.auto.convert.join=false and hive.optimize.reducededuplication=true > - > > Key: HIVE-11132 > URL: https://issues.apache.org/jira/browse/HIVE-11132 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Rich Haase >Assignee: Ashutosh Chauhan > Attachments: HIVE-11132.2.patch, HIVE-11132.patch > > > Queries using join and group by produce multiple output rows with the same > key when hive.auto.convert.join=false and > hive.optimize.reducededuplication=true. This interaction between > configuration parameters is unexpected and should be well documented at the > very least and should likely be considered a bug. > e.g. > hive> set hive.auto.convert.join = false; > hive> set hive.optimize.reducededuplication = true; > hive> SELECT foo.id, count(*) as factor > > FROM foo > > JOIN bar ON (foo.id = bar.id and foo.line_id = bar.line_id) > > JOIN split ON (foo.id = split.id and foo.line_id = split.line_id) > > JOIN forecast ON (foo.id = forecast.id AND foo.line_id = > forecast.line_id) > > WHERE foo.order != ‘blah’ AND foo.id = ‘XYZ' > > GROUP BY foo.id; > XYZ 79 > XYZ 74 > XYZ 297 > XYZ 66 > hive> set hive.auto.convert.join = true; > hive> set hive.optimize.reducededuplication = true; > hive> SELECT foo.id, count(*) as factor > > FROM foo > > JOIN bar ON (foo.id = bar.id and foo.line_id = bar.line_id) > > JOIN split ON (foo.id = split.id and foo.line_id = split.line_id) > > JOIN forecast ON (foo.id = forecast.id AND foo.line_id = > forecast.line_id) > > WHERE foo.order != ‘blah’ AND foo.id = ‘XYZ' > > GROUP BY foo.id; > XYZ 516 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11959) add simple test case for TestTableIterable
[ https://issues.apache.org/jira/browse/HIVE-11959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-11959: - Attachment: HIVE-11959.1.patch Test Verifies how the calls to metastore client are batched and the number of objects returned > add simple test case for TestTableIterable > -- > > Key: HIVE-11959 > URL: https://issues.apache.org/jira/browse/HIVE-11959 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-11959.1.patch > > > Adding a test case to TableIterable which was introduced in HIVE-11407 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11473) Upgrade Spark dependency to 1.5 [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-11473: -- Attachment: HIVE-11473.1-spark.patch I tried the patch locally with Spark 1.5.0 and passed some simple queries. [~xuefuz], to run the tests, would you mind package a new Spark tar and upload to our repo? Thanks. > Upgrade Spark dependency to 1.5 [Spark Branch] > -- > > Key: HIVE-11473 > URL: https://issues.apache.org/jira/browse/HIVE-11473 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Jimmy Xiang >Assignee: Rui Li > Attachments: HIVE-11473.1-spark.patch > > > In Spark 1.5, SparkListener interface is changed. So HoS may fail to create > the spark client if the un-implemented event callback method is invoked. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11287) Hive Metastore does not tolerate leading spaces in JDBC url
[
https://issues.apache.org/jira/browse/HIVE-11287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Xin Yu updated HIVE-11287:
---
Attachment: HIVE-11287.1.patch
> Hive Metastore does not tolerate leading spaces in JDBC url
> ---
>
> Key: HIVE-11287
> URL: https://issues.apache.org/jira/browse/HIVE-11287
> Project: Hive
> Issue Type: Bug
> Components: Configuration, Metastore
>Affects Versions: 1.2.0, 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Chen Xin Yu
>Priority: Minor
> Labels: newbie
> Attachments: HIVE-11287.1.patch
>
>
> The hive metastore is configured with
> {code}
>
>
>javax.jdo.option.ConnectionURL
>
>
>jdbc:mysql://hostname/hive
>
>
> {code}
> The initialization fails with an error
> {code}
> java.sql.SQLException: No suitable driver found for
> jdbc:mysql://hostname/hive
> at java.sql.DriverManager.getConnection(DriverManager.java:689)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> at
> com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
> at com.jolbox.bonecp.BoneCP.obtainInternalConnection(BoneCP.java:269)
> at
> com.jolbox.bonecp.ConnectionHandle.(ConnectionHandle.java:242)
> at
> com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:85)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4243) Fix column names in FileSinkOperator
[
https://issues.apache.org/jira/browse/HIVE-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907341#comment-14907341
]
Hive QA commented on HIVE-4243:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12761949/HIVE-4243.patch
{color:red}ERROR:{color} -1 due to 54 failed/errored test(s), 9582 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_nonacid_from_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_dynamic_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_tmp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_file_dump
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_boolean
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_char
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_decimal
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_varchar
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_after_multiple_inserts
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_all_types
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_binary_join_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_count_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_data_types
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_string_concat
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_values_dynamic_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_values_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_values_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_decimal
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_after_multiple_inserts
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_types
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_binary_join_groupby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_count_distinct
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_data_types
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_string_concat
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_decimal
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_count_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_data_types
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_string_concat
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.pig.TestHCatStorer.testDateCharTypes[3]
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteDecimalXY[3]
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteDecimalX[3]
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5405/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TR
[jira] [Commented] (HIVE-11915) BoneCP returns closed connections from the pool
[
https://issues.apache.org/jira/browse/HIVE-11915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907338#comment-14907338
]
Thejas M Nair commented on HIVE-11915:
--
Review comments -
- please remove the 2 imports that are no longer relevant to the change
- Change - LOG.error("BONECP returned a closed connection " +
e.getMessage()); to - LOG.warn("Error setting up new connection, getting
another connection from pool", e);
(the pool used might not be boncep; error might not be due to closed
connection; should be warn not error; and should indicate it is being retried;
full exception stack trace can be very useful).
- getConnAttemptCount = 10; Should we keep that generic and have it set to 10
in all cases and not bonecp specific ? It shouldn't hurt.
> BoneCP returns closed connections from the pool
> ---
>
> Key: HIVE-11915
> URL: https://issues.apache.org/jira/browse/HIVE-11915
> Project: Hive
> Issue Type: Bug
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11915.01.patch, HIVE-11915.WIP.patch,
> HIVE-11915.patch
>
>
> It's a very old bug in BoneCP and it will never be fixed... There are
> multiple workarounds on the internet but according to responses they are all
> unreliable. We should upgrade to HikariCP (which in turn is only supported by
> DN 4), meanwhile try some shamanic rituals. In this JIRA we will try a
> relatively weak drum.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[ https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-11923: -- Attachment: HIVE-11923.2.txt HIVE-11923.2.branchllap.txt Patches for master and llap branch. For the llap branch, HIVE-11572 should ideally be applied first for the llap branch. Have seen the llap tests complete in about an hour. The patch also reduces unnecessary logging from the MiniCluster. > allow qtests to run via a single client session for tez and llap > > > Key: HIVE-11923 > URL: https://issues.apache.org/jira/browse/HIVE-11923 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-11923.1.txt, HIVE-11923.2.branchllap.txt, > HIVE-11923.2.txt, HIVE-11923.branch-1.txt > > > Launching a new session - AM and containers for each test adds unnecessary > overheads. Running via a single session should reduce the run time > significantly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11944) Address the review items on HIVE-11778
[ https://issues.apache.org/jira/browse/HIVE-11944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-11944: Affects Version/s: beeline-cli-branch > Address the review items on HIVE-11778 > -- > > Key: HIVE-11944 > URL: https://issues.apache.org/jira/browse/HIVE-11944 > Project: Hive > Issue Type: Sub-task > Components: CLI >Affects Versions: beeline-cli-branch >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: beeline-cli-branch > > Attachments: HIVE-11944.1-beeline-cli.patch > > > This jira will address review items from https://reviews.apache.org/r/38247/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11941) Update committer list
[ https://issues.apache.org/jira/browse/HIVE-11941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907334#comment-14907334 ] Ferdinand Xu commented on HIVE-11941: - Hi [~ctang.ma], Could you add [~lars_francke] to the committer list and move [~sushanth] to the PMC list as well? > Update committer list > - > > Key: HIVE-11941 > URL: https://issues.apache.org/jira/browse/HIVE-11941 > Project: Hive > Issue Type: Bug >Reporter: Chaoyu Tang >Priority: Minor > Attachments: HIVE-11941.patch > > > Please update the committer list in http://hive.apache.org/people.html: > --- > Name: Chaoyu Tang > Apache ID: ctang > Organization: Cloudera (www.cloudera.com) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11831) TXN tables in Oracle should be created with ROWDEPENDENCIES
[ https://issues.apache.org/jira/browse/HIVE-11831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11831: Release Note: ROWDEPENDENCIES cannot be added to the table after it has already been created. If you hit this issue on an existing database, you might want to (this requires stopping all Hive workloads for the duration), for each table (see the patch for what tables need to be updated; locks and txns table are the ones most affected): 1) create temp table 2) move contents of existing table to temp table 3) drop existing table 4) create new table with ROWDEPENDENCIES, as per the attached patch 5) move data back from temp table > TXN tables in Oracle should be created with ROWDEPENDENCIES > --- > > Key: HIVE-11831 > URL: https://issues.apache.org/jira/browse/HIVE-11831 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.0.0 > > Attachments: HIVE-11831.01.patch, HIVE-11831.patch > > > These frequently-updated tables may otherwise suffer from spurious deadlocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11941) Update committer list
[ https://issues.apache.org/jira/browse/HIVE-11941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-11941: --- Attachment: HIVE-11941.patch [~Ferd] Please review the patch and it is same as that I posted in HIVE-10718. Thanks. > Update committer list > - > > Key: HIVE-11941 > URL: https://issues.apache.org/jira/browse/HIVE-11941 > Project: Hive > Issue Type: Bug >Reporter: Chaoyu Tang >Priority: Minor > Attachments: HIVE-11941.patch > > > Please update the committer list in http://hive.apache.org/people.html: > --- > Name: Chaoyu Tang > Apache ID: ctang > Organization: Cloudera (www.cloudera.com) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11642) LLAP: make sure tests pass #3
[ https://issues.apache.org/jira/browse/HIVE-11642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11642: Attachment: HIVE-11642.10.patch A more recent diff in case some conflicts accrued during the queue time > LLAP: make sure tests pass #3 > - > > Key: HIVE-11642 > URL: https://issues.apache.org/jira/browse/HIVE-11642 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11642.01.patch, HIVE-11642.02.patch, > HIVE-11642.03.patch, HIVE-11642.04.patch, HIVE-11642.05.patch, > HIVE-11642.08.patch, HIVE-11642.09.patch, HIVE-11642.10.patch, > HIVE-11642.patch > > > Tests should pass against the most recent branch and Tez 0.8. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11915) BoneCP returns closed connections from the pool
[ https://issues.apache.org/jira/browse/HIVE-11915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11915: Attachment: HIVE-11915.01.patch Patch > BoneCP returns closed connections from the pool > --- > > Key: HIVE-11915 > URL: https://issues.apache.org/jira/browse/HIVE-11915 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-11915.01.patch, HIVE-11915.WIP.patch, > HIVE-11915.patch > > > It's a very old bug in BoneCP and it will never be fixed... There are > multiple workarounds on the internet but according to responses they are all > unreliable. We should upgrade to HikariCP (which in turn is only supported by > DN 4), meanwhile try some shamanic rituals. In this JIRA we will try a > relatively weak drum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11903) Add zookeeper metrics to HS2
[ https://issues.apache.org/jira/browse/HIVE-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907318#comment-14907318 ] Szehon Ho commented on HIVE-11903: -- Thanks, unfortunately I think this might be a more difficult than we anticipated and we would need to do some research, as the number of singletons is not going to be useful. Can we hook into the Watcher interface to increment/decrement connection count? There is some examples in HiveServer2. > Add zookeeper metrics to HS2 > > > Key: HIVE-11903 > URL: https://issues.apache.org/jira/browse/HIVE-11903 > Project: Hive > Issue Type: Sub-task > Components: Diagnosability >Reporter: Szehon Ho >Assignee: Yongzhi Chen > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11903.1.patch, HIVE-11903.2.patch > > > Potential metrics are active zookeeper connections, locks taken by type, etc. > Can refine as we go along. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11903) Add zookeeper metrics to HS2
[ https://issues.apache.org/jira/browse/HIVE-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-11903: Attachment: HIVE-11903.2.patch patch 2 add the test for zp connection and remove the jdbc client connection. > Add zookeeper metrics to HS2 > > > Key: HIVE-11903 > URL: https://issues.apache.org/jira/browse/HIVE-11903 > Project: Hive > Issue Type: Sub-task > Components: Diagnosability >Reporter: Szehon Ho >Assignee: Yongzhi Chen > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11903.1.patch, HIVE-11903.2.patch > > > Potential metrics are active zookeeper connections, locks taken by type, etc. > Can refine as we go along. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11952) disable q tests that are both slow and less relevant
[ https://issues.apache.org/jira/browse/HIVE-11952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11952: Attachment: HIVE-11952.patch Small patch. [~spena] can you check if it makes sense? > disable q tests that are both slow and less relevant > > > Key: HIVE-11952 > URL: https://issues.apache.org/jira/browse/HIVE-11952 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11952.patch > > > We will disable several tests that test obscure and old features and take > inordinate amount of time, and file JIRAs to look at their perf if someone > still cares about them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11945) ORC with non-local reads may not be reusing connection to DN
[ https://issues.apache.org/jira/browse/HIVE-11945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907277#comment-14907277 ] Rajesh Balamohan commented on HIVE-11945: - Thanks for the review comments [~sershe] & [~prasanth_j]. I will upload revised patch with review comments addressed (to fix zcr & s3 case). > ORC with non-local reads may not be reusing connection to DN > > > Key: HIVE-11945 > URL: https://issues.apache.org/jira/browse/HIVE-11945 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: HIVE-11945.1.patch > > > When “seek + readFully(buffer, offset, length)” is used, DFSInputStream ends > up going via “readWithStrategy()”. This sets up BlockReader with length > equivalent to that of the block size. So until this position is reached, > RemoteBlockReader2.peer would not be added to the PeerCache (Plz refer > RemoteBlockReader2.close() in HDFS). So eventually the next call to the same > DN would end opening a new socket. In ORC, when it is not a data local read, > this has a the possibility of opening/closing lots of connections with DN. > In random reads, it would be good to set this length to the amount of data > that is to be read (e.g pread call in DFSInputStream which sets up the > BlockReader’s length correctly & the code path returns the Peer back to peer > cache properly). “readFully(position, buffer, offset, length)” follows this > code path and ends up reusing the connections properly. Creating this JIRA to > fix this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11935) Access HiveMetaStoreClient.currentMetaVars should be synchronized
[
https://issues.apache.org/jira/browse/HIVE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907257#comment-14907257
]
Hive QA commented on HIVE-11935:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12761939/HIVE-11935.1.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9581 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.streaming.TestStreaming.testAddPartition
org.apache.hive.hcatalog.streaming.TestStreaming.testInterleavedTransactionBatchCommits
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbortAndCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5404/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5404/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5404/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12761939 - PreCommit-HIVE-TRUNK-Build
> Access HiveMetaStoreClient.currentMetaVars should be synchronized
> -
>
> Key: HIVE-11935
> URL: https://issues.apache.org/jira/browse/HIVE-11935
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-11935.1.patch
>
>
> We saw intermittent failure of the following stack:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.isCompatibleWith(HiveMetaStoreClient.java:287)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy9.isCompatibleWith(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:206)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.createHiveDB(BaseSemanticAnalyzer.java:205)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.(DDLSemanticAnalyzer.java:223)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get(SemanticAnalyzerFactory.java:259)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:409)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1116)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:388)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:375)
> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$P
[jira] [Commented] (HIVE-11937) Improve StatsOptimizer to deal with query with additional constant columns
[ https://issues.apache.org/jira/browse/HIVE-11937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907244#comment-14907244 ] Ashutosh Chauhan commented on HIVE-11937: - +1 > Improve StatsOptimizer to deal with query with additional constant columns > -- > > Key: HIVE-11937 > URL: https://issues.apache.org/jira/browse/HIVE-11937 > Project: Hive > Issue Type: Improvement >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11937.01.patch > > > Right now StatsOptimizer can deal with query such as "select count(1) from > src" by directly looking into the metastore. However, it can not deal with > "select '1' as one, count(1) from src" which has an additional constant > column. We may improve it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11110) Reorder applyPreJoinOrderingTransforms, add NotNULL/FilterMerge rules, improve Filter selectivity estimation
[
https://issues.apache.org/jira/browse/HIVE-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laljo John Pullokkaran updated HIVE-0:
--
Attachment: HIVE-0.17.patch
> Reorder applyPreJoinOrderingTransforms, add NotNULL/FilterMerge rules,
> improve Filter selectivity estimation
>
>
> Key: HIVE-0
> URL: https://issues.apache.org/jira/browse/HIVE-0
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-0-10.patch, HIVE-0-11.patch,
> HIVE-0-12.patch, HIVE-0-branch-1.2.patch, HIVE-0.1.patch,
> HIVE-0.13.patch, HIVE-0.14.patch, HIVE-0.15.patch,
> HIVE-0.16.patch, HIVE-0.17.patch, HIVE-0.2.patch,
> HIVE-0.4.patch, HIVE-0.5.patch, HIVE-0.6.patch,
> HIVE-0.7.patch, HIVE-0.8.patch, HIVE-0.9.patch,
> HIVE-0.91.patch, HIVE-0.92.patch, HIVE-0.patch
>
>
> Query
> {code}
> select count(*)
> from store_sales
> ,store_returns
> ,date_dim d1
> ,date_dim d2
> where d1.d_quarter_name = '2000Q1'
>and d1.d_date_sk = ss_sold_date_sk
>and ss_customer_sk = sr_customer_sk
>and ss_item_sk = sr_item_sk
>and ss_ticket_number = sr_ticket_number
>and sr_returned_date_sk = d2.d_date_sk
>and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3’);
> {code}
> The store_sales table is partitioned on ss_sold_date_sk, which is also used
> in a join clause. The join clause should add a filter “filterExpr:
> ss_sold_date_sk is not null”, which should get pushed the MetaStore when
> fetching the stats. Currently this is not done in CBO planning, which results
> in the stats from __HIVE_DEFAULT_PARTITION__ to be fetched and considered in
> the optimization phase. In particular, this increases the NDV for the join
> columns and may result in wrong planning.
> Including HiveJoinAddNotNullRule in the optimization phase solves this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11928) ORC footer section can also exceed protobuf message limit
[ https://issues.apache.org/jira/browse/HIVE-11928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907240#comment-14907240 ] Owen O'Malley commented on HIVE-11928: -- LGTM +1 > ORC footer section can also exceed protobuf message limit > - > > Key: HIVE-11928 > URL: https://issues.apache.org/jira/browse/HIVE-11928 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Jagruti Varia >Assignee: Prasanth Jayachandran > Attachments: HIVE-11928-branch-1.patch, HIVE-11928.1.patch, > HIVE-11928.1.patch, HIVE-11928.2.patch > > > Similar to HIVE-11592 but for orc footer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11907) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix TestJdbcDriver2
[ https://issues.apache.org/jira/browse/HIVE-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907227#comment-14907227 ] Ashutosh Chauhan commented on HIVE-11907: - We should use RowResolver for non-cbo & cbo-ast path and RowSchema for cbo-op path to compute Result Schema. > CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix > TestJdbcDriver2 > - > > Key: HIVE-11907 > URL: https://issues.apache.org/jira/browse/HIVE-11907 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11907.01.patch > > > TestJdbcDriver2 is currently failing on return path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11942) MetaException(message:The threadlocal Deadline is null, please register it first.)
[
https://issues.apache.org/jira/browse/HIVE-11942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907216#comment-14907216
]
Wei Zheng commented on HIVE-11942:
--
False alarm. Tez local mode bypasses some setup including Deadline register.
> MetaException(message:The threadlocal Deadline is null, please register it
> first.)
> --
>
> Key: HIVE-11942
> URL: https://issues.apache.org/jira/browse/HIVE-11942
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: WIP.patch
>
>
> I got such exception when running qtest unionDistinct_1.q with my WIP patch
> for another JIRA (attached). I tried the same qfile on master w/o my patch
> and couldn't reproduce.
> But I don't have any change that's related to metastore, so I guess maybe my
> code exposed some bug.
> {code}
> 2015-09-23T17:02:05,385 ERROR [main]: ql.Driver
> (SessionState.java:printError(967)) - FAILED: RuntimeException
> org.apache.hadoop.hive.ql.parse.SemanticException: MetaException(message:The
> threadlocal Deadline is null, please register it first.)
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.parse.SemanticException: MetaException(message:The
> threadlocal Deadline is null, please register it first.)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:151)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:617)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:252)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10143)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:212)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:240)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:240)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:434)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:310)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1209)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1085)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1075)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1084)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1058)
> at
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.runTest(TestMiniTezCliDriver.java:147)
> at
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_unionDistinct_1(TestMiniTezCliDriver.java:131)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: org.apache.hadoop.hive.ql.parse.SemanticException:
> MetaException(message:The threadlocal Deadline is null, please registe
[jira] [Resolved] (HIVE-11942) MetaException(message:The threadlocal Deadline is null, please register it first.)
[
https://issues.apache.org/jira/browse/HIVE-11942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng resolved HIVE-11942.
--
Resolution: Invalid
> MetaException(message:The threadlocal Deadline is null, please register it
> first.)
> --
>
> Key: HIVE-11942
> URL: https://issues.apache.org/jira/browse/HIVE-11942
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: WIP.patch
>
>
> I got such exception when running qtest unionDistinct_1.q with my WIP patch
> for another JIRA (attached). I tried the same qfile on master w/o my patch
> and couldn't reproduce.
> But I don't have any change that's related to metastore, so I guess maybe my
> code exposed some bug.
> {code}
> 2015-09-23T17:02:05,385 ERROR [main]: ql.Driver
> (SessionState.java:printError(967)) - FAILED: RuntimeException
> org.apache.hadoop.hive.ql.parse.SemanticException: MetaException(message:The
> threadlocal Deadline is null, please register it first.)
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.parse.SemanticException: MetaException(message:The
> threadlocal Deadline is null, please register it first.)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:151)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:617)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:252)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10143)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:212)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:240)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:240)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:434)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:310)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1209)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1085)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1075)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1084)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1058)
> at
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.runTest(TestMiniTezCliDriver.java:147)
> at
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_unionDistinct_1(TestMiniTezCliDriver.java:131)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: org.apache.hadoop.hive.ql.parse.SemanticException:
> MetaException(message:The threadlocal Deadline is null, please register it
> first.)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.getPartitionsFromServer(Parti
[jira] [Commented] (HIVE-11894) CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table column name in CTAS queries
[ https://issues.apache.org/jira/browse/HIVE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907207#comment-14907207 ] Ashutosh Chauhan commented on HIVE-11894: - I think we can always use resultSchema regardless of whether we are on cbo or not. That will simplify the code. > CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table > column name in CTAS queries > --- > > Key: HIVE-11894 > URL: https://issues.apache.org/jira/browse/HIVE-11894 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11894.01.patch, HIVE-11894.02.patch > > > To repro, run lineage2.q with return path turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11903) Add zookeeper metrics to HS2
[ https://issues.apache.org/jira/browse/HIVE-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907203#comment-14907203 ] Szehon Ho commented on HIVE-11903: -- 2. Sorry you are right, my mistake. 3. Sorry , I mistype, number of zookeeper connections. > Add zookeeper metrics to HS2 > > > Key: HIVE-11903 > URL: https://issues.apache.org/jira/browse/HIVE-11903 > Project: Hive > Issue Type: Sub-task > Components: Diagnosability >Reporter: Szehon Ho >Assignee: Yongzhi Chen > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11903.1.patch > > > Potential metrics are active zookeeper connections, locks taken by type, etc. > Can refine as we go along. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11942) MetaException(message:The threadlocal Deadline is null, please register it first.)
[
https://issues.apache.org/jira/browse/HIVE-11942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng reassigned HIVE-11942:
Assignee: Wei Zheng
> MetaException(message:The threadlocal Deadline is null, please register it
> first.)
> --
>
> Key: HIVE-11942
> URL: https://issues.apache.org/jira/browse/HIVE-11942
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: WIP.patch
>
>
> I got such exception when running qtest unionDistinct_1.q with my WIP patch
> for another JIRA (attached). I tried the same qfile on master w/o my patch
> and couldn't reproduce.
> But I don't have any change that's related to metastore, so I guess maybe my
> code exposed some bug.
> {code}
> 2015-09-23T17:02:05,385 ERROR [main]: ql.Driver
> (SessionState.java:printError(967)) - FAILED: RuntimeException
> org.apache.hadoop.hive.ql.parse.SemanticException: MetaException(message:The
> threadlocal Deadline is null, please register it first.)
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.parse.SemanticException: MetaException(message:The
> threadlocal Deadline is null, please register it first.)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:151)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:617)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:252)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10143)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:212)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:240)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:240)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:434)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:310)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1209)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1085)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1075)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1084)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1058)
> at
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.runTest(TestMiniTezCliDriver.java:147)
> at
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_unionDistinct_1(TestMiniTezCliDriver.java:131)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: org.apache.hadoop.hive.ql.parse.SemanticException:
> MetaException(message:The threadlocal Deadline is null, please register it
> first.)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.getPartitionsFromServer(
[jira] [Resolved] (HIVE-9748) MetastoreDirectSql fails for zero item queries
[
https://issues.apache.org/jira/browse/HIVE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin resolved HIVE-9748.
Resolution: Duplicate
> MetastoreDirectSql fails for zero item queries
> --
>
> Key: HIVE-9748
> URL: https://issues.apache.org/jira/browse/HIVE-9748
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>
> Metastore Direct SQL throws a SQL Exception
> {code}
> 2015-02-21 00:29:00,238 WARN [pool-3-thread-10]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(2400)) - Direct SQL failed
> MetaException(message:See previous errors; Error executing SQL query "select
> count("COLUMN_NAME") from "PART_COL_STATS" where "DB_NAME" = ? and
> "TABLE_NAME" = ? and "COLUMN_NAME" in () and "PARTITION_NAME" in () group by
> "PARTITION_NAME"
> ".)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1448)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.partsFoundForPartitions(MetaStoreDirectSql.java:1098)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1081)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6100)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6096)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2365)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6115)
> {code}
> The query to trigger the issue an EXPLAIN query with column + partition stats
> on.
> {code}
> explain select count(1) from store_sales where '2014-10-01' =ss_sold_date ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11517) Vectorized auto_smb_mapjoin_14.q produces different results
[
https://issues.apache.org/jira/browse/HIVE-11517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-11517:
Fix Version/s: 2.0.0
1.3.0
> Vectorized auto_smb_mapjoin_14.q produces different results
> ---
>
> Key: HIVE-11517
> URL: https://issues.apache.org/jira/browse/HIVE-11517
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-11517.01.patch, HIVE-11517.02.patch
>
>
> Converted Q file to use ORC and turned on vectorization.
> The query:
> {code}
> select count(*) from (
> select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2
> b on a.key = b.key
> ) subq1
> {code}
> produces 10 instead of 22.
> The query:
> {code}
> select src1.key, src1.cnt1, src2.cnt1 from
> (
> select key, count(*) as cnt1 from
> (
> select a.key as key, a.value as val1, b.value as val2 from tbl1 a join
> tbl2 b on a.key = b.key
> ) subq1 group by key
> ) src1
> join
> (
> select key, count(*) as cnt1 from
> (
> select a.key as key, a.value as val1, b.value as val2 from tbl1 a join
> tbl2 b on a.key = b.key
> ) subq2 group by key
> ) src2
> {code}
> produces:
> {code}
> 0 3 3
> 2 1 1
> 4 1 1
> 5 3 3
> 8 1 1
> 9 1 1
> {code}
> instead of:
> {code}
> 0 9 9
> 2 1 1
> 4 1 1
> 5 9 9
> 8 1 1
> 9 1 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11517) Vectorized auto_smb_mapjoin_14.q produces different results
[
https://issues.apache.org/jira/browse/HIVE-11517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907186#comment-14907186
]
Matt McCline commented on HIVE-11517:
-
Committed to master and branch-1
> Vectorized auto_smb_mapjoin_14.q produces different results
> ---
>
> Key: HIVE-11517
> URL: https://issues.apache.org/jira/browse/HIVE-11517
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-11517.01.patch, HIVE-11517.02.patch
>
>
> Converted Q file to use ORC and turned on vectorization.
> The query:
> {code}
> select count(*) from (
> select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2
> b on a.key = b.key
> ) subq1
> {code}
> produces 10 instead of 22.
> The query:
> {code}
> select src1.key, src1.cnt1, src2.cnt1 from
> (
> select key, count(*) as cnt1 from
> (
> select a.key as key, a.value as val1, b.value as val2 from tbl1 a join
> tbl2 b on a.key = b.key
> ) subq1 group by key
> ) src1
> join
> (
> select key, count(*) as cnt1 from
> (
> select a.key as key, a.value as val1, b.value as val2 from tbl1 a join
> tbl2 b on a.key = b.key
> ) subq2 group by key
> ) src2
> {code}
> produces:
> {code}
> 0 3 3
> 2 1 1
> 4 1 1
> 5 3 3
> 8 1 1
> 9 1 1
> {code}
> instead of:
> {code}
> 0 9 9
> 2 1 1
> 4 1 1
> 5 9 9
> 8 1 1
> 9 1 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-8824) Exception happened in case set hive.metastore.try.direct.sql to true when running 'show table stats xxx' on impala with hive metastore db is mysql
[ https://issues.apache.org/jira/browse/HIVE-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907179#comment-14907179 ] Sergey Shelukhin edited comment on HIVE-8824 at 9/24/15 10:44 PM: -- It is already set to ansi quotes actually (should be, anyway, unless there's some bug). See doDbSpecificInit... method was (Author: sershe): It is already set to ansi quotes actually. > Exception happened in case set hive.metastore.try.direct.sql to true when > running 'show table stats xxx' on impala with hive metastore db is mysql > --- > > Key: HIVE-8824 > URL: https://issues.apache.org/jira/browse/HIVE-8824 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.13.1 >Reporter: lixuebin > > Hive Metstore Database is MySQL. > First: > set hive-site.xml hive.metastore.try.direct.sql true > second: > On impala > show table stats x; > Remark: > MYSQL does not support table name and column name with double quotes, such as > : select * from "testtb" where "id"='1234'; > should be: select * from testtb where id='1234'; > MetaStoreDirectSql.java should be checked. > The Exception: > metastore.ObjectStore (ObjectStore.java:handleDirectSqlError(2243)) - Direct > SQL failed, falling back to ORM > javax.jdo.JDODataStoreException: Error executing SQL query "select > "COLUMN_NAME", "COLUMN_TYPE", "LONG_LOW_VALUE", "LONG_HIGH_VALUE", > "DOUBLE_LOW_VALUE", "DOUBLE_HIGH_VALUE", "BIG_DECIMAL_LOW_VALUE", > "BIG_DECIMAL_HIGH_VALUE", "NUM_NULLS", "NUM_DISTINCTS", "AVG_COL_LEN", > "MAX_COL_LEN", "NUM_TRUES", "NUM_FALSES", "LAST_ANALYZED" from > "TAB_COL_STATS" where "DB_NAME" = ? and "TABLE_NAME" = ? and "COLUMN_NAME" > in (?)". > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTableStats(MetaStoreDirectSql.java:879) > at > org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:5749) > at > org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:5746) > at > org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2208) > at > org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:5746) > at > org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:5740) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108) > at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:3559) > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105) > at com.sun.proxy.$Proxy1.get_table_statistics_req(Unknown Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:9528) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:9512) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > NestedThrowablesStackTrace: > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error > in your SQL syntax; check the manual that corresponds to your MySQL server > version for the right syntax to use near '"TAB_COL_STATS" where "DB_NAME" = > 'tpcds' and "TABLE_
[jira] [Updated] (HIVE-9748) MetastoreDirectSql fails for zero item queries
[
https://issues.apache.org/jira/browse/HIVE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-9748:
---
Summary: MetastoreDirectSql fails for zero item queries (was:
MetastoreDirectSql fails for zero partition queries)
> MetastoreDirectSql fails for zero item queries
> --
>
> Key: HIVE-9748
> URL: https://issues.apache.org/jira/browse/HIVE-9748
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>
> Metastore Direct SQL throws a SQL Exception
> {code}
> 2015-02-21 00:29:00,238 WARN [pool-3-thread-10]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(2400)) - Direct SQL failed
> MetaException(message:See previous errors; Error executing SQL query "select
> count("COLUMN_NAME") from "PART_COL_STATS" where "DB_NAME" = ? and
> "TABLE_NAME" = ? and "COLUMN_NAME" in () and "PARTITION_NAME" in () group by
> "PARTITION_NAME"
> ".)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1448)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.partsFoundForPartitions(MetaStoreDirectSql.java:1098)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1081)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6100)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6096)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2365)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6115)
> {code}
> The query to trigger the issue an EXPLAIN query with column + partition stats
> on.
> {code}
> explain select count(1) from store_sales where '2014-10-01' =ss_sold_date ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-9748) MetastoreDirectSql fails for zero partition queries
[
https://issues.apache.org/jira/browse/HIVE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-9748:
--
Assignee: Sergey Shelukhin (was: Thejas M Nair)
> MetastoreDirectSql fails for zero partition queries
> ---
>
> Key: HIVE-9748
> URL: https://issues.apache.org/jira/browse/HIVE-9748
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>
> Metastore Direct SQL throws a SQL Exception
> {code}
> 2015-02-21 00:29:00,238 WARN [pool-3-thread-10]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(2400)) - Direct SQL failed
> MetaException(message:See previous errors; Error executing SQL query "select
> count("COLUMN_NAME") from "PART_COL_STATS" where "DB_NAME" = ? and
> "TABLE_NAME" = ? and "COLUMN_NAME" in () and "PARTITION_NAME" in () group by
> "PARTITION_NAME"
> ".)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1448)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.partsFoundForPartitions(MetaStoreDirectSql.java:1098)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1081)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6100)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:6096)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2365)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6115)
> {code}
> The query to trigger the issue an EXPLAIN query with column + partition stats
> on.
> {code}
> explain select count(1) from store_sales where '2014-10-01' =ss_sold_date ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8824) Exception happened in case set hive.metastore.try.direct.sql to true when running 'show table stats xxx' on impala with hive metastore db is mysql
[ https://issues.apache.org/jira/browse/HIVE-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907179#comment-14907179 ] Sergey Shelukhin commented on HIVE-8824: It is already set to ansi quotes actually. > Exception happened in case set hive.metastore.try.direct.sql to true when > running 'show table stats xxx' on impala with hive metastore db is mysql > --- > > Key: HIVE-8824 > URL: https://issues.apache.org/jira/browse/HIVE-8824 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.13.1 >Reporter: lixuebin > > Hive Metstore Database is MySQL. > First: > set hive-site.xml hive.metastore.try.direct.sql true > second: > On impala > show table stats x; > Remark: > MYSQL does not support table name and column name with double quotes, such as > : select * from "testtb" where "id"='1234'; > should be: select * from testtb where id='1234'; > MetaStoreDirectSql.java should be checked. > The Exception: > metastore.ObjectStore (ObjectStore.java:handleDirectSqlError(2243)) - Direct > SQL failed, falling back to ORM > javax.jdo.JDODataStoreException: Error executing SQL query "select > "COLUMN_NAME", "COLUMN_TYPE", "LONG_LOW_VALUE", "LONG_HIGH_VALUE", > "DOUBLE_LOW_VALUE", "DOUBLE_HIGH_VALUE", "BIG_DECIMAL_LOW_VALUE", > "BIG_DECIMAL_HIGH_VALUE", "NUM_NULLS", "NUM_DISTINCTS", "AVG_COL_LEN", > "MAX_COL_LEN", "NUM_TRUES", "NUM_FALSES", "LAST_ANALYZED" from > "TAB_COL_STATS" where "DB_NAME" = ? and "TABLE_NAME" = ? and "COLUMN_NAME" > in (?)". > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321) > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTableStats(MetaStoreDirectSql.java:879) > at > org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:5749) > at > org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:5746) > at > org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2208) > at > org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:5746) > at > org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:5740) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108) > at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:3559) > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105) > at com.sun.proxy.$Proxy1.get_table_statistics_req(Unknown Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:9528) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:9512) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > NestedThrowablesStackTrace: > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error > in your SQL syntax; check the manual that corresponds to your MySQL server > version for the right syntax to use near '"TAB_COL_STATS" where "DB_NAME" = > 'tpcds' and "TABLE_NAME" = 'et_date_dim' and ' at line 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11954) Extend logic to choose side table in MapJoin Conversion algorithm
[ https://issues.apache.org/jira/browse/HIVE-11954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-11954: --- Attachment: HIVE-11954.patch > Extend logic to choose side table in MapJoin Conversion algorithm > - > > Key: HIVE-11954 > URL: https://issues.apache.org/jira/browse/HIVE-11954 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11954.patch > > > Selection of side table (in memory/hash table) in MapJoin Conversion > algorithm needs to be more sophisticated. > In an N way Map Join, Hive should pick an input stream as side table (in > memory table) that has least cost in producing relation (like TS(FIL|Proj)*). > Cost based choice needs extended cost model; without return path its going to > be hard to do this. > For the time being we could employ a modified cost based algorithm for side > table selection. > New algorithm is described below: > 1. Identify the candidate set of inputs for side table (in memory/hash table) > from the inputs (based on conditional task size) > 2. For each of the input identify its cost, memory requirement. Cost is 1 for > each heavy weight relation op (Join, GB, PTF/Windowing, TF, etc.). Cost for > an input is the total no of heavy weight ops in its branch. > 3. Order set from #1 on cost & memory req (ascending order) > 4. Pick the first element from #3 as the side table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11928) ORC footer section can also exceed protobuf message limit
[ https://issues.apache.org/jira/browse/HIVE-11928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-11928: - Attachment: HIVE-11928.2.patch Addressed [~owen.omalley]'s comment here.. https://issues.apache.org/jira/browse/HIVE-11592?focusedCommentId=14906942&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14906942 [~owen.omalley] Can you please take a look at the new changes? > ORC footer section can also exceed protobuf message limit > - > > Key: HIVE-11928 > URL: https://issues.apache.org/jira/browse/HIVE-11928 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Jagruti Varia >Assignee: Prasanth Jayachandran > Attachments: HIVE-11928-branch-1.patch, HIVE-11928.1.patch, > HIVE-11928.1.patch, HIVE-11928.2.patch > > > Similar to HIVE-11592 but for orc footer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11533) Loop optimization for SIMD in Long*CompareLong*
[
https://issues.apache.org/jira/browse/HIVE-11533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907099#comment-14907099
]
Hive QA commented on HIVE-11533:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12761938/HIVE-11533.1.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5403/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5403/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5403/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-5403/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
1528135..68d6cfd master -> origin/master
+ git reset --hard HEAD
HEAD is now at 1528135 HIVE-11926: Stats annotation might not extract stats for
varchar/decimal columns (Chaoyu Tang, reviewed by Xuefu Zhang)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 68d6cfd HIVE-10785 : Support aggregate push down through joins
(Ashutosh Chauhan via Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12761938 - PreCommit-HIVE-TRUNK-Build
> Loop optimization for SIMD in Long*CompareLong*
> ---
>
> Key: HIVE-11533
> URL: https://issues.apache.org/jira/browse/HIVE-11533
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11533.1.patch
>
>
> Long*CompareLong* classes can be optimized with subtraction and bitwise
> operators for better SIMD optimization.
> {code}
> for(int i = 0; i != n; i++) {
> outputVector[i] = vector1[0] > vector2[i] ? 1 : 0;
> }
> {code}
> This issue will cover following classes;
> - LongColEqualLongColumn
> - LongColGreaterLongColumn
> - LongColGreaterEqualLongColumn
> - LongColLessLongColumn
> - LongColLessEqualLongColumn
> - LongScalarEqualLongColumn
> - LongScalarGreaterLongColumn
> - LongScalarGreaterEqualLongColumn
> - LongScalarLessLongColumn
> - LongScalarLessEqualLongColumn
> - LongColEqualLongScalar
> - LongColGreaterLongScalar
> - LongColGreaterEqualLongScalar
> - LongColLessLongScalar
> - LongColLessEqualLongScalar
> - LongScalarEqualLongScalar
> - LongScalarGreaterLongScalar
> - LongScalarGreaterEqualLongScalar
> - LongScalarLessLongScalar
> - LongScalarLessEqualLongScalar
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11932) JDBC Driver appends an extra "/" when configuring connection by reading httpPath from ZooKeeper
[
https://issues.apache.org/jira/browse/HIVE-11932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907095#comment-14907095
]
Hive QA commented on HIVE-11932:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12761874/HIVE-11932.1.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9580 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.streaming.TestStreaming.testInterleavedTransactionBatchCommits
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5402/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5402/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5402/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12761874 - PreCommit-HIVE-TRUNK-Build
> JDBC Driver appends an extra "/" when configuring connection by reading
> httpPath from ZooKeeper
> ---
>
> Key: HIVE-11932
> URL: https://issues.apache.org/jira/browse/HIVE-11932
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-11932.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11895) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix udaf_percentile_approx_23.q
[ https://issues.apache.org/jira/browse/HIVE-11895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11895: --- Attachment: HIVE-11895.01.patch > CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix > udaf_percentile_approx_23.q > - > > Key: HIVE-11895 > URL: https://issues.apache.org/jira/browse/HIVE-11895 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11895.01.patch > > > Due to a type conversion problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11915) BoneCP returns closed connections from the pool
[ https://issues.apache.org/jira/browse/HIVE-11915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907069#comment-14907069 ] Sergey Shelukhin commented on HIVE-11915: - [~thejas] ping? > BoneCP returns closed connections from the pool > --- > > Key: HIVE-11915 > URL: https://issues.apache.org/jira/browse/HIVE-11915 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-11915.WIP.patch, HIVE-11915.patch > > > It's a very old bug in BoneCP and it will never be fixed... There are > multiple workarounds on the internet but according to responses they are all > unreliable. We should upgrade to HikariCP (which in turn is only supported by > DN 4), meanwhile try some shamanic rituals. In this JIRA we will try a > relatively weak drum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11945) ORC with non-local reads may not be reusing connection to DN
[ https://issues.apache.org/jira/browse/HIVE-11945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907063#comment-14907063 ] Prasanth Jayachandran commented on HIVE-11945: -- Looking at the implementation of readFully(long offset, byte[] buffer, int offset, int length) in FSInputStream, it seems like there are 2 seeks + 1 read with this method. The second seek is to restore the position to the old position. I am guessing this additional seek will be a significant overhead for some filesystems (S3?) right? > ORC with non-local reads may not be reusing connection to DN > > > Key: HIVE-11945 > URL: https://issues.apache.org/jira/browse/HIVE-11945 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: HIVE-11945.1.patch > > > When “seek + readFully(buffer, offset, length)” is used, DFSInputStream ends > up going via “readWithStrategy()”. This sets up BlockReader with length > equivalent to that of the block size. So until this position is reached, > RemoteBlockReader2.peer would not be added to the PeerCache (Plz refer > RemoteBlockReader2.close() in HDFS). So eventually the next call to the same > DN would end opening a new socket. In ORC, when it is not a data local read, > this has a the possibility of opening/closing lots of connections with DN. > In random reads, it would be good to set this length to the amount of data > that is to be read (e.g pread call in DFSInputStream which sets up the > BlockReader’s length correctly & the code path returns the Peer back to peer > cache properly). “readFully(position, buffer, offset, length)” follows this > code path and ends up reusing the connections properly. Creating this JIRA to > fix this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10785) Support aggregate push down through joins
[
https://issues.apache.org/jira/browse/HIVE-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-10785:
Issue Type: New Feature (was: Bug)
> Support aggregate push down through joins
> -
>
> Key: HIVE-10785
> URL: https://issues.apache.org/jira/browse/HIVE-10785
> Project: Hive
> Issue Type: New Feature
> Components: CBO, Logical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Ashutosh Chauhan
> Fix For: 2.0.0
>
> Attachments: HIVE-10785.2.patch, HIVE-10785.3.patch,
> HIVE-10785.4.patch, HIVE-10785.5.patch, HIVE-10785.6.patch, HIVE-10785.patch
>
>
> Enable {{AggregateJoinTransposeRule}} in CBO that pushes Aggregate through
> Join operators. The rule has been extended in Calcite 1.4 to cover complex
> cases e.g. Aggregate operators comprising UDAF. The decision on whether to
> push the Aggregate through Join or not should be cost-driven.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10785) Support aggregate push down through joins
[
https://issues.apache.org/jira/browse/HIVE-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-10785:
Fix Version/s: 2.0.0
> Support aggregate push down through joins
> -
>
> Key: HIVE-10785
> URL: https://issues.apache.org/jira/browse/HIVE-10785
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Ashutosh Chauhan
> Fix For: 2.0.0
>
> Attachments: HIVE-10785.2.patch, HIVE-10785.3.patch,
> HIVE-10785.4.patch, HIVE-10785.5.patch, HIVE-10785.6.patch, HIVE-10785.patch
>
>
> Enable {{AggregateJoinTransposeRule}} in CBO that pushes Aggregate through
> Join operators. The rule has been extended in Calcite 1.4 to cover complex
> cases e.g. Aggregate operators comprising UDAF. The decision on whether to
> push the Aggregate through Join or not should be cost-driven.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11950) WebHCat status file doesn't show UTF8 character
[ https://issues.apache.org/jira/browse/HIVE-11950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-11950: -- Attachment: HIVE-11950.1.patch > WebHCat status file doesn't show UTF8 character > --- > > Key: HIVE-11950 > URL: https://issues.apache.org/jira/browse/HIVE-11950 > Project: Hive > Issue Type: Bug > Components: WebHCat >Affects Versions: 1.2.1 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11950.1.patch > > > If we do a select on a UTF8 table and store the console output into the > status file (enablelog=true), the UTF8 character is garbled. The reason is we > don't specify encoding when opening stdout/stderr in statusdir. This will > cause problem especially on Windows, when the default OS encoding is not UTF8. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11902) Abort txn cleanup thread throws SyntaxErrorException
[
https://issues.apache.org/jira/browse/HIVE-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907015#comment-14907015
]
Deepesh Khandelwal commented on HIVE-11902:
---
Thanks [~ekoifman] for the review and commit!
> Abort txn cleanup thread throws SyntaxErrorException
>
>
> Key: HIVE-11902
> URL: https://issues.apache.org/jira/browse/HIVE-11902
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Deepesh Khandelwal
>Assignee: Deepesh Khandelwal
> Fix For: 1.3.0
>
> Attachments: HIVE-11902.patch
>
>
> When cleaning left over transactions we see the DeadTxnReaper code threw the
> following exception:
> {noformat}
> 2015-09-21 05:23:38,148 WARN [DeadTxnReaper-0]: txn.TxnHandler
> (TxnHandler.java:performTimeOuts(1876)) - Aborting timedout transactions
> failed due to You have an error in your SQL syntax; check the manual that
> corresponds to your MySQL server version for the right syntax to use near ')'
> at line 1(SQLState=42000,ErrorCode=1064)
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error
> in your SQL syntax; check the manual that corresponds to your MySQL server
> version for the right syntax to use near ')' at line 1
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
> at com.mysql.jdbc.Util.getInstance(Util.java:360)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526)
> at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1618)
> at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1549)
> at
> com.jolbox.bonecp.StatementHandle.executeUpdate(StatementHandle.java:497)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.abortTxns(TxnHandler.java:1275)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.performTimeOuts(TxnHandler.java:1866)
> at
> org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService$TimedoutTxnReaper.run(AcidHouseKeeperService.java:87)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The problem here is that the method {{abortTxns(Connection dbConn, List
> txnids)}} in
> metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> creates the following bad query when txnids list is empty.
> {code}
> delete from HIVE_LOCKS where hl_txnid in ();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11903) Add zookeeper metrics to HS2
[ https://issues.apache.org/jira/browse/HIVE-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906981#comment-14906981 ] Yongzhi Chen commented on HIVE-11903: - [~szehon], thanks for review the patch. For the suggestions: 1. I will remove JDBC zoopkeeper. 2. Just check the patch everywhere, it seems all has null checks. A little bit confused. I will add metrics based on lock-type. 3. For I already add the test of number of locks for zookeeper locks, do you need add one for metastore_hive_locks ? Thanks > Add zookeeper metrics to HS2 > > > Key: HIVE-11903 > URL: https://issues.apache.org/jira/browse/HIVE-11903 > Project: Hive > Issue Type: Sub-task > Components: Diagnosability >Reporter: Szehon Ho >Assignee: Yongzhi Chen > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11903.1.patch > > > Potential metrics are active zookeeper connections, locks taken by type, etc. > Can refine as we go along. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11592) ORC metadata section can sometimes exceed protobuf message size limit
[
https://issues.apache.org/jira/browse/HIVE-11592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906942#comment-14906942
]
Owen O'Malley commented on HIVE-11592:
--
Ok, after looking at the protobuf code, I'd propose that we always set the
limit at 1gb.
The case that I was worried about doesn't seem to happen, because the limit is
just set as a safety and won't accidentally cause streams to be accepted
incorrectly even if the message could end at that location.
> ORC metadata section can sometimes exceed protobuf message size limit
> -
>
> Key: HIVE-11592
> URL: https://issues.apache.org/jira/browse/HIVE-11592
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-11592.1.patch, HIVE-11592.2.patch,
> HIVE-11592.3.patch
>
>
> If there are too many small stripes and with many columns, the overhead for
> storing metadata (column stats) can exceed the default protobuf message size
> of 64MB. Reading such files will throw the following exception
> {code}
> Exception in thread "main"
> com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
> large. May be malicious. Use CodedInputStream.setSizeLimit() to increase
> the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
> at
> com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StringStatistics.(OrcProto.java:1331)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StringStatistics.(OrcProto.java:1281)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StringStatistics$1.parsePartialFrom(OrcProto.java:1374)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StringStatistics$1.parsePartialFrom(OrcProto.java:1369)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$ColumnStatistics.(OrcProto.java:4887)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$ColumnStatistics.(OrcProto.java:4803)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$ColumnStatistics$1.parsePartialFrom(OrcProto.java:4990)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$ColumnStatistics$1.parsePartialFrom(OrcProto.java:4985)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeStatistics.(OrcProto.java:12925)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeStatistics.(OrcProto.java:12872)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeStatistics$1.parsePartialFrom(OrcProto.java:12961)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeStatistics$1.parsePartialFrom(OrcProto.java:12956)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$Metadata.(OrcProto.java:13599)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$Metadata.(OrcProto.java:13546)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$Metadata$1.parsePartialFrom(OrcProto.java:13635)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$Metadata$1.parsePartialFrom(OrcProto.java:13630)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$Metadata.parseFrom(OrcProto.java:13746)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl$MetaInfoObjExtractor.(ReaderImpl.java:468)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:314)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
> at org.apache.hadoop.hive.ql.io.orc.FileDump.main(FileDump.java:67)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.
[jira] [Assigned] (HIVE-11947) mssql upgrade scripts contains invalid character
[ https://issues.apache.org/jira/browse/HIVE-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huan Huang reassigned HIVE-11947: - Assignee: Huan Huang (was: Wei Zheng) > mssql upgrade scripts contains invalid character > > > Key: HIVE-11947 > URL: https://issues.apache.org/jira/browse/HIVE-11947 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.2.0, 1.1.0 >Reporter: Huan Huang >Assignee: Huan Huang > Attachments: HIVE-11947.patch > > > upgrade scripts dont execute as a result -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11947) mssql upgrade scripts contains invalid character
[ https://issues.apache.org/jira/browse/HIVE-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-11947: Assignee: Wei Zheng (was: Huan Huang) > mssql upgrade scripts contains invalid character > > > Key: HIVE-11947 > URL: https://issues.apache.org/jira/browse/HIVE-11947 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.2.0, 1.1.0 >Reporter: Huan Huang >Assignee: Wei Zheng > Attachments: HIVE-11947.patch > > > upgrade scripts dont execute as a result -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10785) Support aggregate push down through joins
[
https://issues.apache.org/jira/browse/HIVE-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906922#comment-14906922
]
Jesus Camacho Rodriguez commented on HIVE-10785:
+1 (pending QA)
> Support aggregate push down through joins
> -
>
> Key: HIVE-10785
> URL: https://issues.apache.org/jira/browse/HIVE-10785
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-10785.2.patch, HIVE-10785.3.patch,
> HIVE-10785.4.patch, HIVE-10785.5.patch, HIVE-10785.6.patch, HIVE-10785.patch
>
>
> Enable {{AggregateJoinTransposeRule}} in CBO that pushes Aggregate through
> Join operators. The rule has been extended in Calcite 1.4 to cover complex
> cases e.g. Aggregate operators comprising UDAF. The decision on whether to
> push the Aggregate through Join or not should be cost-driven.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11947) mssql upgrade scripts contains invalid character
[ https://issues.apache.org/jira/browse/HIVE-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-11947: Assignee: Huan Huang > mssql upgrade scripts contains invalid character > > > Key: HIVE-11947 > URL: https://issues.apache.org/jira/browse/HIVE-11947 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.2.0, 1.1.0 >Reporter: Huan Huang >Assignee: Huan Huang > Attachments: HIVE-11947.patch > > > upgrade scripts dont execute as a result -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10598) Vectorization borks when column is added to table.
[
https://issues.apache.org/jira/browse/HIVE-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906909#comment-14906909
]
Hive QA commented on HIVE-10598:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12761857/HIVE-10598.05.patch
{color:red}ERROR:{color} -1 due to 54 failed/errored test(s), 9566 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vectorized_parquet.q-vector_char_mapjoin1.q-tez_insert_overwrite_local_directory_1.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_delete
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_delete_own_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_update
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_update_own_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_diff_part_cols
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_diff_part_cols2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge_incompat2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_non_string_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_partitioned_date_time
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_part_project
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_part_varchar
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.ql.TestTxnCommands.testTimeOutReaper
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactAfterAbort
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactWhileStreaming
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactAfterAbort
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactWhileStreaming
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableLegacy
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableNoBase
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableWithBase
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorWithAborted
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorWithOpenInMiddle
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorTableLegacy
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorTableNoBase
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorTableWithBase
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorWithAborted
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorWithOpenInMiddle
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorTableLegacy
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorTableNoBase
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorTableWithBase
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorWithAborted
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorWithOpenInMiddle
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorTableLegacy
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorTableNoBase
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorTableWithBase
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorWithAborted
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorWithOpenInMiddle
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.mapreduce.TestHCatExternalPartitioned.testHCatPartitionedTable[3]
org.apache.hive.hcatalog.mapreduce.TestHCatMutablePartitioned.testHCatPartitionedTable[3]
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[3]
org.apache.hive.hcatalog.pig.TestHCatStorer.testEmptyStore[3]
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreFuncAllSimpleTypes[3]
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5401/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5401/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit
[jira] [Updated] (HIVE-11947) mssql upgrade scripts contains invalid character
[ https://issues.apache.org/jira/browse/HIVE-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huan Huang updated HIVE-11947: -- Attachment: (was: HIVE-11947.patch) > mssql upgrade scripts contains invalid character > > > Key: HIVE-11947 > URL: https://issues.apache.org/jira/browse/HIVE-11947 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.2.0, 1.1.0 >Reporter: Huan Huang > Attachments: HIVE-11947.patch > > > upgrade scripts dont execute as a result -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11947) mssql upgrade scripts contains invalid character
[ https://issues.apache.org/jira/browse/HIVE-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huan Huang updated HIVE-11947: -- Attachment: HIVE-11947.patch > mssql upgrade scripts contains invalid character > > > Key: HIVE-11947 > URL: https://issues.apache.org/jira/browse/HIVE-11947 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.2.0, 1.1.0 >Reporter: Huan Huang > Attachments: HIVE-11947.patch > > > upgrade scripts dont execute as a result -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11947) mssql upgrade scripts contains invalid character
[ https://issues.apache.org/jira/browse/HIVE-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huan Huang updated HIVE-11947: -- Attachment: HIVE-11947.patch > mssql upgrade scripts contains invalid character > > > Key: HIVE-11947 > URL: https://issues.apache.org/jira/browse/HIVE-11947 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.2.0, 1.1.0 >Reporter: Huan Huang > Attachments: HIVE-11947.patch > > > upgrade scripts dont execute as a result -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11945) ORC with non-local reads may not be reusing connection to DN
[
https://issues.apache.org/jira/browse/HIVE-11945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906867#comment-14906867
]
Sergey Shelukhin commented on HIVE-11945:
-
{noformat}
if (zcr != null) {
{noformat}
branch after the removed seek in RecordReaderUtils is not changed. Does it
depend on this seek?
Other than that, the patch makes sense. Did you test that it fixes the issue?
> ORC with non-local reads may not be reusing connection to DN
>
>
> Key: HIVE-11945
> URL: https://issues.apache.org/jira/browse/HIVE-11945
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-11945.1.patch
>
>
> When “seek + readFully(buffer, offset, length)” is used, DFSInputStream ends
> up going via “readWithStrategy()”. This sets up BlockReader with length
> equivalent to that of the block size. So until this position is reached,
> RemoteBlockReader2.peer would not be added to the PeerCache (Plz refer
> RemoteBlockReader2.close() in HDFS). So eventually the next call to the same
> DN would end opening a new socket. In ORC, when it is not a data local read,
> this has a the possibility of opening/closing lots of connections with DN.
> In random reads, it would be good to set this length to the amount of data
> that is to be read (e.g pread call in DFSInputStream which sets up the
> BlockReader’s length correctly & the code path returns the Peer back to peer
> cache properly). “readFully(position, buffer, offset, length)” follows this
> code path and ends up reusing the connections properly. Creating this JIRA to
> fix this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11928) ORC footer section can also exceed protobuf message limit
[ https://issues.apache.org/jira/browse/HIVE-11928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-11928: - Attachment: HIVE-11928.1.patch This patch seems to have dropped from precommit queue. Reuploading the same patch again. > ORC footer section can also exceed protobuf message limit > - > > Key: HIVE-11928 > URL: https://issues.apache.org/jira/browse/HIVE-11928 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Jagruti Varia >Assignee: Prasanth Jayachandran > Attachments: HIVE-11928-branch-1.patch, HIVE-11928.1.patch, > HIVE-11928.1.patch > > > Similar to HIVE-11592 but for orc footer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11723) Incorrect string literal escaping
[
https://issues.apache.org/jira/browse/HIVE-11723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906809#comment-14906809
]
Szehon Ho commented on HIVE-11723:
--
+1
> Incorrect string literal escaping
> -
>
> Key: HIVE-11723
> URL: https://issues.apache.org/jira/browse/HIVE-11723
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.1, 2.0.0
>Reporter: Uri Laserson
>Assignee: Yongzhi Chen
> Attachments: HIVE-11723.1.patch
>
>
> When I execute the following queries
> {code}
> CREATE TABLE t_hive (f1 STRING);
> INSERT INTO t_hive VALUES ('Cooper\'s');
> SELECT * FROM t_hive;
> {code}
> via the Hive shell or through HiveServer2 directly (via impyla), I would
> expect that the result to be
> {code}
> Cooper's
> {code}
> but instead I actually get
> {code}
> Cooper\'s
> {code}
> Actually, I'm not sure how that {{INSERT}} query is not even a syntax error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906790#comment-14906790
]
Xuefu Zhang commented on HIVE-11880:
Looks like the patch needs to be rebased.
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-11306) Add a bloom-1 filter for Hybrid MapJoin spills
[ https://issues.apache.org/jira/browse/HIVE-11306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-11306: Assignee: Wei Zheng (was: Gopal V) > Add a bloom-1 filter for Hybrid MapJoin spills > -- > > Key: HIVE-11306 > URL: https://issues.apache.org/jira/browse/HIVE-11306 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Wei Zheng > Attachments: HIVE-11306.1.patch, HIVE-11306.2.patch, > HIVE-11306.3.patch, HIVE-11306.5.patch > > > HIVE-9277 implemented Spillable joins for Tez, which suffers from a > corner-case performance issue when joining wide small tables against a narrow > big table (like a user info table join events stream). > The fact that the wide table is spilled causes extra IO, even though the nDV > of the join key might be in the thousands. > A cheap bloom-1 filter would add a massive performance gain for such queries, > massively cutting down on the spill IO costs for the big-table spills. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11947) mssql upgrade scripts contains invalid character
[
https://issues.apache.org/jira/browse/HIVE-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906771#comment-14906771
]
Ashutosh Chauhan commented on HIVE-11947:
-
e.g., {{upgrade-0.14.0-to-1.1.0.mssql.sql}} fails with
{{'005-HIVE-9296.mssql.sql;': Invalid filename}}
> mssql upgrade scripts contains invalid character
>
>
> Key: HIVE-11947
> URL: https://issues.apache.org/jira/browse/HIVE-11947
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.2.0, 1.1.0
>Reporter: Huan Huang
>
> upgrade scripts dont execute as a result
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11306) Add a bloom-1 filter for Hybrid MapJoin spills
[ https://issues.apache.org/jira/browse/HIVE-11306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-11306: - Attachment: HIVE-11306.5.patch Upload patch 5 for testing > Add a bloom-1 filter for Hybrid MapJoin spills > -- > > Key: HIVE-11306 > URL: https://issues.apache.org/jira/browse/HIVE-11306 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Wei Zheng > Attachments: HIVE-11306.1.patch, HIVE-11306.2.patch, > HIVE-11306.3.patch, HIVE-11306.5.patch > > > HIVE-9277 implemented Spillable joins for Tez, which suffers from a > corner-case performance issue when joining wide small tables against a narrow > big table (like a user info table join events stream). > The fact that the wide table is spilled causes extra IO, even though the nDV > of the join key might be in the thousands. > A cheap bloom-1 filter would add a massive performance gain for such queries, > massively cutting down on the spill IO costs for the big-table spills. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
