[jira] [Commented] (HIVE-16595) fix syntax in Hplsql.g4

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002182#comment-16002182
 ] 

Hive QA commented on HIVE-16595:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867038/HIVE-16595.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10654 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1] 
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
 (batchId=155)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5128/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5128/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5128/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867038 - PreCommit-HIVE-Build

> fix syntax in Hplsql.g4
> ---
>
> Key: HIVE-16595
> URL: https://issues.apache.org/jira/browse/HIVE-16595
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Reporter: Yishuang Lu
>Assignee: Yishuang Lu
> Fix For: 1.2.3
>
> Attachments: HIVE-16595.1.patch, HIVE-16595.2.patch, 
> HIVE-16595.3.patch
>
>
> According to https://github.com/antlr/antlr4/issues/118, incorrect error 
> message might return if the start rule does not contain an explicit EOF 
> transition. It is better to add EOF for the first rule in grammar.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16558) In the hiveserver2.jsp page, when you click Drilldown to view the details of the Closed Queries, the Chinese show garbled

[ZhangBing Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhangBing Lin updated HIVE-16558:
-
Description: 
In QueryProfileImpl.jamon,We see the following settings：


  

HiveServer2






  
So we should set the response code to utf-8, which can avoid Chinese garbled or 
other languages garbled,Please check it!
 This patch can only solve Drilldown read data from the cache when the 
Chinese show abnormal, and can not solve the Explain plan from the database 
query Chinese garbled, because the Chinese stored in the database is garbled.
 If you want to solve this problem, you need to modify the database 
encoding format.
 I try to modify the mysql 'columns_v2' table field 'comment' encoding 
utf8，Is this okay？



  was:
In QueryProfileImpl.jamon,We see the following settings：


  

HiveServer2






  
So we should set the response code to utf-8, which can avoid Chinese garbled or 
other languages garbled,Please check it!
 This patch can only solve Drilldown read data from the cache when the 
Chinese show abnormal, and can not solve the Explain plan from the database 
query Chinese garbled, because the Chinese stored in the database is garbled.
 If you want to solve this problem, you need to modify the database 
encoding format.
 For example, the use of metastore is mysql, the storage of Chinese 
comments on the table is latin1 coding, and mysql support for utf-8 is not very 
good, limit the length of 767 characters.



> In the hiveserver2.jsp page, when you click Drilldown to view the details of 
> the Closed Queries, the Chinese show garbled
> -
>
> Key: HIVE-16558
> URL: https://issues.apache.org/jira/browse/HIVE-16558
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Fix For: 3.0.0
>
> Attachments: HIVE-16558.1.patch, HIVE-16558.2.patch
>
>
> In QueryProfileImpl.jamon,We see the following settings：
> 
> 
>   
> 
> HiveServer2
> 
> 
> 
> 
> 
>   
> So we should set the response code to utf-8, which can avoid Chinese garbled 
> or other languages garbled,Please check it!
>  This patch can only solve Drilldown read data from the cache when the 
> Chinese show abnormal, and can not solve the Explain plan from the database 
> query Chinese garbled, because the Chinese stored in the database is garbled.
>  If you want to solve this problem, you need to modify the database 
> encoding format.
>  I try to modify the mysql 'columns_v2' table field 'comment' encoding 
> utf8，Is this okay？
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-16558) In the hiveserver2.jsp page, when you click Drilldown to view the details of the Closed Queries, the Chinese show garbled

[ZhangBing Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000167#comment-16000167
 ] 

ZhangBing Lin edited comment on HIVE-16558 at 5/9/17 7:01 AM:
--

[~xuefuz] 
 This patch can only solve Drilldown read data from the cache when the 
Chinese show abnormal, and can not solve the Explain plan from the database 
query Chinese garbled, because the Chinese stored in the database is garbled.
 If you want to solve this problem, you need to modify the database 
encoding format.
 I try to modify the mysql 'columns_v2' table field 'comment' encoding 
utf8，Is this okay？


was (Author: linzhangbing):
[~xuefuz] 
 This patch can only solve Drilldown read data from the cache when the 
Chinese show abnormal, and can not solve the Explain plan from the database 
query Chinese garbled, because the Chinese stored in the database is garbled.
 If you want to solve this problem, you need to modify the database 
encoding format.
 For example, the use of metastore is mysql, the storage of Chinese 
comments on the table is latin1 coding, and mysql support for utf-8 is not very 
good, limit the length of 767 characters.
Do we add a discussion to see how to solve this problem in Chinese garbled? 
Do you have any good idea for this?

> In the hiveserver2.jsp page, when you click Drilldown to view the details of 
> the Closed Queries, the Chinese show garbled
> -
>
> Key: HIVE-16558
> URL: https://issues.apache.org/jira/browse/HIVE-16558
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Fix For: 3.0.0
>
> Attachments: HIVE-16558.1.patch, HIVE-16558.2.patch
>
>
> In QueryProfileImpl.jamon,We see the following settings：
> 
> 
>   
> 
> HiveServer2
> 
> 
> 
> 
> 
>   
> So we should set the response code to utf-8, which can avoid Chinese garbled 
> or other languages garbled,Please check it!
>  This patch can only solve Drilldown read data from the cache when the 
> Chinese show abnormal, and can not solve the Explain plan from the database 
> query Chinese garbled, because the Chinese stored in the database is garbled.
>  If you want to solve this problem, you need to modify the database 
> encoding format.
>  I try to modify the mysql 'columns_v2' table field 'comment' encoding 
> utf8，Is this okay？
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16530) Add HS2 operation logs and improve logs for REPL commands

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002186#comment-16002186
 ] 

Sankar Hariappan commented on HIVE-16530:
-

Thanks [~thejas] for the commit!

> Add HS2 operation logs and improve logs for REPL commands
> -
>
> Key: HIVE-16530
> URL: https://issues.apache.org/jira/browse/HIVE-16530
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 2.2.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: Bootstrap_ReplDump_Console_Log.png, 
> Bootstrap_ReplLoad_Console_Log.png, HIVE-16530.01.patch, HIVE-16530.02.patch, 
> Incremental_ReplDump_Console_Log.png, Incremental_ReplLoad_Console_Log.png
>
>
> This is the log format that is being proposed for Hive Repl query logs
> For bootstrap case:
> Hive will log a message for each object as it is being bootstrapped and it 
> will be in the following sequence
> - Tables first (views are tables for this purpose) at time including 
> partitions (depth first), followed by functions, constraints 
> - The ordering is based on the ordering of listStatus API of HDFS
> - For each object, a message at the beginning of the replication will be 
> logged
> - Every partition bootstrapped will be followed by a message saying the 
> number of partitions bootstrapped so far (for the table) and the partition 
> name
> - And a message at the end of bootstrap of an object
> Incremental case:
> - We will have DB Name, event id and event type  will be part of the log 
> header (for debugging/troubleshooting)
> - We will have information of current event ID and total number of events to 
> replicate for every event replicated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-16609:
--
Attachment: HIVE-16609.1.patch

> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16609.1.patch
>
>
> A variation of alter_partition_change_col.q produces wrong result:
> {code}
> SET hive.exec.dynamic.partition.mode = nonstrict;
> create table alter_partition_change_col0 (c1 string, c2 string);
> load data local inpath 'dec.txt' overwrite into table 
> alter_partition_change_col0;
> create table alter_partition_change_col1 (c1 string, c2 string) partitioned 
> by (p1 string comment 'Column p1', p2 string comment 'Column p2');
> insert overwrite table alter_partition_change_col1 partition (p1, p2)
>   select c1, c2, 'abc', '123' from alter_partition_change_col0
>   union all
>   select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
> select * from alter_partition_change_col1 where 
> p1='__HIVE_DEFAULT_PARTITION__' or lower(p1)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> We need another condition containing a udf so the condition is not recognized 
> by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like 
> HIVE-11208 breaks it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16610) Semijoin Hint : Should be able to handle more than one hint per alias

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-16610:
--
Status: Patch Available  (was: In Progress)

> Semijoin Hint : Should be able to handle more than one hint per alias
> -
>
> Key: HIVE-16610
> URL: https://issues.apache.org/jira/browse/HIVE-16610
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>
> Currently the semi join hints can be used to create only one semi join 
> optimization per alias which is very limiting.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-16609:
--
Status: Patch Available  (was: Open)

> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16609.1.patch
>
>
> A variation of alter_partition_change_col.q produces wrong result:
> {code}
> SET hive.exec.dynamic.partition.mode = nonstrict;
> create table alter_partition_change_col0 (c1 string, c2 string);
> load data local inpath 'dec.txt' overwrite into table 
> alter_partition_change_col0;
> create table alter_partition_change_col1 (c1 string, c2 string) partitioned 
> by (p1 string comment 'Column p1', p2 string comment 'Column p2');
> insert overwrite table alter_partition_change_col1 partition (p1, p2)
>   select c1, c2, 'abc', '123' from alter_partition_change_col0
>   union all
>   select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
> select * from alter_partition_change_col1 where 
> p1='__HIVE_DEFAULT_PARTITION__' or lower(p1)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> We need another condition containing a udf so the condition is not recognized 
> by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like 
> HIVE-11208 breaks it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16558) In the hiveserver2.jsp page, when you click Drilldown to view the details of the Closed Queries, the Chinese show garbled，Use mysql to store hive metadata

[ZhangBing Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhangBing Lin updated HIVE-16558:
-
Summary: In the hiveserver2.jsp page, when you click Drilldown to view the 
details of the Closed Queries, the Chinese show garbled，Use mysql to store hive 
metadata  (was: In the hiveserver2.jsp page, when you click Drilldown to view 
the details of the Closed Queries, the Chinese show garbled)

> In the hiveserver2.jsp page, when you click Drilldown to view the details of 
> the Closed Queries, the Chinese show garbled，Use mysql to store hive metadata
> --
>
> Key: HIVE-16558
> URL: https://issues.apache.org/jira/browse/HIVE-16558
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Fix For: 3.0.0
>
> Attachments: HIVE-16558.1.patch, HIVE-16558.2.patch
>
>
> In QueryProfileImpl.jamon,We see the following settings：
> 
> 
>   
> 
> HiveServer2
> 
> 
> 
> 
> 
>   
> So we should set the response code to utf-8, which can avoid Chinese garbled 
> or other languages garbled,Please check it!
>  This patch can only solve Drilldown read data from the cache when the 
> Chinese show abnormal, and can not solve the Explain plan from the database 
> query Chinese garbled, because the Chinese stored in the database is garbled.
>  If you want to solve this problem, you need to modify the database 
> encoding format.
>  I try to modify the mysql 'columns_v2' table field 'comment' encoding 
> utf8，Is this okay？
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16558) Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, when you click Drilldown to view the details of the Closed Queries, the Chinese show garbled

[ZhangBing Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhangBing Lin updated HIVE-16558:
-
Summary: Prerequisites Use mysql to store hive metadata，in the 
hiveserver2.jsp page, when you click Drilldown to view the details of the 
Closed Queries, the Chinese show garbled  (was: In the hiveserver2.jsp page, 
when you click Drilldown to view the details of the Closed Queries, the Chinese 
show garbled，Use mysql to store hive metadata)

> Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, 
> when you click Drilldown to view the details of the Closed Queries, the 
> Chinese show garbled
> 
>
> Key: HIVE-16558
> URL: https://issues.apache.org/jira/browse/HIVE-16558
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Fix For: 3.0.0
>
> Attachments: HIVE-16558.1.patch, HIVE-16558.2.patch
>
>
> In QueryProfileImpl.jamon,We see the following settings：
> 
> 
>   
> 
> HiveServer2
> 
> 
> 
> 
> 
>   
> So we should set the response code to utf-8, which can avoid Chinese garbled 
> or other languages garbled,Please check it!
>  This patch can only solve Drilldown read data from the cache when the 
> Chinese show abnormal, and can not solve the Explain plan from the database 
> query Chinese garbled, because the Chinese stored in the database is garbled.
>  If you want to solve this problem, you need to modify the database 
> encoding format.
>  I try to modify the mysql 'columns_v2' table field 'comment' encoding 
> utf8，Is this okay？
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16610) Semijoin Hint : Should be able to handle more than one hint per alias

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-16610:
--
Attachment: HIVE-16610.1.patch

Initial patch. Go to RB for details at,
https://reviews.apache.org/r/59080/

> Semijoin Hint : Should be able to handle more than one hint per alias
> -
>
> Key: HIVE-16610
> URL: https://issues.apache.org/jira/browse/HIVE-16610
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-16610.1.patch
>
>
> Currently the semi join hints can be used to create only one semi join 
> optimization per alias which is very limiting.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1600#comment-1600
 ] 

Sankar Hariappan commented on HIVE-13652:
-

Thanks [~thejas] for the commit!

> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, TODOC3.0, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16269) enable incremental function dump to be loaded via repl load

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002232#comment-16002232
 ] 

Hive QA commented on HIVE-16269:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867041/HIVE-16269.2.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10656 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=148)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5129/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5129/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5129/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867041 - PreCommit-HIVE-Build

> enable incremental function dump to be loaded via repl load 
> 
>
> Key: HIVE-16269
> URL: https://issues.apache.org/jira/browse/HIVE-16269
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Attachments: HIVE-16269.1.patch, HIVE-16269.2.patch
>
>
> depends if there is additional spec elements we put out as part of HIVE-16268



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases

[liyunzhang_intel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002260#comment-16002260
 ] 

liyunzhang_intel commented on HIVE-16600:
-

[~lirui]: thanks for review.
bq.I remember you mentioned there was something wrong when you first enable 
parallel order by for multi insert. Have you figured out what was the cause?
{code}
set hive.exec.reducers.bytes.per.reducer=256; 
set hive.optimize.sampling.orderby=true;
set hive.execution.engine=spark;
drop table if exists e1; 
drop table if exists e2; 
create table e1 (key string, keyD double);
create table e2 (key string, keyD double, value string);
FROM (select key, cast(key as double) as keyD, value from src order by key) a
INSERT OVERWRITE TABLE e1
SELECT key, COUNT(distinct value) group by key 
INSERT OVERWRITE TABLE e2
SELECT key, sum(keyD), value group by key, value;
select * from e1; 
select * from e2; 
{code}
what i was confused last week is 
the result of e1 and e2 is not order by the key when we set  
hive.exec.reducers.bytes.per.reducer=256( this makes the parallel order by).
But forget my confuse because the result of e1 and e2 should not be in order 
because this is a orderby+groupby case, the result of groupby may not be in 
order.




> Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel 
> order by in multi_insert cases
> 
>
> Key: HIVE-16600
> URL: https://issues.apache.org/jira/browse/HIVE-16600
> Project: Hive
>  Issue Type: Sub-task
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16600.1.patch
>
>
> multi_insert_gby.case.q
> {code}
> set hive.exec.reducers.bytes.per.reducer=256;
> set hive.optimize.sampling.orderby=true;
> drop table if exists e1;
> drop table if exists e2;
> create table e1 (key string, value string);
> create table e2 (key string);
> FROM (select key, cast(key as double) as keyD, value from src order by key) a
> INSERT OVERWRITE TABLE e1
> SELECT key, value
> INSERT OVERWRITE TABLE e2
> SELECT key;
> select * from e1;
> select * from e2;
> {code} 
> the parallelism of Sort is 1 even we enable parallel order 
> by("hive.optimize.sampling.orderby" is set as "true").  This is not 
> reasonable because the parallelism  should be calcuated by  
> [Utilities.estimateReducers|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L170]
> this is because SetSparkReducerParallelism#needSetParallelism returns false 
> when [children size of 
> RS|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L207]
>  is greater than 1.
> in this case, the children size of {{RS[2]}} is two.
> the logical plan of the case
> {code}
>TS[0]-SEL[1]-RS[2]-SEL[3]-SEL[4]-FS[5]
> -SEL[6]-FS[7]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16558) Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, when you click Drilldown to view the details of the Closed Queries, the Chinese show garbled

[ZhangBing Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhangBing Lin updated HIVE-16558:
-
Affects Version/s: (was: 2.1.0)

> Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, 
> when you click Drilldown to view the details of the Closed Queries, the 
> Chinese show garbled
> 
>
> Key: HIVE-16558
> URL: https://issues.apache.org/jira/browse/HIVE-16558
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Fix For: 3.0.0
>
> Attachments: HIVE-16558.1.patch, HIVE-16558.2.patch
>
>
> In QueryProfileImpl.jamon,We see the following settings：
> 
> 
>   
> 
> HiveServer2
> 
> 
> 
> 
> 
>   
> So we should set the response code to utf-8, which can avoid Chinese garbled 
> or other languages garbled,Please check it!
>  This patch can only solve Drilldown read data from the cache when the 
> Chinese show abnormal, and can not solve the Explain plan from the database 
> query Chinese garbled, because the Chinese stored in the database is garbled.
>  If you want to solve this problem, you need to modify the database 
> encoding format.
>  I try to modify the mysql 'columns_v2' table field 'comment' encoding 
> utf8，Is this okay？
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16558) Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, when you click Drilldown to view the details of the Closed Queries, the Chinese show garbled

[ZhangBing Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhangBing Lin updated HIVE-16558:
-
Affects Version/s: 3.0.0

> Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, 
> when you click Drilldown to view the details of the Closed Queries, the 
> Chinese show garbled
> 
>
> Key: HIVE-16558
> URL: https://issues.apache.org/jira/browse/HIVE-16558
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Fix For: 3.0.0
>
> Attachments: HIVE-16558.1.patch, HIVE-16558.2.patch
>
>
> In QueryProfileImpl.jamon,We see the following settings：
> 
> 
>   
> 
> HiveServer2
> 
> 
> 
> 
> 
>   
> So we should set the response code to utf-8, which can avoid Chinese garbled 
> or other languages garbled,Please check it!
>  This patch can only solve Drilldown read data from the cache when the 
> Chinese show abnormal, and can not solve the Explain plan from the database 
> query Chinese garbled, because the Chinese stored in the database is garbled.
>  If you want to solve this problem, you need to modify the database 
> encoding format.
>  I try to modify the mysql 'columns_v2' table field 'comment' encoding 
> utf8，Is this okay？
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16559) Parquet schema evolution for partitioned tables may break if table and partition serdes differ

[Barna Zsombor Klara (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002276#comment-16002276
 ] 

Barna Zsombor Klara commented on HIVE-16559:


Just to clarify, technically it is possible to fix this issue in the 
{{ObjectInspectorConverters}} by matching the converters between the input and 
output fields based on field names (currently they are matched based on field 
order). But this would mean an overhead whenever we select from a table, even 
when there is no schema evolution. I find this tradeoff to be not worth it 
especially since altering the table with the cascade option yields correct 
results with a one time overhead, when the column changes are propagated to the 
partitions.

> Parquet schema evolution for partitioned tables may break if table and 
> partition serdes differ
> --
>
> Key: HIVE-16559
> URL: https://issues.apache.org/jira/browse/HIVE-16559
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Fix For: 3.0.0
>
> Attachments: HIVE-16559.01.patch
>
>
> Parquet schema evolution should make it possible to have partitions/tables 
>  backed by files with different schemas. Hive should match the table columns 
> with file columns based on the column name if possible.
> However if the serde for a table is missing columns from the serde of a 
> partition Hive fails to match the columns together.
> Steps to reproduce:
> {code}
> CREATE TABLE myparquettable_parted
> (
>   name string,
>   favnumber int,
>   favcolor string,
>   age int,
>   favpet string
> )
> PARTITIONED BY (day string)
> STORED AS PARQUET;
> INSERT OVERWRITE TABLE myparquettable_parted
> PARTITION(day='2017-04-04')
> SELECT
>'mary' as name,
>5 AS favnumber,
>'blue' AS favcolor,
>35 AS age,
>'dog' AS favpet;
> alter table myparquettable_parted
> REPLACE COLUMNS
> (
> favnumber int,
> age int
> );   

[jira] [Commented] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002305#comment-16002305
 ] 

Hive QA commented on HIVE-16582:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867035/HIVE-16582.4.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10655 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=226)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5130/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5130/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5130/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867035 - PreCommit-HIVE-Build

> HashTableLoader should log info about the input, rows, size etc.
> 
>
> Key: HIVE-16582
> URL: https://issues.apache.org/jira/browse/HIVE-16582
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-16582.1.patch, HIVE-16582.2.patch, 
> HIVE-16582.3.patch, HIVE-16582.4.patch
>
>
> Will be useful to log the following info during hash table loading
> - input name
> - number of rows 
> - estimated data size (LLAP tracks this)
> - object cache key



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16607) ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-16607:
--
Attachment: HIVE-16607.02.patch

Forgot to add the last change to the patch:

Operation constructor should create a new queryId

> ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID
> -
>
> Key: HIVE-16607
> URL: https://issues.apache.org/jira/browse/HIVE-16607
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, Logging
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16607.02.patch, HIVE-16607.patch
>
>
> Creating a new {{QueryState}} object regenerates the HIVEQUERYID stored in 
> the {{HiveConf}}.
> In HiveServer logs it makes hard to follow the life of the query since a new 
> queryid is assigned to the query during the execution.
> Since BeeLine is showing the operation logs based on the queryid, only the 
> first several line of the logs is showed in BeeLine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16613) SaslClientHandler.sendHello is eating exceptions

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-16613:
--
Component/s: Spark

> SaslClientHandler.sendHello is eating exceptions
> 
>
> Key: HIVE-16613
> URL: https://issues.apache.org/jira/browse/HIVE-16613
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16613) SaslClientHandler.sendHello is eating exceptions

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li reassigned HIVE-16613:
-


> SaslClientHandler.sendHello is eating exceptions
> 
>
> Key: HIVE-16613
> URL: https://issues.apache.org/jira/browse/HIVE-16613
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16613) SaslClientHandler.sendHello is eating exceptions

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-16613:
--
Description: 
If we hit exceptions when sending hello to server, the method fails silently. 
Finally the remote driver fails with SASL timeout instead of the real cause, 
making it difficult for debugging.
To reproduce: throw some exception in {{KryoMessageCodec.encode}}.

> SaslClientHandler.sendHello is eating exceptions
> 
>
> Key: HIVE-16613
> URL: https://issues.apache.org/jira/browse/HIVE-16613
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>
> If we hit exceptions when sending hello to server, the method fails silently. 
> Finally the remote driver fails with SASL timeout instead of the real cause, 
> making it difficult for debugging.
> To reproduce: throw some exception in {{KryoMessageCodec.encode}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16613) SaslClientHandler.sendHello is eating exceptions

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-16613:
--
Attachment: HIVE-16613.1.patch

> SaslClientHandler.sendHello is eating exceptions
> 
>
> Key: HIVE-16613
> URL: https://issues.apache.org/jira/browse/HIVE-16613
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-16613.1.patch
>
>
> If we hit exceptions when sending hello to server, the method fails silently. 
> Finally the remote driver fails with SASL timeout instead of the real cause, 
> making it difficult for debugging.
> To reproduce: throw some exception in {{KryoMessageCodec.encode}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16613) SaslClientHandler.sendHello is eating exceptions

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-16613:
--
Status: Patch Available  (was: Open)

> SaslClientHandler.sendHello is eating exceptions
> 
>
> Key: HIVE-16613
> URL: https://issues.apache.org/jira/browse/HIVE-16613
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-16613.1.patch
>
>
> If we hit exceptions when sending hello to server, the method fails silently. 
> Finally the remote driver fails with SASL timeout instead of the real cause, 
> making it difficult for debugging.
> To reproduce: throw some exception in {{KryoMessageCodec.encode}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-2615) CTAS with literal NULL creates VOID type

[Lantao Jin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lantao Jin reassigned HIVE-2615:


Assignee: Lantao Jin  (was: Zhuoluo Yang)

> CTAS with literal NULL creates VOID type
> 
>
> Key: HIVE-2615
> URL: https://issues.apache.org/jira/browse/HIVE-2615
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0
>Reporter: David Phillips
>Assignee: Lantao Jin
> Fix For: 0.12.0
>
> Attachments: HIVE-2615.1.patch
>
>
> Create the table with a column that always contains NULL:
> {quote}
> hive> create table bad as select 1 x, null z from dual; 
> {quote}
> Because there's no type, Hive gives it the VOID type:
> {quote}
> hive> describe bad;
> OK
> x int 
> z void
> {quote}
> This seems weird, because AFAIK, there is no normal way to create a column of 
> type VOID.  The problem is that the table can't be queried:
> {quote}
> hive> select * from bad;
> OK
> Failed with exception java.io.IOException:java.lang.RuntimeException: 
> Internal error: no LazyObject for VOID
> {quote}
> Worse, even if you don't select that field, the query fails at runtime:
> {quote}
> hive> select x from bad;
> ...
> FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.MapRedTask
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16593) SparkClientFactory.stop may prevent JVM from exiting

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002387#comment-16002387
 ] 

Hive QA commented on HIVE-16593:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867046/HIVE-16593.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10655 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=226)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal] 
(batchId=9)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5131/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5131/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5131/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867046 - PreCommit-HIVE-Build

> SparkClientFactory.stop may prevent JVM from exiting
> 
>
> Key: HIVE-16593
> URL: https://issues.apache.org/jira/browse/HIVE-16593
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-16593.1.patch
>
>
> When we receive SIGINT more than once, we call System.exit to terminate the 
> JVM. System.exit runs the shutdown hooks which in turn calls 
> SparkClientFactory.stop. All the methods in SparkClientFactory are 
> synchronized and at this point, we may be waiting to create a SparkClientImpl 
> in SparkClientFactory.createClient. Therefore SparkClientFactory.stop will be 
> blocked until SparkClientFactory.createClient returns.
> The reason why we may be waiting to create SparkClientImpl is usually to wait 
> for RemoteDriver to connect. If RemoteDriver runs into problem, the JVM won't 
> exit until we timeout.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16556) Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES table

[Naveen Gangam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002421#comment-16002421
 ] 

Naveen Gangam commented on HIVE-16556:
--

Fix has been pushed to master. Thank you for your contributions [~vihangk1]

> Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES 
> table
> 
>
> Key: HIVE-16556
> URL: https://issues.apache.org/jira/browse/HIVE-16556
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16556.01.patch, HIVE-16556.02.patch, 
> HIVE-16556.03.patch, HIVE-16556.04.patch, HIVE-16556.05.patch
>
>
> sub-task to modify schema tool and its related changes so that the new table 
> is added to the schema when schematool initializes or upgrades the schema.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16556) Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES table

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-16556:
-
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

> Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES 
> table
> 
>
> Key: HIVE-16556
> URL: https://issues.apache.org/jira/browse/HIVE-16556
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Fix For: 3.0.0
>
> Attachments: HIVE-16556.01.patch, HIVE-16556.02.patch, 
> HIVE-16556.03.patch, HIVE-16556.04.patch, HIVE-16556.05.patch
>
>
> sub-task to modify schema tool and its related changes so that the new table 
> is added to the schema when schematool initializes or upgrades the schema.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-2615) CTAS with literal NULL creates VOID type

[Lantao Jin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lantao Jin reassigned HIVE-2615:


Assignee: Zhuoluo Yang  (was: Lantao Jin)

> CTAS with literal NULL creates VOID type
> 
>
> Key: HIVE-2615
> URL: https://issues.apache.org/jira/browse/HIVE-2615
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0
>Reporter: David Phillips
>Assignee: Zhuoluo Yang
> Fix For: 0.12.0
>
> Attachments: HIVE-2615.1.patch
>
>
> Create the table with a column that always contains NULL:
> {quote}
> hive> create table bad as select 1 x, null z from dual; 
> {quote}
> Because there's no type, Hive gives it the VOID type:
> {quote}
> hive> describe bad;
> OK
> x int 
> z void
> {quote}
> This seems weird, because AFAIK, there is no normal way to create a column of 
> type VOID.  The problem is that the table can't be queried:
> {quote}
> hive> select * from bad;
> OK
> Failed with exception java.io.IOException:java.lang.RuntimeException: 
> Internal error: no LazyObject for VOID
> {quote}
> Worse, even if you don't select that field, the query fails at runtime:
> {quote}
> hive> select x from bad;
> ...
> FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.MapRedTask
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16272) support for drop function in incremental replication

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek updated HIVE-16272:
---
Attachment: HIVE-16272.1.patch

This is not readily applicable on master as it depends on HIVE-16269 changes, 
just adding here for first cut.

> support for drop function in incremental replication 
> -
>
> Key: HIVE-16272
> URL: https://issues.apache.org/jira/browse/HIVE-16272
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Attachments: HIVE-16272.1.patch
>
>
> drop function should work in incremental dump and incremental load



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16330) Improve plans for scalar subquery with aggregates

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002463#comment-16002463
 ] 

Hive QA commented on HIVE-16330:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867015/HIVE-16330.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10655 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select]
 (batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=148)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_corr_multi_rows]
 (batchId=88)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query1] (batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query30] 
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query6] (batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query81] 
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query9] (batchId=230)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5132/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5132/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5132/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867015 - PreCommit-HIVE-Build

> Improve plans for scalar subquery with aggregates
> -
>
> Key: HIVE-16330
> URL: https://issues.apache.org/jira/browse/HIVE-16330
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16330.1.patch
>
>
> Scalar subquery plans are generated with a count(*) on subquery which is fed 
> to {{sq_count_check}} UDF. This is to make sure at runtime that there is at 
> most one row generated by scalar subquery. 
> We can avoid generating this extra count(*) for scalar subqueries with 
> aggregates and windowing since such queries are guaranteed to generate at 
> most one row. e.g. {code:SQL} select * from part where p_size > (select 
> max(p_size) from part) {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16584) Warning messages should use LogHelper.printInfo instead of printing to the infoStream directly

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002471#comment-16002471
 ] 

Peter Vary commented on HIVE-16584:
---

Could you please review [~thejas], [~ashutoshc]?

Thanks,
Peter

> Warning messages should use LogHelper.printInfo instead of printing to the 
> infoStream directly
> --
>
> Key: HIVE-16584
> URL: https://issues.apache.org/jira/browse/HIVE-16584
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16584.02.patch, HIVE-16584.03.patch, 
> HIVE-16584.patch
>
>
> There are several cases when warning messages are printed to the console 
> outputstream directly. These warnings do not show up in the BeeLine output.
> We should use the the printInfo method instead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-16584) Warning messages should use LogHelper.printInfo instead of printing to the infoStream directly

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000483#comment-16000483
 ] 

Peter Vary edited comment on HIVE-16584 at 5/9/17 10:52 AM:


The patch contains the following changes above the previous ones:
- Without the patch the original code prints the warning messages regardless 
off the isSilent flag. This new patch reproduces the original behavior for 
HiveCli with using printInfo("Message", false) instead of printInfo("Message").
- For the RelOptHiveTable.updateColStats, I kept the printInfo("Message") since 
the original behavior was to print the warning message only if the session is 
not silent. 
- Also fixed some comment messages.


was (Author: pvary):
The patch contains the following changes above the previous ones:
- Without the patch the original code prints the warning messages regardless 
off the isSilent flag. This new patch reproduces the original behavior for 
HiveCli with using printInfo("Message", false) instead of printInfo("Message").
- For the RelOptHiveTable.updateColStats, I kept the printInfo("Message") since 
the original behavior was to print the warning message will be printed only if 
the session is not silent. 
- Also fixed some comment messages.

> Warning messages should use LogHelper.printInfo instead of printing to the 
> infoStream directly
> --
>
> Key: HIVE-16584
> URL: https://issues.apache.org/jira/browse/HIVE-16584
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16584.02.patch, HIVE-16584.03.patch, 
> HIVE-16584.patch
>
>
> There are several cases when warning messages are printed to the console 
> outputstream directly. These warnings do not show up in the BeeLine output.
> We should use the the printInfo method instead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16602) Implement shared scans with Tez

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-16602:
---
Attachment: HIVE-16602.01.patch

> Implement shared scans with Tez
> ---
>
> Key: HIVE-16602
> URL: https://issues.apache.org/jira/browse/HIVE-16602
> Project: Hive
>  Issue Type: New Feature
>  Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16602.01.patch, HIVE-16602.patch
>
>
> Given a query plan, the goal is to identify scans on input tables that can be 
> merged so the data is read only once. Optimization will be carried out at the 
> physical level.
> In the longer term, identification of equivalent expressions and 
> reutilization of intermediary results should be done at the logical layer via 
> Spool operator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16593) SparkClientFactory.stop may prevent JVM from exiting

[Rui Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002515#comment-16002515
 ] 

Rui Li commented on HIVE-16593:
---

Failures not related. [~xuefuz], could you take a look? Thanks.

> SparkClientFactory.stop may prevent JVM from exiting
> 
>
> Key: HIVE-16593
> URL: https://issues.apache.org/jira/browse/HIVE-16593
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-16593.1.patch
>
>
> When we receive SIGINT more than once, we call System.exit to terminate the 
> JVM. System.exit runs the shutdown hooks which in turn calls 
> SparkClientFactory.stop. All the methods in SparkClientFactory are 
> synchronized and at this point, we may be waiting to create a SparkClientImpl 
> in SparkClientFactory.createClient. Therefore SparkClientFactory.stop will be 
> blocked until SparkClientFactory.createClient returns.
> The reason why we may be waiting to create SparkClientImpl is usually to wait 
> for RemoteDriver to connect. If RemoteDriver runs into problem, the JVM won't 
> exit until we timeout.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16558) Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, when you click Drilldown to view the details of the Closed Queries, the Chinese show garbled

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002530#comment-16002530
 ] 

Hive QA commented on HIVE-16558:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867055/HIVE-16558.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10657 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=148)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDate (batchId=178)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5133/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5133/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5133/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867055 - PreCommit-HIVE-Build

> Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, 
> when you click Drilldown to view the details of the Closed Queries, the 
> Chinese show garbled
> 
>
> Key: HIVE-16558
> URL: https://issues.apache.org/jira/browse/HIVE-16558
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Fix For: 3.0.0
>
> Attachments: HIVE-16558.1.patch, HIVE-16558.2.patch
>
>
> In QueryProfileImpl.jamon,We see the following settings：
> 
> 
>   
> 
> HiveServer2
> 
> 
> 
> 
> 
>   
> So we should set the response code to utf-8, which can avoid Chinese garbled 
> or other languages garbled,Please check it!
>  This patch can only solve Drilldown read data from the cache when the 
> Chinese show abnormal, and can not solve the Explain plan from the database 
> query Chinese garbled, because the Chinese stored in the database is garbled.
>  If you want to solve this problem, you need to modify the database 
> encoding format.
>  I try to modify the mysql 'columns_v2' table field 'comment' encoding 
> utf8，Is this okay？
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16558) Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, when you click Drilldown to view the details of the Closed Queries, the Chinese show garbled

[ZhangBing Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002535#comment-16002535
 ] 

ZhangBing Lin commented on HIVE-16558:
--

Test failures are unrelated,please check it!

> Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, 
> when you click Drilldown to view the details of the Closed Queries, the 
> Chinese show garbled
> 
>
> Key: HIVE-16558
> URL: https://issues.apache.org/jira/browse/HIVE-16558
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Fix For: 3.0.0
>
> Attachments: HIVE-16558.1.patch, HIVE-16558.2.patch
>
>
> In QueryProfileImpl.jamon,We see the following settings：
> 
> 
>   
> 
> HiveServer2
> 
> 
> 
> 
> 
>   
> So we should set the response code to utf-8, which can avoid Chinese garbled 
> or other languages garbled,Please check it!
>  This patch can only solve Drilldown read data from the cache when the 
> Chinese show abnormal, and can not solve the Explain plan from the database 
> query Chinese garbled, because the Chinese stored in the database is garbled.
>  If you want to solve this problem, you need to modify the database 
> encoding format.
>  I try to modify the mysql 'columns_v2' table field 'comment' encoding 
> utf8，Is this okay？
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-16558) Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, when you click Drilldown to view the details of the Closed Queries, the Chinese show ga

[ZhangBing Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002535#comment-16002535
 ] 

ZhangBing Lin edited comment on HIVE-16558 at 5/9/17 11:44 AM:
---

Through logging，test failures are unrelated,please check it!


was (Author: linzhangbing):
Test failures are unrelated,please check it!

> Prerequisites Use mysql to store hive metadata，in the hiveserver2.jsp page, 
> when you click Drilldown to view the details of the Closed Queries, the 
> Chinese show garbled
> 
>
> Key: HIVE-16558
> URL: https://issues.apache.org/jira/browse/HIVE-16558
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: ZhangBing Lin
>Assignee: ZhangBing Lin
> Fix For: 3.0.0
>
> Attachments: HIVE-16558.1.patch, HIVE-16558.2.patch
>
>
> In QueryProfileImpl.jamon,We see the following settings：
> 
> 
>   
> 
> HiveServer2
> 
> 
> 
> 
> 
>   
> So we should set the response code to utf-8, which can avoid Chinese garbled 
> or other languages garbled,Please check it!
>  This patch can only solve Drilldown read data from the cache when the 
> Chinese show abnormal, and can not solve the Explain plan from the database 
> query Chinese garbled, because the Chinese stored in the database is garbled.
>  If you want to solve this problem, you need to modify the database 
> encoding format.
>  I try to modify the mysql 'columns_v2' table field 'comment' encoding 
> utf8，Is this okay？
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002603#comment-16002603
 ] 

Hive QA commented on HIVE-16609:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867059/HIVE-16609.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10657 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[select_dummy_source] 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_4] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcr] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup4] 
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=73)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[pcr] (batchId=123)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorNoBaseLotsOfDeltas 
(batchId=254)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5134/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5134/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5134/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867059 - PreCommit-HIVE-Build

> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16609.1.patch
>
>
> A variation of alter_partition_change_col.q produces wrong result:
> {code}
> SET hive.exec.dynamic.partition.mode = nonstrict;
> create table alter_partition_change_col0 (c1 string, c2 string);
> load data local inpath 'dec.txt' overwrite into table 
> alter_partition_change_col0;
> create table alter_partition_change_col1 (c1 string, c2 string) partitioned 
> by (p1 string comment 'Column p1', p2 string comment 'Column p2');
> insert overwrite table alter_partition_change_col1 partition (p1, p2)
>   select c1, c2, 'abc', '123' from alter_partition_change_col0
>   union all
>   select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
> select * from alter_partition_change_col1 where 
> p1='__HIVE_DEFAULT_PARTITION__' or lower(p1)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> We need another condition containing a udf so the condition is not recognized 
> by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like 
> HIVE-11208 breaks it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16615) Support Time Zone Specifiers (i.e. "at time zone X")

[Carter Shanklin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carter Shanklin updated HIVE-16615:
---
Summary: Support Time Zone Specifiers (i.e. "at time zone X")  (was: 
Support Time Zone Specifiers (i.e. "at time zone X")

> Support Time Zone Specifiers (i.e. "at time zone X")
> 
>
> Key: HIVE-16615
> URL: https://issues.apache.org/jira/browse/HIVE-16615
> Project: Hive
>  Issue Type: Improvement
>Reporter: Carter Shanklin
>
> HIVE-14412 introduces a timestamp-aware timezone.
> SQL has a concept of "time zone specifier" which applies to any datetime 
> value expression (which covers time/timestamp with and without timezones). 
> Hive lacks a time type so we can put that aside for a while.
> Examples:
>   a. select time_stamp_with_time_zone at time zone '-8:00';
>   b. select time_stamp_without_time_zone at time zone LOCAL;
> These statements would adjust the expression from its original timezone into 
> a known target timezone.
> Using  the time zone specifier results in a data type that has a time zone. 
> If the original expression lacked a time zone, the result has a time zone. If 
> the original expression had a time zone, the result still has a time zone, 
> possibly a different one.
> LOCAL means to use the session's original default time zone displacement.
> The standard says that dates are not supported with time zone specifiers. It 
> seems common to ignore this rule and allow this, by converting the date to a 
> timestamp and then applying the usual rule.
> The standard only requires an interval or the LOCAL keyword. Some databases 
> allow time zone identifiers like PST.
> Reference: SQL:2011 section 6.31



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16614) Support "set local time zone" statement

[Carter Shanklin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carter Shanklin updated HIVE-16614:
---
Description: 
HIVE-14412 introduces a timezone-aware timestamp.

SQL has a concept of default time zone displacements, which are transparently 
applied when converting between timezone-unaware types and timezone-aware types 
and, in Hive's case, are also used to shift a timezone aware type to a 
different time zone, depending on configuration.

SQL also provides that the default time zone displacement be settable at a 
session level, so that clients can access a database simultaneously from 
different time zones and see time values in their own time zone.

Currently the time zone displacement is fixed and is set based on the system 
time zone where the Hive client runs (HiveServer2 or Hive CLI). It will be more 
convenient for users if they have the ability to set their time zone of choice.

SQL defines "set time zone" with 2 ways of specifying the time zone, first 
using an interval and second using the special keyword LOCAL.

Examples:
• set time zone '-8:00';
• set time zone LOCAL;

LOCAL means to set the current default time zone displacement to the session's 
original default time zone displacement.

Reference: SQL:2011 section 19.4


  was:
HIVE-14412 introduces a timestamp-aware timezone.

SQL has a concept of default time zone displacements, which are transparently 
applied when converting between timezone-unaware types and timezone-aware types 
and, in Hive's case, are also used to shift a timezone aware type to a 
different time zone, depending on configuration.

SQL also provides that the default time zone displacement be settable at a 
session level, so that clients can access a database simultaneously from 
different time zones and see time values in their own time zone.

Currently the time zone displacement is fixed and is set based on the system 
time zone where the Hive client runs (HiveServer2 or Hive CLI). It will be more 
convenient for users if they have the ability to set their time zone of choice.

SQL defines "set time zone" with 2 ways of specifying the time zone, first 
using an interval and second using the special keyword LOCAL.

Examples:
• set time zone '-8:00';
• set time zone LOCAL;

LOCAL means to set the current default time zone displacement to the session's 
original default time zone displacement.

Reference: SQL:2011 section 19.4



> Support "set local time zone" statement
> ---
>
> Key: HIVE-16614
> URL: https://issues.apache.org/jira/browse/HIVE-16614
> Project: Hive
>  Issue Type: Improvement
>Reporter: Carter Shanklin
>
> HIVE-14412 introduces a timezone-aware timestamp.
> SQL has a concept of default time zone displacements, which are transparently 
> applied when converting between timezone-unaware types and timezone-aware 
> types and, in Hive's case, are also used to shift a timezone aware type to a 
> different time zone, depending on configuration.
> SQL also provides that the default time zone displacement be settable at a 
> session level, so that clients can access a database simultaneously from 
> different time zones and see time values in their own time zone.
> Currently the time zone displacement is fixed and is set based on the system 
> time zone where the Hive client runs (HiveServer2 or Hive CLI). It will be 
> more convenient for users if they have the ability to set their time zone of 
> choice.
> SQL defines "set time zone" with 2 ways of specifying the time zone, first 
> using an interval and second using the special keyword LOCAL.
> Examples:
>   • set time zone '-8:00';
>   • set time zone LOCAL;
> LOCAL means to set the current default time zone displacement to the 
> session's original default time zone displacement.
> Reference: SQL:2011 section 19.4



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16615) Support Time Zone Specifiers (i.e. "at time zone X")

[Carter Shanklin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carter Shanklin updated HIVE-16615:
---
Description: 
HIVE-14412 introduces a timezone-aware timestamp.

SQL has a concept of "time zone specifier" which applies to any datetime value 
expression (which covers time/timestamp with and without timezones). Hive lacks 
a time type so we can put that aside for a while.

Examples:
a. select time_stamp_with_time_zone at time zone '-8:00';
b. select time_stamp_without_time_zone at time zone LOCAL;

These statements would adjust the expression from its original timezone into a 
known target timezone.

Using  the time zone specifier results in a data type that has a time zone. If 
the original expression lacked a time zone, the result has a time zone. If the 
original expression had a time zone, the result still has a time zone, possibly 
a different one.

LOCAL means to use the session's original default time zone displacement.

The standard says that dates are not supported with time zone specifiers. It 
seems common to ignore this rule and allow this, by converting the date to a 
timestamp and then applying the usual rule.

The standard only requires an interval or the LOCAL keyword. Some databases 
allow time zone identifiers like PST.

Reference: SQL:2011 section 6.31

  was:
HIVE-14412 introduces a timestamp-aware timezone.

SQL has a concept of "time zone specifier" which applies to any datetime value 
expression (which covers time/timestamp with and without timezones). Hive lacks 
a time type so we can put that aside for a while.

Examples:
a. select time_stamp_with_time_zone at time zone '-8:00';
b. select time_stamp_without_time_zone at time zone LOCAL;

These statements would adjust the expression from its original timezone into a 
known target timezone.

Using  the time zone specifier results in a data type that has a time zone. If 
the original expression lacked a time zone, the result has a time zone. If the 
original expression had a time zone, the result still has a time zone, possibly 
a different one.

LOCAL means to use the session's original default time zone displacement.

The standard says that dates are not supported with time zone specifiers. It 
seems common to ignore this rule and allow this, by converting the date to a 
timestamp and then applying the usual rule.

The standard only requires an interval or the LOCAL keyword. Some databases 
allow time zone identifiers like PST.

Reference: SQL:2011 section 6.31


> Support Time Zone Specifiers (i.e. "at time zone X")
> 
>
> Key: HIVE-16615
> URL: https://issues.apache.org/jira/browse/HIVE-16615
> Project: Hive
>  Issue Type: Improvement
>Reporter: Carter Shanklin
>
> HIVE-14412 introduces a timezone-aware timestamp.
> SQL has a concept of "time zone specifier" which applies to any datetime 
> value expression (which covers time/timestamp with and without timezones). 
> Hive lacks a time type so we can put that aside for a while.
> Examples:
>   a. select time_stamp_with_time_zone at time zone '-8:00';
>   b. select time_stamp_without_time_zone at time zone LOCAL;
> These statements would adjust the expression from its original timezone into 
> a known target timezone.
> Using  the time zone specifier results in a data type that has a time zone. 
> If the original expression lacked a time zone, the result has a time zone. If 
> the original expression had a time zone, the result still has a time zone, 
> possibly a different one.
> LOCAL means to use the session's original default time zone displacement.
> The standard says that dates are not supported with time zone specifiers. It 
> seems common to ignore this rule and allow this, by converting the date to a 
> timestamp and then applying the usual rule.
> The standard only requires an interval or the LOCAL keyword. Some databases 
> allow time zone identifiers like PST.
> Reference: SQL:2011 section 6.31



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases

[Remus Rusanu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002677#comment-16002677
 ] 

Remus Rusanu commented on HIVE-16113:
-

I think the issue is that the partition pruning logic expects {{null}} values 
to percolate through the expression tree and trigger the {{isUnknown}} case in 
{{PartitionPrunner.prunePartitionNames}}. But expressions like CASE, COALESCE 
or NVL (maybe other?) stop null bubbling and can evaluate the tree to a 
resolute {{false}} instead, causing overaggressive partition elimination. If 
the expression has these functions (NVL, COALESCE) to handle {{null}} values at 
a *row* level we're evaluating them and taking a decision at *partition* level 
(prune/not prune), and the 'prune' case is not safe. Perhaps we should consider 
functions like NVL/COALESCE 'special' and inject 'true' instead in the pruning 
expression? 

> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153  WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main] 
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}} 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-11064) ALTER TABLE CASCADE ERROR unbalanced calls to openTransaction/commitTransaction

[Chaoyu Tang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002678#comment-16002678
 ] 

Chaoyu Tang commented on HIVE-11064:


The issue was caused by the discrepancy in column definition between a table 
and its partition. The first command "ALTER TABLE test1 CHANGE name name1 
string;" changed the table's column "name" to "name1", the 2nd command "ALTER 
TABLE test1 CHANGE name1 name string cascade;" with "cascade" clause attempted 
to change the partition column "name1" to "name" which did actually not exist. 
When executing the 2nd command, Hive failed in validateTableCols (validating 
partition columns against its table) in getMPartitionColumnStatistics. It is 
the root cause to the seen issue in this JIRA though the thrown exception and 
its message is not so informative.
The issue has been fixed as a side effect of HIVE-16147. 

> ALTER TABLE CASCADE ERROR unbalanced calls to 
> openTransaction/commitTransaction
> ---
>
> Key: HIVE-11064
> URL: https://issues.apache.org/jira/browse/HIVE-11064
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.1.0
> Environment: CDH5.4.0
>Reporter: fatkun
>Assignee: Chaoyu Tang
>
> my hive version  hive-1.1.0-cdh5.4.0
> follower this step, the exception throw
>  
> use hive client
> {code}
> CREATE TABLE test1 (name string) PARTITIONED BY (pt string);
> ALTER TABLE test1 ADD PARTITION (pt='1');
> ALTER TABLE test1 CHANGE name name1 string;
> ALTER TABLE test1 CHANGE name1 name string cascade;
> {code}
> then throw exception,
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. 
> java.lang.RuntimeException: commitTransaction was called but 
> openTransactionCalls = 0. This probably indicates that there are unbalanced 
> calls to openTransaction/commitTransaction
>  
> metasotre log
> {quote}
> MetaException(message:java.lang.RuntimeException: commitTransaction was 
> called but openTransactionCalls = 0. This probably indicates that there are 
> unbalanced calls to openTransaction/commitTransaction)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5257)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3338)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_cascade(HiveMetaStore.java:3290)
>   at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
>   at com.sun.proxy.$Proxy5.alter_table_with_cascade(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_cascade.getResult(ThriftHiveMetastore.java:9131)
>   at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_cascade.getResult(ThriftHiveMetastore.java:9115)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at 
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
>   at 
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>   at 
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: commitTransaction was called but 
> openTransactionCalls = 0. This probably indicates that there are unbalanced 
> calls to openTransaction/commitTransaction
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:448)
>   at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
>   at com.sun

[jira] [Reopened] (HIVE-11064) ALTER TABLE CASCADE ERROR unbalanced calls to openTransaction/commitTransaction

[Chaoyu Tang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaoyu Tang reopened HIVE-11064:


> ALTER TABLE CASCADE ERROR unbalanced calls to 
> openTransaction/commitTransaction
> ---
>
> Key: HIVE-11064
> URL: https://issues.apache.org/jira/browse/HIVE-11064
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.1.0
> Environment: CDH5.4.0
>Reporter: fatkun
>Assignee: Chaoyu Tang
>
> my hive version  hive-1.1.0-cdh5.4.0
> follower this step, the exception throw
>  
> use hive client
> {code}
> CREATE TABLE test1 (name string) PARTITIONED BY (pt string);
> ALTER TABLE test1 ADD PARTITION (pt='1');
> ALTER TABLE test1 CHANGE name name1 string;
> ALTER TABLE test1 CHANGE name1 name string cascade;
> {code}
> then throw exception,
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. 
> java.lang.RuntimeException: commitTransaction was called but 
> openTransactionCalls = 0. This probably indicates that there are unbalanced 
> calls to openTransaction/commitTransaction
>  
> metasotre log
> {quote}
> MetaException(message:java.lang.RuntimeException: commitTransaction was 
> called but openTransactionCalls = 0. This probably indicates that there are 
> unbalanced calls to openTransaction/commitTransaction)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5257)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3338)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_cascade(HiveMetaStore.java:3290)
>   at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
>   at com.sun.proxy.$Proxy5.alter_table_with_cascade(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_cascade.getResult(ThriftHiveMetastore.java:9131)
>   at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_cascade.getResult(ThriftHiveMetastore.java:9115)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at 
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
>   at 
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>   at 
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: commitTransaction was called but 
> openTransactionCalls = 0. This probably indicates that there are unbalanced 
> calls to openTransaction/commitTransaction
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:448)
>   at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
>   at com.sun.proxy.$Proxy0.commitTransaction(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:242)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3318)
>   ... 19 more
> {quote}
> I debug the code, may this function "private void 
> updatePartColumnStatsForAlterColumns" wrong.some transaction rollback, but I 
> don't known the exact error.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HIVE-11064) ALTER TABLE CASCADE ERROR unbalanced calls to openTransaction/commitTransaction

[Chaoyu Tang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaoyu Tang resolved HIVE-11064.

   Resolution: Fixed
Fix Version/s: 2.4.0
   3.0.0

It has been fixed in HIVE-16147.

> ALTER TABLE CASCADE ERROR unbalanced calls to 
> openTransaction/commitTransaction
> ---
>
> Key: HIVE-11064
> URL: https://issues.apache.org/jira/browse/HIVE-11064
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.1.0
> Environment: CDH5.4.0
>Reporter: fatkun
>Assignee: Chaoyu Tang
> Fix For: 3.0.0, 2.4.0
>
>
> my hive version  hive-1.1.0-cdh5.4.0
> follower this step, the exception throw
>  
> use hive client
> {code}
> CREATE TABLE test1 (name string) PARTITIONED BY (pt string);
> ALTER TABLE test1 ADD PARTITION (pt='1');
> ALTER TABLE test1 CHANGE name name1 string;
> ALTER TABLE test1 CHANGE name1 name string cascade;
> {code}
> then throw exception,
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. 
> java.lang.RuntimeException: commitTransaction was called but 
> openTransactionCalls = 0. This probably indicates that there are unbalanced 
> calls to openTransaction/commitTransaction
>  
> metasotre log
> {quote}
> MetaException(message:java.lang.RuntimeException: commitTransaction was 
> called but openTransactionCalls = 0. This probably indicates that there are 
> unbalanced calls to openTransaction/commitTransaction)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5257)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3338)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_cascade(HiveMetaStore.java:3290)
>   at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
>   at com.sun.proxy.$Proxy5.alter_table_with_cascade(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_cascade.getResult(ThriftHiveMetastore.java:9131)
>   at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_cascade.getResult(ThriftHiveMetastore.java:9115)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at 
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
>   at 
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>   at 
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: commitTransaction was called but 
> openTransactionCalls = 0. This probably indicates that there are unbalanced 
> calls to openTransaction/commitTransaction
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:448)
>   at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
>   at com.sun.proxy.$Proxy0.commitTransaction(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:242)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3318)
>   ... 19 more
> {quote}
> I debug the code, may this function "private void 
> updatePartColumnStatsForAlterColumns" wrong.some transaction rollback, but I 
> don't known the exact error.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16610) Semijoin Hint : Should be able to handle more than one hint per alias

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002691#comment-16002691
 ] 

Hive QA commented on HIVE-16610:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867061/HIVE-16610.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10657 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample_islocalmode_hook] 
(batchId=12)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_semijoin_user_level]
 (batchId=140)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5135/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5135/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5135/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867061 - PreCommit-HIVE-Build

> Semijoin Hint : Should be able to handle more than one hint per alias
> -
>
> Key: HIVE-16610
> URL: https://issues.apache.org/jira/browse/HIVE-16610
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-16610.1.patch
>
>
> Currently the semi join hints can be used to create only one semi join 
> optimization per alias which is very limiting.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer

[Yongzhi Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongzhi Chen updated HIVE-16451:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Committed the fix into master. Thanks [~pvary] for your contribution. 

> Race condition between HiveStatement.getQueryLog and 
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, JDBC
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 3.0.0
>
> Attachments: HIVE-16451.02.patch, HIVE-16451.03.patch, 
> HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
>   private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
>   }
> {code}
> {code:title=HiveStatement.getQueryLog}
>   public List getQueryLog(boolean incremental, int fetchSize)
>   throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
>   if (stmtHandle != null) {
> [..]
>   } else {
> if (isQueryClosed) {
>   throw new ClosedOrCancelledStatementException("Method getQueryLog() 
> failed. The " +
>   "statement has been closed or cancelled.");
> } else {
>   return logs;
> }
>   }
> } catch (SQLException e) {
> [..]
> }
> [..]
>   }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to 
> true:
> {code:title=HiveStatement.closeClientOperation}
>   void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
>   }
> {code}
> The {{initFlags}} sets it to false:
> {code}
>   private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
>   }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but 
> before the {{initFlags}}, then we will have a following warning if verbose 
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method 
> getQueryLog() failed. The statement has been closed or cancelled. 
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing 
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method 
> > getQueryLog() failed. The statement has been closed or cancelled. 
> > (state=,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16607) ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002795#comment-16002795
 ] 

Hive QA commented on HIVE-16607:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867067/HIVE-16607.02.patch

{color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10658 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5136/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5136/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5136/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867067 - PreCommit-HIVE-Build

> ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID
> -
>
> Key: HIVE-16607
> URL: https://issues.apache.org/jira/browse/HIVE-16607
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, Logging
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16607.02.patch, HIVE-16607.patch
>
>
> Creating a new {{QueryState}} object regenerates the HIVEQUERYID stored in 
> the {{HiveConf}}.
> In HiveServer logs it makes hard to follow the life of the query since a new 
> queryid is assigned to the query during the execution.
> Since BeeLine is showing the operation logs based on the queryid, only the 
> first several line of the logs is showed in BeeLine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002794#comment-16002794
 ] 

Xuefu Zhang commented on HIVE-14412:


+1

> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.10.patch, HIVE-14412.11.patch, 
> HIVE-14412.12.patch, HIVE-14412.1.patch, HIVE-14412.2.patch, 
> HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, 
> HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch, HIVE-14412.9.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by 
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or 
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16590) Make initializing dag names in SparkWork thread safe for parallel compilation (HIVE-13512)

[Yongzhi Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongzhi Chen updated HIVE-16590:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Push the fix into master. Thanks [~stakiar] for your contribution.

> Make initializing dag names in SparkWork thread safe for parallel compilation 
> (HIVE-13512)
> --
>
> Key: HIVE-16590
> URL: https://issues.apache.org/jira/browse/HIVE-16590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-16590.1.patch, HIVE-16590.2.patch
>
>
> In HIVE-13512 some modifications to {{TezWork}} were made to avoid generating 
> duplicate dag ids during parallel compilation. We should do the equivalent 
> for {{SparkWork}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16616) Javadoc generation is full of errors

[Janos Gub (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janos Gub reassigned HIVE-16616:



> Javadoc generation is full of errors
> 
>
> Key: HIVE-16616
> URL: https://issues.apache.org/jira/browse/HIVE-16616
> Project: Hive
>  Issue Type: Bug
>Reporter: Janos Gub
>Assignee: Janos Gub
>
> Trying to generate documentation from javadoc, full of errors. This ticket is 
> to track the process of cleaning the project of those.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16617) Clean up javadoc from errors in module hive-shims

[Janos Gub (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janos Gub reassigned HIVE-16617:



> Clean up javadoc from errors in module hive-shims
> -
>
> Key: HIVE-16617
> URL: https://issues.apache.org/jira/browse/HIVE-16617
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Janos Gub
>Assignee: Janos Gub
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16607) ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002813#comment-16002813
 ] 

Peter Vary commented on HIVE-16607:
---

[~aihuaxu]: Could you please review?

[~pxiong]: I have modified the patch you submitted by HIVE-13566 (Auto-gather 
column stats). You might want to review the changes.

Thanks,
Peter

> ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID
> -
>
> Key: HIVE-16607
> URL: https://issues.apache.org/jira/browse/HIVE-16607
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, Logging
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16607.02.patch, HIVE-16607.patch
>
>
> Creating a new {{QueryState}} object regenerates the HIVEQUERYID stored in 
> the {{HiveConf}}.
> In HiveServer logs it makes hard to follow the life of the query since a new 
> queryid is assigned to the query during the execution.
> Since BeeLine is showing the operation logs based on the queryid, only the 
> first several line of the logs is showed in BeeLine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16618) Clean up javadoc from errors in module hive-common

[Janos Gub (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janos Gub reassigned HIVE-16618:



> Clean up javadoc from errors in module hive-common
> --
>
> Key: HIVE-16618
> URL: https://issues.apache.org/jira/browse/HIVE-16618
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Janos Gub
>Assignee: Janos Gub
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16617) Clean up javadoc from errors in module hive-shims

[Janos Gub (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janos Gub updated HIVE-16617:
-
Attachment: HIVE-16617.patch

> Clean up javadoc from errors in module hive-shims
> -
>
> Key: HIVE-16617
> URL: https://issues.apache.org/jira/browse/HIVE-16617
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HIVE-16617.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16618) Clean up javadoc from errors in module hive-common

[Janos Gub (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janos Gub updated HIVE-16618:
-
Attachment: HIVE-16618.patch

> Clean up javadoc from errors in module hive-common
> --
>
> Key: HIVE-16618
> URL: https://issues.apache.org/jira/browse/HIVE-16618
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HIVE-16618.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-10865) Beeline needs to support DELIMITER command

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002822#comment-16002822
 ] 

Sergio Peña commented on HIVE-10865:


Patchs looks good [~stakiar]
+1

> Beeline needs to support DELIMITER command
> --
>
> Key: HIVE-10865
> URL: https://issues.apache.org/jira/browse/HIVE-10865
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Sahil Takiar
> Attachments: HIVE-10865.1.patch, HIVE-10865.2.patch, 
> HIVE-10865.3.patch, HIVE-10865.4.patch, HIVE-10865.5.patch, HIVE-10865.6.patch
>
>
> MySQL Client provides a DELIMITER command to set statement delimiter.
> Beeline needs to support a similar command to allow commands having 
> semi-colon as non-statement delimiter (as with MySQL stored procedures). This 
> is a follow-up jira for HIVE-10659



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16607) ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002823#comment-16002823
 ] 

Aihua Xu commented on HIVE-16607:
-

[~pvary] Can you create a code review for the patch? Some basic questions: I 
thought we assign queryId before the query starts the compilation. Can you 
point out where we reset the new queryId? The Builder pattern seems to be a 
good idea. Can we just call it Builder (so it will be QueryState.Builder. Looks 
like typically it's named like this?)?

> ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID
> -
>
> Key: HIVE-16607
> URL: https://issues.apache.org/jira/browse/HIVE-16607
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, Logging
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16607.02.patch, HIVE-16607.patch
>
>
> Creating a new {{QueryState}} object regenerates the HIVEQUERYID stored in 
> the {{HiveConf}}.
> In HiveServer logs it makes hard to follow the life of the query since a new 
> queryid is assigned to the query during the execution.
> Since BeeLine is showing the operation logs based on the queryid, only the 
> first several line of the logs is showed in BeeLine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases

[Remus Rusanu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002855#comment-16002855
 ] 

Remus Rusanu commented on HIVE-16113:
-

As for the {{OR}} case, I think that is being removed because 
{{PcrExprProcFactory.GenericFuncExprProcessor.handleUdfOr}} shortcuts the case 
{{WalkState.TRUE}} over the UNKNOWN of the {{col<5}} case. This determinism 
later makes {{PartitionConditionRemover}} eliminate the Filter operator 
completely.

> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153  WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main] 
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}} 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16469) Parquet timestamp table property is not always taken into account

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-16469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-16469:
---
   Resolution: Fixed
Fix Version/s: 2.4.0
   3.0.0
   Status: Resolved  (was: Patch Available)

Thanks [~zsombor.klara] for the patch. I committed to master and branch-2

> Parquet timestamp table property is not always taken into account
> -
>
> Key: HIVE-16469
> URL: https://issues.apache.org/jira/browse/HIVE-16469
> Project: Hive
>  Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-16469.01.patch, HIVE-16469.02.patch, 
> HIVE-16469.03.patch, HIVE-16469.04.patch, HIVE-16469.05.patch
>
>
> The parquet timestamp timezone property is currently copied over into the 
> JobConf in the FetchOperator, but this may be too late for some execution 
> paths.
> We should:
> 1 - copy the property over earlier
> 2 - set the default value on the JobConf if no property is set, and fail in 
> the ParquetRecordReader if the property is missing from the JobConf
> We should add extra validations for the cases when:
> - the property was not set by accident on the JobConf (unexpected execution 
> path)
> - an incorrect/invalid timezone id is being set on the table



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16619) Clean up javadoc from errors in module hive-serde

[Janos Gub (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janos Gub updated HIVE-16619:
-
Attachment: HIVE-16619.patch

> Clean up javadoc from errors in module hive-serde
> -
>
> Key: HIVE-16619
> URL: https://issues.apache.org/jira/browse/HIVE-16619
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HIVE-16619.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002874#comment-16002874
 ] 

Peter Vary commented on HIVE-16451:
---

Thanks [~ychena], [~anishek] and [~vihangk1] for the review and the commit!

> Race condition between HiveStatement.getQueryLog and 
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, JDBC
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 3.0.0
>
> Attachments: HIVE-16451.02.patch, HIVE-16451.03.patch, 
> HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
>   private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
>   }
> {code}
> {code:title=HiveStatement.getQueryLog}
>   public List getQueryLog(boolean incremental, int fetchSize)
>   throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
>   if (stmtHandle != null) {
> [..]
>   } else {
> if (isQueryClosed) {
>   throw new ClosedOrCancelledStatementException("Method getQueryLog() 
> failed. The " +
>   "statement has been closed or cancelled.");
> } else {
>   return logs;
> }
>   }
> } catch (SQLException e) {
> [..]
> }
> [..]
>   }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to 
> true:
> {code:title=HiveStatement.closeClientOperation}
>   void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
>   }
> {code}
> The {{initFlags}} sets it to false:
> {code}
>   private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
>   }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but 
> before the {{initFlags}}, then we will have a following warning if verbose 
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method 
> getQueryLog() failed. The statement has been closed or cancelled. 
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing 
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method 
> > getQueryLog() failed. The statement has been closed or cancelled. 
> > (state=,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16619) Clean up javadoc from errors in module hive-serde

[Janos Gub (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janos Gub reassigned HIVE-16619:



> Clean up javadoc from errors in module hive-serde
> -
>
> Key: HIVE-16619
> URL: https://issues.apache.org/jira/browse/HIVE-16619
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Janos Gub
>Assignee: Janos Gub
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-10865) Beeline needs to support DELIMITER command

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-10865:
---
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Thanks [~stakiar]. I committed this to master.

> Beeline needs to support DELIMITER command
> --
>
> Key: HIVE-10865
> URL: https://issues.apache.org/jira/browse/HIVE-10865
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-10865.1.patch, HIVE-10865.2.patch, 
> HIVE-10865.3.patch, HIVE-10865.4.patch, HIVE-10865.5.patch, HIVE-10865.6.patch
>
>
> MySQL Client provides a DELIMITER command to set statement delimiter.
> Beeline needs to support a similar command to allow commands having 
> semi-colon as non-statement delimiter (as with MySQL stored procedures). This 
> is a follow-up jira for HIVE-10659



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16572) Rename a partition should not drop its column stats

[Chaoyu Tang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaoyu Tang updated HIVE-16572:
---
   Resolution: Fixed
Fix Version/s: 2.4.0
   3.0.0
   Status: Resolved  (was: Patch Available)

Committed to 3.0.0and 2.4.0. Thanks [~ychena] for review.

> Rename a partition should not drop its column stats
> ---
>
> Key: HIVE-16572
> URL: https://issues.apache.org/jira/browse/HIVE-16572
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-16572.1.patch, HIVE-16572.patch
>
>
> The column stats for the table sample_pt partition (dummy=1) is as following:
> {code}
> hive> describe formatted sample_pt partition (dummy=1) code;
> OK
> # col_namedata_type   min 
> max num_nulls   distinct_count  
> avg_col_len max_col_len num_trues   
> num_falses  comment 
>   
>  
> code  string  
> 0   303 6.985 
>   7   
> from deserializer   
> Time taken: 0.259 seconds, Fetched: 3 row(s)
> {code}
> But when this partition is renamed, say
> alter table sample_pt partition (dummy=1) rename to partition (dummy=11);
> The COLUMN_STATS in partition description are true, but column stats are 
> actually all deleted.
> {code}
> hive> describe formatted sample_pt partition (dummy=11);
> OK
> # col_namedata_type   comment 
>
> code  string  
> description   string  
> salaryint 
> total_emp int 
>
> # Partition Information
> # col_namedata_type   comment 
>
> dummy int 
>
> # Detailed Partition Information   
> Partition Value:  [11] 
> Database: default  
> Table:sample_pt
> CreateTime:   Thu Mar 30 23:03:59 EDT 2017 
> LastAccessTime:   UNKNOWN  
> Location: file:/user/hive/warehouse/apache/sample_pt/dummy=11 
>  
> Partition Parameters:  
>   COLUMN_STATS_ACCURATE   
> {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"code\":\"true\",\"description\":\"true\",\"salary\":\"true\",\"total_emp\":\"true\"}}
>   numFiles1   
>   numRows 200 
>   rawDataSize 10228   
>   totalSize   10428   
>   transient_lastDdlTime   1490929439  
>
> # Storage Information  
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe  
>  
> InputFormat:  org.apache.hadoop.mapred.TextInputFormat 
> OutputFormat: 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat   
> Compressed:   No   
> Num Buckets:  -1   
> Bucket Columns:   []   
> Sort Columns: []   
> Storage Desc Params:   
>   serialization.format1   
> Time taken: 6.783 seconds, Fetched: 37 row(s)
> ===
> hive> describe formatted sample_pt partition (dummy=11) code;
> OK
> # col_namedata_type   comment 
>  
>   
>  
> code  string  from deserializer   
>  
> Time taken: 9.429 seconds, Fetched: 3 row(s)
> {code}
> The column stats should not be drop when a partition is renamed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16613) SaslClientHandler.sendHello is eating exceptions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002915#comment-16002915
 ] 

Hive QA commented on HIVE-16613:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867076/HIVE-16613.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 10658 tests passed

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5137/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5137/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5137/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867076 - PreCommit-HIVE-Build

> SaslClientHandler.sendHello is eating exceptions
> 
>
> Key: HIVE-16613
> URL: https://issues.apache.org/jira/browse/HIVE-16613
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-16613.1.patch
>
>
> If we hit exceptions when sending hello to server, the method fails silently. 
> Finally the remote driver fails with SASL timeout instead of the real cause, 
> making it difficult for debugging.
> To reproduce: throw some exception in {{KryoMessageCodec.encode}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16620) show create table on partitioned view does not show partitioned on statement

[Anbu Cheeralan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anbu Cheeralan updated HIVE-16620:
--
Description: 
show create on partitioned view does not show the partitioned on clause.

Here is an example.
{code:borderStyle=solid}
use default
create table t1 (a int) partitioned by (b int)
insert into t1 partition(b) values (5, 1)
select * from t1
create or replace view v1 partitioned on (b) as select * from t1
explain dependency select * from v1 where b=1
show create table v1
{code}
This results in 

{code:borderStyle=solid}
CREATE VIEW `v1` AS select `t1`.`a`, `t1`.`b` from `default`.`t1`
{code}
I am expecting this to show 
{code:borderStyle=solid}
create view v1 partitioned on (b) as select  `t1`.`a`, `t1`.`b` from 
`default`.`t1`
{code}


  was:
show create on partitioned view does not show the partitioned on clause.

Here is an example.
```SQL
use default
create table t1 (a int) partitioned by (b int)
insert into t1 partition(b) values (5, 1)
select * from t1
create or replace view v1 partitioned on (b) as select * from t1
explain dependency select * from v1 where b=1
show create table v1
```
This results in 
```
CREATE VIEW `v1` AS select `t1`.`a`, `t1`.`b` from `default`.`t1`
```
I am expecting this to show 
create view v1 partitioned on (b) as select  `t1`.`a`, `t1`.`b` from 
`default`.`t1`
```



> show create table on partitioned view does not show partitioned on statement
> 
>
> Key: HIVE-16620
> URL: https://issues.apache.org/jira/browse/HIVE-16620
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Anbu Cheeralan
>Priority: Minor
>
> show create on partitioned view does not show the partitioned on clause.
> Here is an example.
> {code:borderStyle=solid}
> use default
> create table t1 (a int) partitioned by (b int)
> insert into t1 partition(b) values (5, 1)
> select * from t1
> create or replace view v1 partitioned on (b) as select * from t1
> explain dependency select * from v1 where b=1
> show create table v1
> {code}
> This results in 
> {code:borderStyle=solid}
> CREATE VIEW `v1` AS select `t1`.`a`, `t1`.`b` from `default`.`t1`
> {code}
> I am expecting this to show 
> {code:borderStyle=solid}
> create view v1 partitioned on (b) as select  `t1`.`a`, `t1`.`b` from 
> `default`.`t1`
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16620) show create on partitioned view does not show "partitioned on" statement

[Anbu Cheeralan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anbu Cheeralan updated HIVE-16620:
--
Summary: show create on partitioned view does not show "partitioned on" 
statement  (was: show create table on partitioned view does not show 
partitioned on statement)

> show create on partitioned view does not show "partitioned on" statement
> 
>
> Key: HIVE-16620
> URL: https://issues.apache.org/jira/browse/HIVE-16620
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Anbu Cheeralan
>Priority: Minor
>
> show create on partitioned view does not show the partitioned on clause.
> Here is an example.
> {code:borderStyle=solid}
> use default
> create table t1 (a int) partitioned by (b int)
> insert into t1 partition(b) values (5, 1)
> select * from t1
> create or replace view v1 partitioned on (b) as select * from t1
> explain dependency select * from v1 where b=1
> show create table v1
> {code}
> This results in 
> {code:borderStyle=solid}
> CREATE VIEW `v1` AS select `t1`.`a`, `t1`.`b` from `default`.`t1`
> {code}
> I am expecting this to show 
> {code:borderStyle=solid}
> create view v1 partitioned on (b) as select  `t1`.`a`, `t1`.`b` from 
> `default`.`t1`
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002950#comment-16002950
 ] 

Ashutosh Chauhan commented on HIVE-14412:
-

Is there any reason to use timestamptz alias for type timestamp with time zone? 
timestamptz is non-standard. My suggestion is to not use reserve the word in 
grammar if not its not needed.

> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.10.patch, HIVE-14412.11.patch, 
> HIVE-14412.12.patch, HIVE-14412.1.patch, HIVE-14412.2.patch, 
> HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, 
> HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch, HIVE-14412.9.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by 
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or 
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16625) Extra '\0' characters in the output, when SeparatedValuesOutputFormat is used and the quoting is disabled

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary reassigned HIVE-16625:
-


> Extra '\0' characters in the output, when SeparatedValuesOutputFormat is used 
> and the quoting is disabled
> -
>
> Key: HIVE-16625
> URL: https://issues.apache.org/jira/browse/HIVE-16625
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, Testing Infrastructure
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> If the output format is using {{SeparatedValuesOutputFormat}}, and the 
> quoting is disabled (by default is disabled), and the value of the cell 
> contains the separator character, then the the output is "quoted" with '\0' 
> characters.
> To reproduce:
> {code}
> create table quotes(s string);
> insert into quotes values('a\ta');
> !set outputFormat tsv2
> select * from quotes;
> {code}
> The result is:
> {code}
> quotes.s
> ^@a   a^@
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16595) fix syntax in Hplsql.g4

[Yishuang Lu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002996#comment-16002996
 ] 

Yishuang Lu commented on HIVE-16595:


Same change caused another test failed. I also found there was a recent fix for 
BeeLineDriver testing(HIVE-16451).

> fix syntax in Hplsql.g4
> ---
>
> Key: HIVE-16595
> URL: https://issues.apache.org/jira/browse/HIVE-16595
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Reporter: Yishuang Lu
>Assignee: Yishuang Lu
> Fix For: 1.2.3
>
> Attachments: HIVE-16595.1.patch, HIVE-16595.2.patch, 
> HIVE-16595.3.patch
>
>
> According to https://github.com/antlr/antlr4/issues/118, incorrect error 
> message might return if the start rule does not contain an explicit EOF 
> transition. It is better to add EOF for the first rule in grammar.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16595) fix syntax in Hplsql.g4

[Yishuang Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yishuang Lu updated HIVE-16595:
---
Status: Patch Available  (was: Open)

> fix syntax in Hplsql.g4
> ---
>
> Key: HIVE-16595
> URL: https://issues.apache.org/jira/browse/HIVE-16595
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Reporter: Yishuang Lu
>Assignee: Yishuang Lu
> Fix For: 1.2.3
>
> Attachments: HIVE-16595.1.patch, HIVE-16595.2.patch, 
> HIVE-16595.3.patch
>
>
> According to https://github.com/antlr/antlr4/issues/118, incorrect error 
> message might return if the start rule does not contain an explicit EOF 
> transition. It is better to add EOF for the first rule in grammar.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16595) fix syntax in Hplsql.g4

[Yishuang Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yishuang Lu updated HIVE-16595:
---
Status: Open  (was: Patch Available)

> fix syntax in Hplsql.g4
> ---
>
> Key: HIVE-16595
> URL: https://issues.apache.org/jira/browse/HIVE-16595
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Reporter: Yishuang Lu
>Assignee: Yishuang Lu
> Fix For: 1.2.3
>
> Attachments: HIVE-16595.1.patch, HIVE-16595.2.patch, 
> HIVE-16595.3.patch
>
>
> According to https://github.com/antlr/antlr4/issues/118, incorrect error 
> message might return if the start rule does not contain an explicit EOF 
> transition. It is better to add EOF for the first rule in grammar.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-16456:
---
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Patch committed to master. Thanks, Zhihai!

> Kill spark job when InterruptedException happens or driverContext.isShutdown 
> is true.
> -
>
> Key: HIVE-16456
> URL: https://issues.apache.org/jira/browse/HIVE-16456
> Project: Hive
>  Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch
>
>
> Kill spark job when InterruptedException happens or driverContext.isShutdown 
> is true. If the InterruptedException happened in RemoteSparkJobMonitor and 
> LocalSparkJobMonitor, it will be better to kill the job. Also there is a race 
> condition between submit the spark job and query/operation cancellation, it 
> will be better to check driverContext.isShutdown right after submit the spark 
> job. This will guarantee the job being killed no matter when shutdown is 
> called. It is similar as HIVE-15997.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16625) Extra '\0' characters in the output, when SeparatedValuesOutputFormat is used and the quoting is disabled

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-16625:
--
Attachment: HIVE-16625.patch

When unquotedCsvPreference is generated use SelectiveCsvEncoder without any 
column to encode.
Add a qtest to test the case


> Extra '\0' characters in the output, when SeparatedValuesOutputFormat is used 
> and the quoting is disabled
> -
>
> Key: HIVE-16625
> URL: https://issues.apache.org/jira/browse/HIVE-16625
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, Testing Infrastructure
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16625.patch
>
>
> If the output format is using {{SeparatedValuesOutputFormat}}, and the 
> quoting is disabled (by default is disabled), and the value of the cell 
> contains the separator character, then the the output is "quoted" with '\0' 
> characters.
> To reproduce:
> {code}
> create table quotes(s string);
> insert into quotes values('a\ta');
> !set outputFormat tsv2
> select * from quotes;
> {code}
> The result is:
> {code}
> quotes.s
> ^@a   a^@
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16602) Implement shared scans with Tez

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003051#comment-16003051
 ] 

Hive QA commented on HIVE-16602:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867092/HIVE-16602.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10576 tests 
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=148)

[list_bucket_dml_10.q,mapjoin_emit_interval.q,acid_globallimit.q,delete_where_no_match.q,semijoin_hint.q,vector_groupby_grouping_sets4.q,tez_vector_dynpart_hashjoin_2.q,vectorization_part.q,delete_all_non_partitioned.q,delete_all_partitioned.q,auto_join30.q,dynamic_partition_pruning.q,cluster.q,bucketmapjoin7.q,cbo_rp_unionDistinct_2.q,subquery_nested_subquery.q,limit_pushdown3.q,vector_outer_join2.q,smb_mapjoin_18.q,vector_varchar_4.q,ppd_union_view.q,union6.q,vector_decimal_4.q,schema_evol_orc_acid_part.q,cbo_subq_in.q,cross_join.q,parquet_predicate_pushdown.q,llap_vector_nohybridgrace.q,tez_smb_main.q,vectorized_timestamp_funcs.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=149)

[vector_auto_smb_mapjoin_14.q,deleteAnalyze.q,multi_insert.q,tez_dml.q,database.q,vector_bround.q,orc_ppd_schema_evol_1b.q,vector_join30.q,smb_mapjoin_6.q,vector_groupby_grouping_sets2.q,cte_3.q,vector_reduce_groupby_decimal.q,vectorized_dynamic_partition_pruning.q,cbo_views.q,vector_ptf_part_simple.q,vector_decimal_cast.q,groupby_grouping_id2.q,tez_smb_empty.q,orc_merge6.q,cte_mat_1.q,vector_char_mapjoin1.q,cte_5.q,subquery_shared_alias.q,vector_decimal_2.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,vector_string_concat.q,vectorized_context.q,metadata_only_queries.q,auto_sortmerge_join_12.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=156)

[vector_interval_2.q,schema_evol_orc_acid_part_update.q,orc_ppd_varchar.q,metadataonly1.q,join_nullsafe.q,vectorized_shufflejoin.q,limit_pushdown.q,auto_join_nulls.q,metadata_only_queries_with_filters.q,vector_inner_join.q,subquery_notin.q,schema_evol_text_nonvec_part_all_complex.q,vector_coalesce_2.q,vector_between_columns.q,vector_groupby_grouping_sets6.q,table_access_keys_stats.q,udaf_collect_set_2.q,subquery_null_agg.q,update_after_multiple_inserts.q,offset_limit_ppd_optimizer.q,mapjoin_decimal.q,orc_merge_incompat1.q,columnstats_part_coltype.q,vectorized_parquet_types.q,vector_char_2.q,stats_based_fetch_decision.q,auto_sortmerge_join_10.q,extrapolate_part_stats_partial_ndv.q,auto_sortmerge_join_9.q,vector_groupby_grouping_id2.q]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_1]
 (batchId=147)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=98)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5138/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5138/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5138/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867092 - PreCommit-HIVE-Build

> Implement shared scans with Tez
> ---
>
> Key: HIVE-16602
> URL: https://issues.apache.org/jira/browse/HIVE-16602
> Project: Hive
>  Issue Type: New Feature
>  Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16602.01.patch, HIVE-16602.patch
>
>
> Given a query plan, the goal is to identify scans on input tables that can be 
> merged so the data is read only once. Optimization will be carried out at the 
> physical level.
> In the longer term, identification of equivalent expressions and 
> reutilization of intermediary results should be done at the logical layer via 
> Spool operator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Issue Comment Deleted] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases

[Remus Rusanu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remus Rusanu updated HIVE-16113:

Comment: was deleted

(was: As for the {{OR}} case, I think that is being removed because 
{{PcrExprProcFactory.GenericFuncExprProcessor.handleUdfOr}} shortcuts the case 
{{WalkState.TRUE}} over the UNKNOWN of the {{col<5}} case. This determinism 
later makes {{PartitionConditionRemover}} eliminate the Filter operator 
completely.)

> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153  WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main] 
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}} 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16556) Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES table

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003115#comment-16003115
 ] 

Vihang Karajgaonkar commented on HIVE-16556:


Thanks for the review [~ngangam]

> Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES 
> table
> 
>
> Key: HIVE-16556
> URL: https://issues.apache.org/jira/browse/HIVE-16556
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Fix For: 3.0.0
>
> Attachments: HIVE-16556.01.patch, HIVE-16556.02.patch, 
> HIVE-16556.03.patch, HIVE-16556.04.patch, HIVE-16556.05.patch
>
>
> sub-task to modify schema tool and its related changes so that the new table 
> is added to the schema when schematool initializes or upgrades the schema.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases

[Remus Rusanu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003117#comment-16003117
 ] 

Remus Rusanu commented on HIVE-16113:
-

{{OR}}: the partition pruner is reducing the expression to {{partcol = 1}} and 
this reduces the list of partitions retrieved from MD:
{code}
ppList = getPartitionsFromServer(tab, (ExprNodeGenericFuncDesc)compactExpr, 
conf, alias, partColsUsedInFilter, 
oldFilter.equals(compactExpr.getExprString()));
{code}
By now things are already broken, but later the reduced partition list causes 
the {{PcrExprProcFactory.GenericFuncExprProcessor.handleUdfOr}} to see a 
definite TRUE instead of a correct DIVIDE (since the filter expression is only 
evaluated against a reduced list of partitions) and thus cause the 
{{PartitionConditionRemover}} to remove the entire Filter. 

> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153  WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main] 
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}} 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-16609:
--
Attachment: HIVE-16609.2.patch

> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16609.1.patch, HIVE-16609.2.patch
>
>
> A variation of alter_partition_change_col.q produces wrong result:
> {code}
> SET hive.exec.dynamic.partition.mode = nonstrict;
> create table alter_partition_change_col0 (c1 string, c2 string);
> load data local inpath 'dec.txt' overwrite into table 
> alter_partition_change_col0;
> create table alter_partition_change_col1 (c1 string, c2 string) partitioned 
> by (p1 string comment 'Column p1', p2 string comment 'Column p2');
> insert overwrite table alter_partition_change_col1 partition (p1, p2)
>   select c1, c2, 'abc', '123' from alter_partition_change_col0
>   union all
>   select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
> select * from alter_partition_change_col1 where 
> p1='__HIVE_DEFAULT_PARTITION__' or lower(p1)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> We need another condition containing a udf so the condition is not recognized 
> by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like 
> HIVE-11208 breaks it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003190#comment-16003190
 ] 

Daniel Dai commented on HIVE-16609:
---

Fix UT failures.

> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16609.1.patch, HIVE-16609.2.patch
>
>
> A variation of alter_partition_change_col.q produces wrong result:
> {code}
> SET hive.exec.dynamic.partition.mode = nonstrict;
> create table alter_partition_change_col0 (c1 string, c2 string);
> load data local inpath 'dec.txt' overwrite into table 
> alter_partition_change_col0;
> create table alter_partition_change_col1 (c1 string, c2 string) partitioned 
> by (p1 string comment 'Column p1', p2 string comment 'Column p2');
> insert overwrite table alter_partition_change_col1 partition (p1, p2)
>   select c1, c2, 'abc', '123' from alter_partition_change_col0
>   union all
>   select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
> select * from alter_partition_change_col1 where 
> p1='__HIVE_DEFAULT_PARTITION__' or lower(p1)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> We need another condition containing a udf so the condition is not recognized 
> by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like 
> HIVE-11208 breaks it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases

[Remus Rusanu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003300#comment-16003300
 ] 

Remus Rusanu commented on HIVE-16113:
-

I don't think we should handle OR in removeNonPartCols, leaving it as NULL to 
bubble behaves correctly. I made that change and tested it, all good. 

But the NVL/COALESCE/CASE is trickier. I can write a filter like
{code}
where colpart=1 or col < 5
{code}
and works correct, but wrap it in an NVL:
{code}
where NVL(colpart=1 or col < 5, false)
{code}
and the result is no longer correct because the NULL is being replaced  with 
FALSE and this causes overaggressive partition pruning. My plan is to have 
these UDFs (NVL/COALESCE) replaced with NULL in the pruning expr tree.

> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153  WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main] 
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}} 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003319#comment-16003319
 ] 

Hive QA commented on HIVE-16609:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867146/HIVE-16609.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 10666 tests passed

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5139/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5139/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5139/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867146 - PreCommit-HIVE-Build

> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16609.1.patch, HIVE-16609.2.patch
>
>
> A variation of alter_partition_change_col.q produces wrong result:
> {code}
> SET hive.exec.dynamic.partition.mode = nonstrict;
> create table alter_partition_change_col0 (c1 string, c2 string);
> load data local inpath 'dec.txt' overwrite into table 
> alter_partition_change_col0;
> create table alter_partition_change_col1 (c1 string, c2 string) partitioned 
> by (p1 string comment 'Column p1', p2 string comment 'Column p2');
> insert overwrite table alter_partition_change_col1 partition (p1, p2)
>   select c1, c2, 'abc', '123' from alter_partition_change_col0
>   union all
>   select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
> select * from alter_partition_change_col1 where 
> p1='__HIVE_DEFAULT_PARTITION__' or lower(p1)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> We need another condition containing a udf so the condition is not recognized 
> by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like 
> HIVE-11208 breaks it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003324#comment-16003324
 ] 

Gopal V commented on HIVE-16113:


bq. My plan is to have these UDFs (NVL/COALESCE) replaced with NULL in the 
pruning expr tree.

Any UDF which gets a NULL arg because of a non-partition column can be replaced 
with NULL, right? 

That might be better instead of special-casing built-in UDFs?

> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153  WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main] 
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}} 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16584) Warning messages should use LogHelper.printInfo instead of printing to the infoStream directly

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003334#comment-16003334
 ] 

Thejas M Nair commented on HIVE-16584:
--

+1
Thanks for the great test coverage as well!


> Warning messages should use LogHelper.printInfo instead of printing to the 
> infoStream directly
> --
>
> Key: HIVE-16584
> URL: https://issues.apache.org/jira/browse/HIVE-16584
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16584.02.patch, HIVE-16584.03.patch, 
> HIVE-16584.patch
>
>
> There are several cases when warning messages are printed to the console 
> outputstream directly. These warnings do not show up in the BeeLine output.
> We should use the the printInfo method instead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16207) Add support for Complex Types in Fast SerDe

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi updated HIVE-16207:
--
Attachment: HIVE-16207.3.patch

Fixed the last bug in unit tests.

> Add support for Complex Types in Fast SerDe
> ---
>
> Key: HIVE-16207
> URL: https://issues.apache.org/jira/browse/HIVE-16207
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
> Attachments: HIVE-16207.1.patch, HIVE-16207.1.patch.zip, 
> HIVE-16207.2.patch, HIVE-16207.3.patch, partial.patch
>
>
> Add complex type support to Fast SerDe classes.  This is needed for fully 
> supporting complex types in Vectorization



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16626) Enable TestBeeLineWithArgs for MR

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar reassigned HIVE-16626:
--

Assignee: Vihang Karajgaonkar

> Enable TestBeeLineWithArgs for MR
> -
>
> Key: HIVE-16626
> URL: https://issues.apache.org/jira/browse/HIVE-16626
> Project: Hive
>  Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> HIVE-15900 changed the default execution engine of TestBeeLineWithArgs test 
> to Tez. We should include this test with MR as well to catch regressions for 
> HoMR.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16626) Enable TestBeeLineWithArgs for MR

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-16626:
---
Description: HIVE-15900 changed the default execution engine of 
TestBeeLineWithArgs test to Tez. We should include this test with MR as well to 
catch regressions for HoMR.  (was: HIVE-15900 changed the default execution of 
TestBeeLineWithArgs test to Tez. We should include this test with MR as well to 
catch regressions for HoMR.)

> Enable TestBeeLineWithArgs for MR
> -
>
> Key: HIVE-16626
> URL: https://issues.apache.org/jira/browse/HIVE-16626
> Project: Hive
>  Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>
> HIVE-15900 changed the default execution engine of TestBeeLineWithArgs test 
> to Tez. We should include this test with MR as well to catch regressions for 
> HoMR.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003404#comment-16003404
 ] 

Daniel Dai commented on HIVE-16609:
---

[~aihuaxu][~sershe], mind taking a look?

> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16609.1.patch, HIVE-16609.2.patch
>
>
> A variation of alter_partition_change_col.q produces wrong result:
> {code}
> SET hive.exec.dynamic.partition.mode = nonstrict;
> create table alter_partition_change_col0 (c1 string, c2 string);
> load data local inpath 'dec.txt' overwrite into table 
> alter_partition_change_col0;
> create table alter_partition_change_col1 (c1 string, c2 string) partitioned 
> by (p1 string comment 'Column p1', p2 string comment 'Column p2');
> insert overwrite table alter_partition_change_col1 partition (p1, p2)
>   select c1, c2, 'abc', '123' from alter_partition_change_col0
>   union all
>   select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
> select * from alter_partition_change_col1 where 
> p1='__HIVE_DEFAULT_PARTITION__' or lower(p1)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> We need another condition containing a udf so the condition is not recognized 
> by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like 
> HIVE-11208 breaks it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003439#comment-16003439
 ] 

Sergey Shelukhin commented on HIVE-16609:
-

-0.5 for now.
I don't think this is a valid query... as in, it should search for the literal 
string, as it does. The correct way to select "invalid" values is "is null" and 
that has been fixed recently if I remember correctly.
Users should not use the magic constant.

> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16609.1.patch, HIVE-16609.2.patch
>
>
> A variation of alter_partition_change_col.q produces wrong result:
> {code}
> SET hive.exec.dynamic.partition.mode = nonstrict;
> create table alter_partition_change_col0 (c1 string, c2 string);
> load data local inpath 'dec.txt' overwrite into table 
> alter_partition_change_col0;
> create table alter_partition_change_col1 (c1 string, c2 string) partitioned 
> by (p1 string comment 'Column p1', p2 string comment 'Column p2');
> insert overwrite table alter_partition_change_col1 partition (p1, p2)
>   select c1, c2, 'abc', '123' from alter_partition_change_col0
>   union all
>   select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
> select * from alter_partition_change_col1 where 
> p1='__HIVE_DEFAULT_PARTITION__' or lower(p1)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> We need another condition containing a udf so the condition is not recognized 
> by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like 
> HIVE-11208 breaks it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14389) Beeline should not output query and prompt to stdout

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-14389:
---
Attachment: HIVE-14389.02.patch

Fixed the test failures

> Beeline should not output query and prompt to stdout
> 
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-14389.01.patch, HIVE-14389.02.patch
>
>
> It seems that the Beeline prints the query along with the results in the 
> stdout when a script file is passed. The output file in the example below 
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u 
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2> 
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car  |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts 
> to Beeline, this needs to be taken care of. The output files generated by 
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout, 
> which is adding garbage along with results, as just like HIVE CLI does, users 
> would like to have only the results on stdout, not errors/debugging info/etc, 
> like the full query. 
> Query could be printed out, no problem, as long as it is not on stdout (with 
> results), instead, it must be printed out along with the debugging info.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14389) Beeline should not output query and prompt to stdout

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003448#comment-16003448
 ] 

Vihang Karajgaonkar commented on HIVE-14389:


[~pvary] [~thejas] Can you please review?

> Beeline should not output query and prompt to stdout
> 
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-14389.01.patch, HIVE-14389.02.patch
>
>
> It seems that the Beeline prints the query along with the results in the 
> stdout when a script file is passed. The output file in the example below 
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u 
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2> 
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car  |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts 
> to Beeline, this needs to be taken care of. The output files generated by 
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout, 
> which is adding garbage along with results, as just like HIVE CLI does, users 
> would like to have only the results on stdout, not errors/debugging info/etc, 
> like the full query. 
> Query could be printed out, no problem, as long as it is not on stdout (with 
> results), instead, it must be printed out along with the debugging info.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003489#comment-16003489
 ] 

Daniel Dai commented on HIVE-16609:
---

If so, we need to fix .q files containing the magic constant in the where 
statement (eg, alter_partition_change_col.q). How about drop partition 
statement? Shall we still use magic constant?

> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16609.1.patch, HIVE-16609.2.patch
>
>
> A variation of alter_partition_change_col.q produces wrong result:
> {code}
> SET hive.exec.dynamic.partition.mode = nonstrict;
> create table alter_partition_change_col0 (c1 string, c2 string);
> load data local inpath 'dec.txt' overwrite into table 
> alter_partition_change_col0;
> create table alter_partition_change_col1 (c1 string, c2 string) partitioned 
> by (p1 string comment 'Column p1', p2 string comment 'Column p2');
> insert overwrite table alter_partition_change_col1 partition (p1, p2)
>   select c1, c2, 'abc', '123' from alter_partition_change_col0
>   union all
>   select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
> select * from alter_partition_change_col1 where 
> p1='__HIVE_DEFAULT_PARTITION__' or lower(p1)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> We need another condition containing a udf so the condition is not recognized 
> by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like 
> HIVE-11208 breaks it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16612) PerfLogger is configurable, but not extensible

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003490#comment-16003490
 ] 

Ashutosh Chauhan commented on HIVE-16612:
-

Teasing out an interface from perflogger is a good idea. That will pave the way 
for collecting perf related metrics easier.

> PerfLogger is configurable, but not extensible
> --
>
> Key: HIVE-16612
> URL: https://issues.apache.org/jira/browse/HIVE-16612
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning, Query Processor
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
>
> {code}
>   result = (PerfLogger) 
> ReflectionUtils.newInstance(conf.getClassByName(
> conf.getVar(HiveConf.ConfVars.HIVE_PERF_LOGGER)), conf);
> {code}
> The PerfLogger instance is configurable via {{hive.exec.perf.logger}} 
> (HIVE-11891) but the requirement to extend from {{PerfLogger}} cannot be met 
> since HIVE-11149 as the ctor is private. Also useful methods in PerfLogger 
> are also private. I tried to extend PerfLogger for my needs and realized 
> that, as is, the configurability is not usable. At the very least the 
> PerfLogger should make all private members {{protected}}, better the 
> requirement should be an interface not a class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16593) SparkClientFactory.stop may prevent JVM from exiting

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003497#comment-16003497
 ] 

Xuefu Zhang commented on HIVE-16593:


+1

> SparkClientFactory.stop may prevent JVM from exiting
> 
>
> Key: HIVE-16593
> URL: https://issues.apache.org/jira/browse/HIVE-16593
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-16593.1.patch
>
>
> When we receive SIGINT more than once, we call System.exit to terminate the 
> JVM. System.exit runs the shutdown hooks which in turn calls 
> SparkClientFactory.stop. All the methods in SparkClientFactory are 
> synchronized and at this point, we may be waiting to create a SparkClientImpl 
> in SparkClientFactory.createClient. Therefore SparkClientFactory.stop will be 
> blocked until SparkClientFactory.createClient returns.
> The reason why we may be waiting to create SparkClientImpl is usually to wait 
> for RemoteDriver to connect. If RemoteDriver runs into problem, the JVM won't 
> exit until we timeout.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003510#comment-16003510
 ] 

Daniel Dai commented on HIVE-16609:
---

Also need to point out
{code}
select * from alter_partition_change_col1 where p1='__HIVE_DEFAULT_PARTITION__';
{code}
works fine, but
{code}
select * from alter_partition_change_col1 where p1='__HIVE_DEFAULT_PARTITION__' 
or lower(p1)='a';
{code}
is wrong. We need a fix to make them consistent.

> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-16609.1.patch, HIVE-16609.2.patch
>
>
> A variation of alter_partition_change_col.q produces wrong result:
> {code}
> SET hive.exec.dynamic.partition.mode = nonstrict;
> create table alter_partition_change_col0 (c1 string, c2 string);
> load data local inpath 'dec.txt' overwrite into table 
> alter_partition_change_col0;
> create table alter_partition_change_col1 (c1 string, c2 string) partitioned 
> by (p1 string comment 'Column p1', p2 string comment 'Column p2');
> insert overwrite table alter_partition_change_col1 partition (p1, p2)
>   select c1, c2, 'abc', '123' from alter_partition_change_col0
>   union all
>   select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
> select * from alter_partition_change_col1 where 
> p1='__HIVE_DEFAULT_PARTITION__' or lower(p1)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> We need another condition containing a udf so the condition is not recognized 
> by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like 
> HIVE-11208 breaks it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16207) Add support for Complex Types in Fast SerDe

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003523#comment-16003523
 ] 

Hive QA commented on HIVE-16207:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867167/HIVE-16207.3.patch

{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10672 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part_all_primitive]
 (batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_table]
 (batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_1]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_interval_arithmetic]
 (batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
 (batchId=152)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
 (batchId=119)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5140/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5140/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5140/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867167 - PreCommit-HIVE-Build

> Add support for Complex Types in Fast SerDe
> ---
>
> Key: HIVE-16207
> URL: https://issues.apache.org/jira/browse/HIVE-16207
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
> Attachments: HIVE-16207.1.patch, HIVE-16207.1.patch.zip, 
> HIVE-16207.2.patch, HIVE-16207.3.patch, partial.patch
>
>
> Add complex type support to Fast SerDe classes.  This is needed for fully 
> supporting complex types in Vectorization



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16330) Improve plans for scalar subquery with aggregates

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003538#comment-16003538
 ] 

Ashutosh Chauhan commented on HIVE-16330:
-

Seems like golden files need update. Can you also create RB when you update the 
patch?

> Improve plans for scalar subquery with aggregates
> -
>
> Key: HIVE-16330
> URL: https://issues.apache.org/jira/browse/HIVE-16330
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16330.1.patch
>
>
> Scalar subquery plans are generated with a count(*) on subquery which is fed 
> to {{sq_count_check}} UDF. This is to make sure at runtime that there is at 
> most one row generated by scalar subquery. 
> We can avoid generating this extra count(*) for scalar subqueries with 
> aggregates and windowing since such queries are guaranteed to generate at 
> most one row. e.g. {code:SQL} select * from part where p_size > (select 
> max(p_size) from part) {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16330) Improve plans for scalar subquery with aggregates

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003539#comment-16003539
 ] 

Vineet Garg commented on HIVE-16330:


I found a bug in patch and I am working on fixing it. I'll create RB as soon as 
I fix it with golden files update.

> Improve plans for scalar subquery with aggregates
> -
>
> Key: HIVE-16330
> URL: https://issues.apache.org/jira/browse/HIVE-16330
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16330.1.patch
>
>
> Scalar subquery plans are generated with a count(*) on subquery which is fed 
> to {{sq_count_check}} UDF. This is to make sure at runtime that there is at 
> most one row generated by scalar subquery. 
> We can avoid generating this extra count(*) for scalar subqueries with 
> aggregates and windowing since such queries are guaranteed to generate at 
> most one row. e.g. {code:SQL} select * from part where p_size > (select 
> max(p_size) from part) {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14389) Beeline should not output query and prompt to stdout

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003605#comment-16003605
 ] 

Hive QA commented on HIVE-14389:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867174/HIVE-14389.02.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10666 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=144)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5141/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5141/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5141/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867174 - PreCommit-HIVE-Build

> Beeline should not output query and prompt to stdout
> 
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-14389.01.patch, HIVE-14389.02.patch
>
>
> It seems that the Beeline prints the query along with the results in the 
> stdout when a script file is passed. The output file in the example below 
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u 
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2> 
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car  |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts 
> to Beeline, this needs to be taken care of. The output files generated by 
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout, 
> which is adding garbage along with results, as just like HIVE CLI does, users 
> would like to have only the results on stdout, not errors/debugging info/etc, 
> like the full query. 
> Query could be printed out, no problem, as long as it is not on stdout (with 
> results), instead, it must be printed out along with the debugging info.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)