[jira] [Work started] (HIVE-16272) support for drop function in incremental replication

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-16272 started by anishek.
--
> support for drop function in incremental replication 
> -
>
> Key: HIVE-16272
> URL: https://issues.apache.org/jira/browse/HIVE-16272
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
>
> drop function should work in incremental dump and incremental load



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16269) enable incremental function dump to be loaded via repl load

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002096#comment-16002096
 ] 

ASF GitHub Bot commented on HIVE-16269:
---

GitHub user anishek opened a pull request:

https://github.com/apache/hive/pull/182

HIVE-16269 : enable incremental function dump to be loaded via repl load



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anishek/hive HIVE-16269

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/182.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #182


commit c00688d534ebac85e4a6286bba4ba09cf6ab6ebe
Author: Anishek Agarwal 
Date:   2017-05-08T05:01:37Z

HIVE-16269 : enable incremental function dump to be loaded via repl load




> enable incremental function dump to be loaded via repl load 
> 
>
> Key: HIVE-16269
> URL: https://issues.apache.org/jira/browse/HIVE-16269
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Attachments: HIVE-16269.1.patch, HIVE-16269.2.patch
>
>
> depends if there is additional spec elements we put out as part of HIVE-16268



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16269) enable incremental function dump to be loaded via repl load

[anishek (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002094#comment-16002094
 ] 

anishek commented on HIVE-16269:


[~sushanth] / [~thejas] Please review the patch.

> enable incremental function dump to be loaded via repl load 
> 
>
> Key: HIVE-16269
> URL: https://issues.apache.org/jira/browse/HIVE-16269
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Attachments: HIVE-16269.1.patch, HIVE-16269.2.patch
>
>
> depends if there is additional spec elements we put out as part of HIVE-16268



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16269) enable incremental function dump to be loaded via repl load

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek updated HIVE-16269:
---
Attachment: HIVE-16269.2.patch

patch after rebase on current master 

> enable incremental function dump to be loaded via repl load 
> 
>
> Key: HIVE-16269
> URL: https://issues.apache.org/jira/browse/HIVE-16269
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Attachments: HIVE-16269.1.patch, HIVE-16269.2.patch
>
>
> depends if there is additional spec elements we put out as part of HIVE-16268



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16612) PerfLogger is configurable, but not extensible

[Remus Rusanu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remus Rusanu reassigned HIVE-16612:
---


> PerfLogger is configurable, but not extensible
> --
>
> Key: HIVE-16612
> URL: https://issues.apache.org/jira/browse/HIVE-16612
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning, Query Processor
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
>
> {code}
>   result = (PerfLogger) 
> ReflectionUtils.newInstance(conf.getClassByName(
> conf.getVar(HiveConf.ConfVars.HIVE_PERF_LOGGER)), conf);
> {code}
> The PerfLogger instance is configurable via {{hive.exec.perf.logger}} 
> (HIVE-11891) but the requirement to extend from {{PerfLogger}} cannot be met 
> since HIVE-11149 as the ctor is private. Also useful methods in PerfLogger 
> are also private. I tried to extend PerfLogger for my needs and realized 
> that, as is, the configurability is not usable. At the very least the 
> PerfLogger should make all private members {{protected}}, better the 
> requirement should be an interface not a class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16595) fix syntax in Hplsql.g4

[Yishuang Lu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002087#comment-16002087
 ] 

Yishuang Lu commented on HIVE-16595:


Agree. Didn't expect many failures in patch2. Create a new patch and will fix 
the testCliDriver test. 

> fix syntax in Hplsql.g4
> ---
>
> Key: HIVE-16595
> URL: https://issues.apache.org/jira/browse/HIVE-16595
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Reporter: Yishuang Lu
>Assignee: Yishuang Lu
> Fix For: 1.2.3
>
> Attachments: HIVE-16595.1.patch, HIVE-16595.2.patch, 
> HIVE-16595.3.patch
>
>
> According to https://github.com/antlr/antlr4/issues/118, incorrect error 
> message might return if the start rule does not contain an explicit EOF 
> transition. It is better to add EOF for the first rule in grammar.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16595) fix syntax in Hplsql.g4

[Yishuang Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yishuang Lu updated HIVE-16595:
---
Attachment: HIVE-16595.3.patch

> fix syntax in Hplsql.g4
> ---
>
> Key: HIVE-16595
> URL: https://issues.apache.org/jira/browse/HIVE-16595
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Reporter: Yishuang Lu
>Assignee: Yishuang Lu
> Fix For: 1.2.3
>
> Attachments: HIVE-16595.1.patch, HIVE-16595.2.patch, 
> HIVE-16595.3.patch
>
>
> According to https://github.com/antlr/antlr4/issues/118, incorrect error 
> message might return if the start rule does not contain an explicit EOF 
> transition. It is better to add EOF for the first rule in grammar.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases

[Remus Rusanu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remus Rusanu reassigned HIVE-16113:
---

Assignee: Remus Rusanu  (was: Gopal V)

> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153  WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main] 
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}} 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14389) Beeline should not output query and prompt to stdout

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002044#comment-16002044
 ] 

Hive QA commented on HIVE-14389:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867017/HIVE-14389.01.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10655 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_9] 
(batchId=78)
org.apache.hive.beeline.TestBeeLineWithArgs.testGetVariableValue (batchId=218)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryNonEscapedSemiColon 
(batchId=218)
org.apache.hive.beeline.TestBeeLineWithArgs.testShowDbInPrompt (batchId=218)
org.apache.hive.beeline.TestBeeLineWithArgs.testTabInScriptFile (batchId=218)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5126/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5126/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5126/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867017 - PreCommit-HIVE-Build

> Beeline should not output query and prompt to stdout
> 
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-14389.01.patch
>
>
> It seems that the Beeline prints the query along with the results in the 
> stdout when a script file is passed. The output file in the example below 
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u 
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2> 
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car  |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts 
> to Beeline, this needs to be taken care of. The output files generated by 
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout, 
> which is adding garbage along with results, as just like HIVE CLI does, users 
> would like to have only the results on stdout, not errors/debugging info/etc, 
> like the full query. 
> Query could be printed out, no problem, as long as it is not on stdout (with 
> results), instead, it must be printed out along with the debugging info.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16582:
-
Attachment: HIVE-16582.4.patch

Fixed NPE that is causing test failures.

> HashTableLoader should log info about the input, rows, size etc.
> 
>
> Key: HIVE-16582
> URL: https://issues.apache.org/jira/browse/HIVE-16582
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-16582.1.patch, HIVE-16582.2.patch, 
> HIVE-16582.3.patch, HIVE-16582.4.patch
>
>
> Will be useful to log the following info during hash table loading
> - input name
> - number of rows 
> - estimated data size (LLAP tracks this)
> - object cache key



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16207) Add support for Complex Types in Fast SerDe

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002000#comment-16002000
 ] 

Hive QA commented on HIVE-16207:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867002/HIVE-16207.2.patch

{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10660 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part_all_primitive]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_table]
 (batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_1]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_interval_arithmetic]
 (batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
 (batchId=151)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
 (batchId=118)
org.apache.hadoop.hive.ql.exec.vector.TestVectorSerDeRow.testVectorLazySimpleDeserializeRow
 (batchId=270)
org.apache.hive.hcatalog.pig.TestRCFileHCatStorer.testWriteDecimalX 
(batchId=178)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteVarchar 
(batchId=178)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5125/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5125/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5125/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12867002 - PreCommit-HIVE-Build

> Add support for Complex Types in Fast SerDe
> ---
>
> Key: HIVE-16207
> URL: https://issues.apache.org/jira/browse/HIVE-16207
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
> Attachments: HIVE-16207.1.patch, HIVE-16207.1.patch.zip, 
> HIVE-16207.2.patch, partial.patch
>
>
> Add complex type support to Fast SerDe classes.  This is needed for fully 
> supporting complex types in Vectorization



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases

[Rui Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001991#comment-16001991
 ] 

Rui Li commented on HIVE-16600:
---

Thanks [~kellyzly] for working on this. I don't think test failures are related.
I remember you mentioned there was something wrong when you first enable 
parallel order by for multi insert. Have you figured out what was the cause?
Besides, please add a qtest for this. I think it should cover simple multi 
insert, as well as multi insert + limit.

> Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel 
> order by in multi_insert cases
> 
>
> Key: HIVE-16600
> URL: https://issues.apache.org/jira/browse/HIVE-16600
> Project: Hive
>  Issue Type: Sub-task
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16600.1.patch
>
>
> multi_insert_gby.case.q
> {code}
> set hive.exec.reducers.bytes.per.reducer=256;
> set hive.optimize.sampling.orderby=true;
> drop table if exists e1;
> drop table if exists e2;
> create table e1 (key string, value string);
> create table e2 (key string);
> FROM (select key, cast(key as double) as keyD, value from src order by key) a
> INSERT OVERWRITE TABLE e1
> SELECT key, value
> INSERT OVERWRITE TABLE e2
> SELECT key;
> select * from e1;
> select * from e2;
> {code} 
> the parallelism of Sort is 1 even we enable parallel order 
> by("hive.optimize.sampling.orderby" is set as "true").  This is not 
> reasonable because the parallelism  should be calcuated by  
> [Utilities.estimateReducers|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L170]
> this is because SetSparkReducerParallelism#needSetParallelism returns false 
> when [children size of 
> RS|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L207]
>  is greater than 1.
> in this case, the children size of {{RS[2]}} is two.
> the logical plan of the case
> {code}
>TS[0]-SEL[1]-RS[2]-SEL[3]-SEL[4]-FS[5]
> -SEL[6]-FS[7]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001971#comment-16001971
 ] 

Hive QA commented on HIVE-16582:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866998/HIVE-16582.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 50 failed/errored test(s), 10654 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[quotedid_skew] 
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin10] 
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin11] 
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin1] 
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin2] 
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin3] 
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin4] 
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin6] 
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin7] 
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin9] 
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_union_remove_1] 
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_union_remove_2] 
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt10] 
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt11] 
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt12] 
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt14] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt16] 
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt17] 
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt19] 
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt1] 
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt20] 
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt21] 
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt3] 
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt4] 
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt5] 
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt6] 
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt7] 
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt8] 
(batchId=25)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
 (batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoin_union_remove_1]
 (batchId=135)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoin_union_remove_2]
 (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt10] 
(batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt11] 
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt12] 
(batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt14] 
(batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt15] 
(batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt16] 
(batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt17] 
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt19] 
(batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt1] 
(batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt20] 
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt2] 
(batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt3] 
(batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt4] 
(batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt5] 
(batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt6] 
(batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt7] 
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt8] 
(batchId=110)
{noformat}

Test results: 

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

[Carter Shanklin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001958#comment-16001958
 ] 

Carter Shanklin commented on HIVE-14412:


Rui, tried the latest patch looks great.

> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.10.patch, HIVE-14412.11.patch, 
> HIVE-14412.12.patch, HIVE-14412.1.patch, HIVE-14412.2.patch, 
> HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, 
> HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch, HIVE-14412.9.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by 
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or 
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001944#comment-16001944
 ] 

Hive QA commented on HIVE-16582:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866998/HIVE-16582.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 52 failed/errored test(s), 10642 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[quotedid_skew] 
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin10] 
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin11] 
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin1] 
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin2] 
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin3] 
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin4] 
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin6] 
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin7] 
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin9] 
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_union_remove_1] 
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_union_remove_2] 
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt10] 
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt11] 
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt12] 
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt14] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt16] 
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt17] 
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt19] 
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt1] 
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt20] 
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt21] 
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt3] 
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt4] 
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt5] 
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt6] 
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt7] 
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt8] 
(batchId=25)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
 (batchId=95)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
 (batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoin_union_remove_1]
 (batchId=135)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoin_union_remove_2]
 (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt10] 
(batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt11] 
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt12] 
(batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt14] 
(batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt15] 
(batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt16] 
(batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt17] 
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt19] 
(batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt1] 
(batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt20] 
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt2] 
(batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt3] 
(batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt4] 
(batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt5] 
(batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt6] 
(batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt7] 
(batchId=120)

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

[Rui Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001925#comment-16001925
 ] 

Rui Li commented on HIVE-14412:
---

The latest failure are not related.

[~cartershanklin], the Postgres example you mentioned can be achieved with v12 
patch. But since Hive currently doesn't support session zone, user has to 
change system zone to do that. I also added a test case for it:
{code}
  @Test
  public void testConvertFromTimestamp() {
TimeZone defaultZone = TimeZone.getDefault();
try {
  // Use system zone when converting from timestamp to timestamptz
  String s = "2017-06-12 23:12:56.34";
  TimeZone.setDefault(TimeZone.getTimeZone("Europe/London"));
  TimestampTZ tstz1 = TimestampTZ.convert(Timestamp.valueOf(s));
  TimeZone.setDefault(TimeZone.getTimeZone("America/Los_Angeles"));
  TimestampTZ tstz2 = TimestampTZ.convert(Timestamp.valueOf(s));
  Assert.assertTrue(tstz1.compareTo(tstz2) < 0);
} finally {
  TimeZone.setDefault(defaultZone);
}
  }
{code}
What I was trying to say is, by converting to a target timezone, users usually 
expect to see the timestamp displayed in that timezone. E.g. if users want to 
convert '2012-01-01 08:00:00+00' to PST8PDT, they should be expecting 
'2012-01-01 00:00:00-08' right? But w/o storing the zone, all timestamptz 
display in UTC. Session zone doesn't seem to help because the target zone isn't 
necessarily same as session zone.

> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.10.patch, HIVE-14412.11.patch, 
> HIVE-14412.12.patch, HIVE-14412.1.patch, HIVE-14412.2.patch, 
> HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, 
> HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch, HIVE-14412.9.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by 
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or 
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001923#comment-16001923
 ] 

Siddharth Seth commented on HIVE-16343:
---

+1. I'd still test the smap approach for perf.

> LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
> 
>
> Key: HIVE-16343
> URL: https://issues.apache.org/jira/browse/HIVE-16343
> Project: Hive
>  Issue Type: Improvement
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16343.1.patch, HIVE-16343.2.patch
>
>
> Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful 
> for monitoring and also setting up triggers via JMC. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14389) Beeline should not output query and prompt to stdout

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-14389:
---
Status: Patch Available  (was: Open)

> Beeline should not output query and prompt to stdout
> 
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-14389.01.patch
>
>
> It seems that the Beeline prints the query along with the results in the 
> stdout when a script file is passed. The output file in the example below 
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u 
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2> 
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car  |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts 
> to Beeline, this needs to be taken care of. The output files generated by 
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout, 
> which is adding garbage along with results, as just like HIVE CLI does, users 
> would like to have only the results on stdout, not errors/debugging info/etc, 
> like the full query. 
> Query could be printed out, no problem, as long as it is not on stdout (with 
> results), instead, it must be printed out along with the debugging info.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14389) Beeline should not output query and prompt to stdout

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-14389:
---
Attachment: HIVE-14389.01.patch

> Beeline should not output query and prompt to stdout
> 
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-14389.01.patch
>
>
> It seems that the Beeline prints the query along with the results in the 
> stdout when a script file is passed. The output file in the example below 
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u 
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2> 
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car  |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts 
> to Beeline, this needs to be taken care of. The output files generated by 
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout, 
> which is adding garbage along with results, as just like HIVE CLI does, users 
> would like to have only the results on stdout, not errors/debugging info/etc, 
> like the full query. 
> Query could be printed out, no problem, as long as it is not on stdout (with 
> results), instead, it must be printed out along with the debugging info.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16599) NPE in runtime filtering cost when handling SMB Joins

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001901#comment-16001901
 ] 

Hive QA commented on HIVE-16599:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866992/HIVE-16599.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10653 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=148)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5122/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5122/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5122/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866992 - PreCommit-HIVE-Build

> NPE in runtime filtering cost when handling SMB Joins
> -
>
> Key: HIVE-16599
> URL: https://issues.apache.org/jira/browse/HIVE-16599
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-16599.1.patch
>
>
> A test with SMB joins failed with NPE in runtime filtering costing logic.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16568) Support complex types in external LLAP InputFormat

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-16568:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Committed to master

> Support complex types in external LLAP InputFormat
> --
>
> Key: HIVE-16568
> URL: https://issues.apache.org/jira/browse/HIVE-16568
> Project: Hive
>  Issue Type: Sub-task
>  Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 3.0.0
>
> Attachments: HIVE-16568.1.patch, HIVE-16568.2.patch, 
> HIVE-16568.3.patch
>
>
> Currently just supports primitive types



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16330) Improve plans for scalar subquery with aggregates

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-16330:
---
Attachment: HIVE-16330.1.patch

> Improve plans for scalar subquery with aggregates
> -
>
> Key: HIVE-16330
> URL: https://issues.apache.org/jira/browse/HIVE-16330
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16330.1.patch
>
>
> Scalar subquery plans are generated with a count(*) on subquery which is fed 
> to {{sq_count_check}} UDF. This is to make sure at runtime that there is at 
> most one row generated by scalar subquery. 
> We can avoid generating this extra count(*) for scalar subqueries with 
> aggregates and windowing since such queries are guaranteed to generate at 
> most one row. e.g. {code:SQL} select * from part where p_size > (select 
> max(p_size) from part) {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001868#comment-16001868
 ] 

Xuefu Zhang commented on HIVE-16456:


[~zxu] See it now. Thanks for the explanation. +1 from me also.

> Kill spark job when InterruptedException happens or driverContext.isShutdown 
> is true.
> -
>
> Key: HIVE-16456
> URL: https://issues.apache.org/jira/browse/HIVE-16456
> Project: Hive
>  Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch
>
>
> Kill spark job when InterruptedException happens or driverContext.isShutdown 
> is true. If the InterruptedException happened in RemoteSparkJobMonitor and 
> LocalSparkJobMonitor, it will be better to kill the job. Also there is a race 
> condition between submit the spark job and query/operation cancellation, it 
> will be better to check driverContext.isShutdown right after submit the spark 
> job. This will guarantee the job being killed no matter when shutdown is 
> called. It is similar as HIVE-15997.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-10865) Beeline needs to support DELIMITER command

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001866#comment-16001866
 ] 

Hive QA commented on HIVE-10865:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866990/HIVE-10865.6.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10657 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=148)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5121/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5121/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5121/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866990 - PreCommit-HIVE-Build

> Beeline needs to support DELIMITER command
> --
>
> Key: HIVE-10865
> URL: https://issues.apache.org/jira/browse/HIVE-10865
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Sahil Takiar
> Attachments: HIVE-10865.1.patch, HIVE-10865.2.patch, 
> HIVE-10865.3.patch, HIVE-10865.4.patch, HIVE-10865.5.patch, HIVE-10865.6.patch
>
>
> MySQL Client provides a DELIMITER command to set statement delimiter.
> Beeline needs to support a similar command to allow commands having 
> semi-colon as non-statement delimiter (as with MySQL stored procedures). This 
> is a follow-up jira for HIVE-10659



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-16609:
--
Description: 
A variation of alter_partition_change_col.q produces wrong result:
{code}
SET hive.exec.dynamic.partition.mode = nonstrict;
create table alter_partition_change_col0 (c1 string, c2 string);
load data local inpath 'dec.txt' overwrite into table 
alter_partition_change_col0;
create table alter_partition_change_col1 (c1 string, c2 string) partitioned by 
(p1 string comment 'Column p1', p2 string comment 'Column p2');

insert overwrite table alter_partition_change_col1 partition (p1, p2)
  select c1, c2, 'abc', '123' from alter_partition_change_col0
  union all
  select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
select * from alter_partition_change_col1 where p1='__HIVE_DEFAULT_PARTITION__' 
or lower(p1)='a';
{code}
The "select" statement does not produce the rows containing 
"__HIVE_DEFAULT_PARTITION__".

We need another condition containing a udf so the condition is not recognized 
by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like HIVE-11208 
breaks it.

  was:
A variation of drop_partitions_filter4.q produces wrong result:
{code}
create table ptestfilter (a string, b int) partitioned by (c string, d int);
INSERT OVERWRITE TABLE ptestfilter PARTITION (c,d) select 'Col1', 1, null, null;
INSERT OVERWRITE TABLE ptestfilter PARTITION (c,d) select 'Col2', 2, null, 2;
INSERT OVERWRITE TABLE ptestfilter PARTITION (c,d) select 'Col3', 3, 'Uganda', 
null;
select * from ptestfilter where c='__HIVE_DEFAULT_PARTITION__' or lower(c)='a';
{code}
The "select" statement does not produce the rows containing 
"__HIVE_DEFAULT_PARTITION__".

Note "select * from ptestfilter where c is null or lower(c)='a';" works fine.

In the query, c is a non-string partition column, we need another condition 
containing a udf so the condition is not recognized by 
PartFilterExprUtil.makeExpressionTree in ObjectStore. HIVE-11208/HIVE-15923 is 
addressing a similar issue in drop partition, however, select is not covered.


> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>
> A variation of alter_partition_change_col.q produces wrong result:
> {code}
> SET hive.exec.dynamic.partition.mode = nonstrict;
> create table alter_partition_change_col0 (c1 string, c2 string);
> load data local inpath 'dec.txt' overwrite into table 
> alter_partition_change_col0;
> create table alter_partition_change_col1 (c1 string, c2 string) partitioned 
> by (p1 string comment 'Column p1', p2 string comment 'Column p2');
> insert overwrite table alter_partition_change_col1 partition (p1, p2)
>   select c1, c2, 'abc', '123' from alter_partition_change_col0
>   union all
>   select c1, c2, cast(null as string), '123' from alter_partition_change_col0;
> select * from alter_partition_change_col1 where 
> p1='__HIVE_DEFAULT_PARTITION__' or lower(p1)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> We need another condition containing a udf so the condition is not recognized 
> by PartFilterExprUtil.makeExpressionTree in ObjectStore. Looks like 
> HIVE-11208 breaks it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16611) Kryo remove field is not working

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran reassigned HIVE-16611:


Assignee: Prasanth Jayachandran

> Kryo remove field is not working
> 
>
> Key: HIVE-16611
> URL: https://issues.apache.org/jira/browse/HIVE-16611
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 1.2.1
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> I have seen some instances where removeField in SerializationUtilities is not 
> working
> {code}
>   removeField(kryo, Operator.class, "colExprMap");
>   removeField(kryo, AbstractOperatorDesc.class, "statistics");
> {code}
> These 2 fields are not expected to be serialized but some stacktraces where 
> custom UDFs are involved these fields are observed in the stacktrace. 
> {code}
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) 
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) 
> colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) 
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) 
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) 
> at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>  
> ...(snip)...
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) 
> at 
> org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1173)
>  
> at 
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1062) 
> at 
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1076) 
> {code}
> This needs further investigation. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16611) Kryo remove field is not working

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16611:
-
Affects Version/s: (was: 3.0.0)
   1.2.1

> Kryo remove field is not working
> 
>
> Key: HIVE-16611
> URL: https://issues.apache.org/jira/browse/HIVE-16611
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 1.2.1
>Reporter: Prasanth Jayachandran
>
> I have seen some instances where removeField in SerializationUtilities is not 
> working
> {code}
>   removeField(kryo, Operator.class, "colExprMap");
>   removeField(kryo, AbstractOperatorDesc.class, "statistics");
> {code}
> These 2 fields are not expected to be serialized but some stacktraces where 
> custom UDFs are involved these fields are observed in the stacktrace. 
> {code}
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) 
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) 
> colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) 
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) 
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) 
> at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>  
> ...(snip)...
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) 
> at 
> org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1173)
>  
> at 
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1062) 
> at 
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1076) 
> {code}
> This needs further investigation. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16555) Add a new thrift API call for get_metastore_uuid

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001796#comment-16001796
 ] 

Hive QA commented on HIVE-16555:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866991/HIVE-16555.05.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10664 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1] 
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5120/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5120/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5120/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866991 - PreCommit-HIVE-Build

> Add a new thrift API call for get_metastore_uuid
> 
>
> Key: HIVE-16555
> URL: https://issues.apache.org/jira/browse/HIVE-16555
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16555.01.patch, HIVE-16555.02.patch, 
> HIVE-16555.03.patch, HIVE-16555.04.patch, HIVE-16555.05.patch
>
>
> Sub-task of the main JIRA to add the new thrift API



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16207) Add support for Complex Types in Fast SerDe

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi updated HIVE-16207:
--
Attachment: HIVE-16207.2.patch

> Add support for Complex Types in Fast SerDe
> ---
>
> Key: HIVE-16207
> URL: https://issues.apache.org/jira/browse/HIVE-16207
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
> Attachments: HIVE-16207.1.patch, HIVE-16207.1.patch.zip, 
> HIVE-16207.2.patch, partial.patch
>
>
> Add complex type support to Fast SerDe classes.  This is needed for fully 
> supporting complex types in Vectorization



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001781#comment-16001781
 ] 

Xuefu Zhang commented on HIVE-14412:


Since session zone is not currently available in Hive, I think that deserves 
another JIRA for discussion. Here we are adding a new data type, which can be 
developed  orthogonally along with session time zone.

> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.10.patch, HIVE-14412.11.patch, 
> HIVE-14412.12.patch, HIVE-14412.1.patch, HIVE-14412.2.patch, 
> HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, 
> HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch, HIVE-14412.9.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by 
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or 
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

[Carter Shanklin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001774#comment-16001774
 ] 

Carter Shanklin commented on HIVE-14412:


You can store the time zone or adjust the values based on the session time zone.

Consider this Postgres example:
{code}
vagrant=# set time zone 'GMT';
SET
vagrant=# create table test (c1 timestamp with time zone);
CREATE TABLE
vagrant=# insert into test values (date '2012-01-01');
INSERT 0 1
vagrant=# set time zone 'PST8PDT';
SET
vagrant=# insert into test values (date '2012-01-01');
INSERT 0 1
vagrant=# set time zone 'GMT';
SET
vagrant=# select * from test;
   c1

 2012-01-01 00:00:00+00
 2012-01-01 08:00:00+00
(2 rows)
{code}

Yet Postgres is not actually storing the time zone, it tries to be clever and 
automatically adjust based on your session time zone (and works most of the 
time), e.g.:

{code}
vagrant=# set time zone 'GMT';
SET
vagrant=# select extract(timezone from c1) from test;
 date_part
---
 0
 0
(2 rows)

vagrant=# set time zone 'PST8PDT';
SET
vagrant=# select extract(timezone from c1) from test;
 date_part
---
-28800
-28800
(2 rows)
{code}

Essentially saying that everything is in GMT internally but the session time 
zone adjust things so the developer doesn't think about it.

> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.10.patch, HIVE-14412.11.patch, 
> HIVE-14412.12.patch, HIVE-14412.1.patch, HIVE-14412.2.patch, 
> HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, 
> HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch, HIVE-14412.9.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by 
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or 
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16485) Enable outputName for RS operator in explain formatted

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001770#comment-16001770
 ] 

Ashutosh Chauhan commented on HIVE-16485:
-

+1

> Enable outputName for RS operator in explain formatted
> --
>
> Key: HIVE-16485
> URL: https://issues.apache.org/jira/browse/HIVE-16485
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16485.01.patch, HIVE-16485.02.patch, 
> HIVE-16485.03.patch, HIVE-16485.04.patch, HIVE-16485.05.patch, 
> HIVE-16485-disableMasking, plan, query
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16582:
-
Attachment: HIVE-16582.3.patch

> HashTableLoader should log info about the input, rows, size etc.
> 
>
> Key: HIVE-16582
> URL: https://issues.apache.org/jira/browse/HIVE-16582
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-16582.1.patch, HIVE-16582.2.patch, 
> HIVE-16582.3.patch
>
>
> Will be useful to log the following info during hash table loading
> - input name
> - number of rows 
> - estimated data size (LLAP tracks this)
> - object cache key



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16582:
-
Attachment: (was: HIVE-16582.3.patch)

> HashTableLoader should log info about the input, rows, size etc.
> 
>
> Key: HIVE-16582
> URL: https://issues.apache.org/jira/browse/HIVE-16582
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-16582.1.patch, HIVE-16582.2.patch, 
> HIVE-16582.3.patch
>
>
> Will be useful to log the following info during hash table loading
> - input name
> - number of rows 
> - estimated data size (LLAP tracks this)
> - object cache key



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16556) Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES table

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001750#comment-16001750
 ] 

Vihang Karajgaonkar commented on HIVE-16556:


Thanks for the review [~ngangam]. The patch which has the code changes is 
another sub-task of the parent JIRA. That patch is available for review on 
HIVE-16555. I have added you to its review board as well.

> Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES 
> table
> 
>
> Key: HIVE-16556
> URL: https://issues.apache.org/jira/browse/HIVE-16556
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16556.01.patch, HIVE-16556.02.patch, 
> HIVE-16556.03.patch, HIVE-16556.04.patch, HIVE-16556.05.patch
>
>
> sub-task to modify schema tool and its related changes so that the new table 
> is added to the schema when schematool initializes or upgrades the schema.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16582:
-
Attachment: HIVE-16582.3.patch

> HashTableLoader should log info about the input, rows, size etc.
> 
>
> Key: HIVE-16582
> URL: https://issues.apache.org/jira/browse/HIVE-16582
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-16582.1.patch, HIVE-16582.2.patch, 
> HIVE-16582.3.patch
>
>
> Will be useful to log the following info during hash table loading
> - input name
> - number of rows 
> - estimated data size (LLAP tracks this)
> - object cache key



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001751#comment-16001751
 ] 

Prasanth Jayachandran commented on HIVE-16582:
--

[~sershe] could you please take a look at new changes?

> HashTableLoader should log info about the input, rows, size etc.
> 
>
> Key: HIVE-16582
> URL: https://issues.apache.org/jira/browse/HIVE-16582
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-16582.1.patch, HIVE-16582.2.patch, 
> HIVE-16582.3.patch
>
>
> Will be useful to log the following info during hash table loading
> - input name
> - number of rows 
> - estimated data size (LLAP tracks this)
> - object cache key



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16582:
-
Attachment: (was: HIVE-16582.3.patch)

> HashTableLoader should log info about the input, rows, size etc.
> 
>
> Key: HIVE-16582
> URL: https://issues.apache.org/jira/browse/HIVE-16582
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-16582.1.patch, HIVE-16582.2.patch, 
> HIVE-16582.3.patch
>
>
> Will be useful to log the following info during hash table loading
> - input name
> - number of rows 
> - estimated data size (LLAP tracks this)
> - object cache key



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16582:
-
Attachment: HIVE-16582.3.patch

Following are the changes in this patch
- Removes concurrency from memory monitoring equation for LLAP as it makes it 
harder to configure and understand..
- Adds all memory monitoring info to exception message
- Adds input name, cache key, number of entries and estimated size to log msgs 
for debugging which input is triggering the failure

> HashTableLoader should log info about the input, rows, size etc.
> 
>
> Key: HIVE-16582
> URL: https://issues.apache.org/jira/browse/HIVE-16582
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-16582.1.patch, HIVE-16582.2.patch, 
> HIVE-16582.3.patch
>
>
> Will be useful to log the following info during hash table loading
> - input name
> - number of rows 
> - estimated data size (LLAP tracks this)
> - object cache key



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (HIVE-16610) Semijoin Hint : Should be able to handle more than one hint per alias

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-16610 started by Deepak Jaiswal.
-
> Semijoin Hint : Should be able to handle more than one hint per alias
> -
>
> Key: HIVE-16610
> URL: https://issues.apache.org/jira/browse/HIVE-16610
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>
> Currently the semi join hints can be used to create only one semi join 
> optimization per alias which is very limiting.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16610) Semijoin Hint : Should be able to handle more than one hint per alias

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal reassigned HIVE-16610:
-

Assignee: Deepak Jaiswal

> Semijoin Hint : Should be able to handle more than one hint per alias
> -
>
> Key: HIVE-16610
> URL: https://issues.apache.org/jira/browse/HIVE-16610
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>
> Currently the semi join hints can be used to create only one semi join 
> optimization per alias which is very limiting.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16555) Add a new thrift API call for get_metastore_uuid

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001710#comment-16001710
 ] 

Hive QA commented on HIVE-16555:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866987/HIVE-16555.04.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10664 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
 (batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5119/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5119/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5119/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866987 - PreCommit-HIVE-Build

> Add a new thrift API call for get_metastore_uuid
> 
>
> Key: HIVE-16555
> URL: https://issues.apache.org/jira/browse/HIVE-16555
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16555.01.patch, HIVE-16555.02.patch, 
> HIVE-16555.03.patch, HIVE-16555.04.patch, HIVE-16555.05.patch
>
>
> Sub-task of the main JIRA to add the new thrift API



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16599) NPE in runtime filtering cost when handling SMB Joins

[Deepak Jaiswal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001698#comment-16001698
 ] 

Deepak Jaiswal commented on HIVE-16599:
---

RB link https://reviews.apache.org/r/59072/

> NPE in runtime filtering cost when handling SMB Joins
> -
>
> Key: HIVE-16599
> URL: https://issues.apache.org/jira/browse/HIVE-16599
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-16599.1.patch
>
>
> A test with SMB joins failed with NPE in runtime filtering costing logic.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16599) NPE in runtime filtering cost when handling SMB Joins

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-16599:
--
Attachment: HIVE-16599.1.patch

Initial patch

> NPE in runtime filtering cost when handling SMB Joins
> -
>
> Key: HIVE-16599
> URL: https://issues.apache.org/jira/browse/HIVE-16599
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-16599.1.patch
>
>
> A test with SMB joins failed with NPE in runtime filtering costing logic.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16446) org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified by setting t

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001686#comment-16001686
 ] 

Vihang Karajgaonkar commented on HIVE-16446:


Hi [~kalexin] I tried this on CDH 5.8 onwards until 5.10. I could not make it 
work using the {{set fs.s3a.secret.key=}} and {{set 
fs.s3a.access.key=}}. Can you tell me the exact version of CDH where it 
works? Are you using BeeLine or HiveCLI? As far as I know you have to add the 
keys to the core-site.xml to make it work.

> org.apache.hadoop.hive.ql.exec.DDLTask. 
> MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID 
> and Secret Access Key must be specified by setting the fs.s3n.awsAccessKeyId 
> and fs.s3n.awsSecretAccessKey properties
> -
>
> Key: HIVE-16446
> URL: https://issues.apache.org/jira/browse/HIVE-16446
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 1.1.0
>Reporter: Kalexin Baoerjiin
>Assignee: Vihang Karajgaonkar
>
> After upgrading our Cloudera cluster to CDH 5.10.1 we are experiencing the 
> following problem during some Hive DDL.
> 
> SET fs.s3n.awsSecretAccessKey=;
> SET fs.s3n.awsAccessKeyId=;
> 
> ALTER TABLE hive_1k_partitions ADD IF NOT EXISTS partition (year='2014', 
> month='2014-01', dt='2014-01-01', hours='00', minutes='16', seconds='22') 
> location 's3n://'
> 
> Stack trace I was able to recover: 
> [ Message content over the limit has been removed. ]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:383)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:318)
> at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:416)
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:432)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:726)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:693)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:628)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Job Submission failed with exception ‘java.lang.IllegalArgumentException(AWS 
> Access Key ID and Secret Access Key must be specified by setting the 
> fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey properties 
> (respectively).)’
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> [9:31] 
> Logging initialized using configuration in 
> jar:file:/opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/jars/hive-common-1.1.0-cdh5.10.1.jar!/hive-log4j.properties
> In the past we did not have to set s3 key and ID in core-site.xml because we 
> were using them dynamically inside our hive DDL scripts.
> After setting S3 secret key and Access ID in core-site.xml this problem goes 
> away. However this is an incompatibility change from the previous Hive 
> shipped in CDH 5.9. 
> Cloudera 5.10.x release note mentioned (HIVE-14269 : Enhanced write 
> performance for Hive tables stored on Amazon S3.) is the only Hive related 
> changes. 
> https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_new_in_cdh_510.html
> https://issues.apache.org/jira/browse/HIVE-14269



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16599) NPE in runtime filtering cost when handling SMB Joins

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-16599:
--
Status: Patch Available  (was: Open)

> NPE in runtime filtering cost when handling SMB Joins
> -
>
> Key: HIVE-16599
> URL: https://issues.apache.org/jira/browse/HIVE-16599
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>
> A test with SMB joins failed with NPE in runtime filtering costing logic.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16555) Add a new thrift API call for get_metastore_uuid

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-16555:
---
Attachment: HIVE-16555.05.patch

> Add a new thrift API call for get_metastore_uuid
> 
>
> Key: HIVE-16555
> URL: https://issues.apache.org/jira/browse/HIVE-16555
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16555.01.patch, HIVE-16555.02.patch, 
> HIVE-16555.03.patch, HIVE-16555.04.patch, HIVE-16555.05.patch
>
>
> Sub-task of the main JIRA to add the new thrift API



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-10865) Beeline needs to support DELIMITER command

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-10865:

Attachment: HIVE-10865.6.patch

> Beeline needs to support DELIMITER command
> --
>
> Key: HIVE-10865
> URL: https://issues.apache.org/jira/browse/HIVE-10865
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Sahil Takiar
> Attachments: HIVE-10865.1.patch, HIVE-10865.2.patch, 
> HIVE-10865.3.patch, HIVE-10865.4.patch, HIVE-10865.5.patch, HIVE-10865.6.patch
>
>
> MySQL Client provides a DELIMITER command to set statement delimiter.
> Beeline needs to support a similar command to allow commands having 
> semi-colon as non-statement delimiter (as with MySQL stored procedures). This 
> is a follow-up jira for HIVE-10659



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16609) col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce wrong result

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai reassigned HIVE-16609:
-


> col='__HIVE_DEFAULT_PARTITION__' condition in select statement may produce 
> wrong result
> ---
>
> Key: HIVE-16609
> URL: https://issues.apache.org/jira/browse/HIVE-16609
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>
> A variation of drop_partitions_filter4.q produces wrong result:
> {code}
> create table ptestfilter (a string, b int) partitioned by (c string, d int);
> INSERT OVERWRITE TABLE ptestfilter PARTITION (c,d) select 'Col1', 1, null, 
> null;
> INSERT OVERWRITE TABLE ptestfilter PARTITION (c,d) select 'Col2', 2, null, 2;
> INSERT OVERWRITE TABLE ptestfilter PARTITION (c,d) select 'Col3', 3, 
> 'Uganda', null;
> select * from ptestfilter where c='__HIVE_DEFAULT_PARTITION__' or 
> lower(c)='a';
> {code}
> The "select" statement does not produce the rows containing 
> "__HIVE_DEFAULT_PARTITION__".
> Note "select * from ptestfilter where c is null or lower(c)='a';" works fine.
> In the query, c is a non-string partition column, we need another condition 
> containing a udf so the condition is not recognized by 
> PartFilterExprUtil.makeExpressionTree in ObjectStore. HIVE-11208/HIVE-15923 
> is addressing a similar issue in drop partition, however, select is not 
> covered.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16555) Add a new thrift API call for get_metastore_uuid

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-16555:
---
Attachment: HIVE-16555.04.patch

> Add a new thrift API call for get_metastore_uuid
> 
>
> Key: HIVE-16555
> URL: https://issues.apache.org/jira/browse/HIVE-16555
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16555.01.patch, HIVE-16555.02.patch, 
> HIVE-16555.03.patch, HIVE-16555.04.patch
>
>
> Sub-task of the main JIRA to add the new thrift API



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16485) Enable outputName for RS operator in explain formatted

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001616#comment-16001616
 ] 

Hive QA commented on HIVE-16485:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866979/HIVE-16485.05.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10653 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1] 
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=148)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5118/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5118/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5118/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866979 - PreCommit-HIVE-Build

> Enable outputName for RS operator in explain formatted
> --
>
> Key: HIVE-16485
> URL: https://issues.apache.org/jira/browse/HIVE-16485
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16485.01.patch, HIVE-16485.02.patch, 
> HIVE-16485.03.patch, HIVE-16485.04.patch, HIVE-16485.05.patch, 
> HIVE-16485-disableMasking, plan, query
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16595) fix syntax in Hplsql.g4

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001548#comment-16001548
 ] 

Alan Gates commented on HIVE-16595:
---

It seems that patch 1 does it correctly.  Making EOF part of program makes more 
sense, since in the future a program might contain multiple blocks.  Also 
changing it in patch 2 broke a lot of tests.  Why the change?



> fix syntax in Hplsql.g4
> ---
>
> Key: HIVE-16595
> URL: https://issues.apache.org/jira/browse/HIVE-16595
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Reporter: Yishuang Lu
>Assignee: Yishuang Lu
> Fix For: 1.2.3
>
> Attachments: HIVE-16595.1.patch, HIVE-16595.2.patch
>
>
> According to https://github.com/antlr/antlr4/issues/118, incorrect error 
> message might return if the start rule does not contain an explicit EOF 
> transition. It is better to add EOF for the first rule in grammar.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16324) Truncate table should not work when EXTERNAL property of table is true

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001522#comment-16001522
 ] 

Vihang Karajgaonkar commented on HIVE-16324:


Looks like the issue was happening because the parameter value is case 
sensitive. So if I try using {{'EXTERNAL'='TRUE'}} then it works as expected 
but doesn't work with the parameter value set to "true". Looks like it is by 
design. [~ashutoshc] Do you think its a good idea to make it case insensitive? 
If not, I can close this JIRA.

> Truncate table should not work when EXTERNAL property of table is true
> --
>
> Key: HIVE-16324
> URL: https://issues.apache.org/jira/browse/HIVE-16324
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> Currently if you create an external table using the command {{CREATE EXTERNAL 
> TABLE table_name}} the {{TRUNCATE table table_name}} command fails as 
> expected because only managed tables should be allowed to be truncated.
> But if you set the external property of a previously managed table using 
> {{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate 
> table command does not object and deletes all the data from the external 
> table.
> Eg: This works but it should not ..
> {noformat}
> 0: jdbc:hive2://localhost:1/default> create table test_ext2 (col1 string);
> No rows affected (0.424 seconds)
> 0: jdbc:hive2://localhost:1/default> alter table test_ext2 set 
> tblproperties ('EXTERNAL'='true');
> No rows affected (0.149 seconds)
> 0: jdbc:hive2://localhost:1/default> insert into table test_ext2 values 
> ("test");
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the 
> future versions. Consider using a different execution engine (i.e. spark, 
> tez) or using Hive 1.X releases.
> No rows affected (3.447 seconds)
> 0: jdbc:hive2://localhost:1/default>
> 0: jdbc:hive2://localhost:1/default>
> 0: jdbc:hive2://localhost:1/default> select * from test_ext2;
> +-+
> | test_ext2.col1  |
> +-+
> | test|
> +-+
> 1 row selected (0.147 seconds)
> 0: jdbc:hive2://localhost:1/default> truncate table test_ext2;
> No rows affected (0.138 seconds)
> 0: jdbc:hive2://localhost:1/default> select * from test_ext2;
> +-+
> | test_ext2.col1  |
> +-+
> +-+
> No rows selected (0.134 seconds)
> 0: jdbc:hive2://localhost:1/default>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16485) Enable outputName for RS operator in explain formatted

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-16485:
---
Attachment: HIVE-16485.05.patch

> Enable outputName for RS operator in explain formatted
> --
>
> Key: HIVE-16485
> URL: https://issues.apache.org/jira/browse/HIVE-16485
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16485.01.patch, HIVE-16485.02.patch, 
> HIVE-16485.03.patch, HIVE-16485.04.patch, HIVE-16485.05.patch, 
> HIVE-16485-disableMasking, plan, query
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16485) Enable outputName for RS operator in explain formatted

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-16485:
---
Status: Patch Available  (was: Open)

> Enable outputName for RS operator in explain formatted
> --
>
> Key: HIVE-16485
> URL: https://issues.apache.org/jira/browse/HIVE-16485
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16485.01.patch, HIVE-16485.02.patch, 
> HIVE-16485.03.patch, HIVE-16485.04.patch, HIVE-16485.05.patch, 
> HIVE-16485-disableMasking, plan, query
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16485) Enable outputName for RS operator in explain formatted

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-16485:
---
Status: Open  (was: Patch Available)

> Enable outputName for RS operator in explain formatted
> --
>
> Key: HIVE-16485
> URL: https://issues.apache.org/jira/browse/HIVE-16485
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16485.01.patch, HIVE-16485.02.patch, 
> HIVE-16485.03.patch, HIVE-16485.04.patch, HIVE-16485.05.patch, 
> HIVE-16485-disableMasking, plan, query
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16607) ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001476#comment-16001476
 ] 

Hive QA commented on HIVE-16607:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866953/HIVE-16607.patch

{color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 36 failed/errored test(s), 10655 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
 (batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments] 
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
 (batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[select_dummy_source] 
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_10] 
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11] 
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12] 
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13] 
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_16] 
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1] 
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_2] 
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_3] 
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7] 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_4] (batchId=6)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbasestats] 
(batchId=91)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
org.apache.hive.beeline.TestBeeLineWithArgs.testEmbeddedBeelineOutputs 
(batchId=218)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=218)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel 
(batchId=218)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressWithHiveServer2ProgressBarDisabled
 (batchId=218)
org.apache.hive.jdbc.TestJdbcDriver2.testGetQueryLog (batchId=222)
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery (batchId=224)
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable (batchId=224)
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery 
(batchId=224)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithMr.testFetchResultsOfLog
 (batchId=220)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithMr.testFetchResultsOfLogAsync
 (batchId=220)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithMr.testFetchResultsOfLogCleanup
 (batchId=220)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithMr.testFetchResultsOfLogWithExecutionMode
 (batchId=220)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithMr.testFetchResultsOfLogWithOrientation
 (batchId=220)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithMr.testFetchResultsOfLogWithPerformanceMode
 (batchId=220)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithMr.testFetchResultsOfLogWithVerboseMode
 (batchId=220)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithTez.testFetchResultsOfLogCleanup
 (batchId=219)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithTez.testFetchResultsOfLogWithExecutionMode
 (batchId=219)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithTez.testFetchResultsOfLogWithNoneMode
 (batchId=219)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithTez.testFetchResultsOfLogWithPerformanceMode
 (batchId=219)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithTez.testFetchResultsOfLogWithVerboseMode
 (batchId=219)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5117/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5117/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5117/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 36 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866953 - PreCommit-HIVE-Build

> ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID
> -
>
> Key: HIVE-16607
> URL: https://issues.apache.org/jira/browse/HIVE-16607
> Project: 

[jira] [Commented] (HIVE-16590) Make initializing dag names in SparkWork thread safe for parallel compilation (HIVE-13512)

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001438#comment-16001438
 ] 

Sahil Takiar commented on HIVE-16590:
-

Test failures are flaky:

[HIVE-15776|https://issues.apache.org/jira/browse/HIVE-15776] - 
TestMiniLlapLocalCliDriver.testCliDriver vector_if_expr 

> Make initializing dag names in SparkWork thread safe for parallel compilation 
> (HIVE-13512)
> --
>
> Key: HIVE-16590
> URL: https://issues.apache.org/jira/browse/HIVE-16590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16590.1.patch, HIVE-16590.2.patch
>
>
> In HIVE-13512 some modifications to {{TezWork}} were made to avoid generating 
> duplicate dag ids during parallel compilation. We should do the equivalent 
> for {{SparkWork}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16590) Make initializing dag names in SparkWork thread safe for parallel compilation (HIVE-13512)

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001418#comment-16001418
 ] 

Hive QA commented on HIVE-16590:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866939/HIVE-16590.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10655 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5116/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5116/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5116/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866939 - PreCommit-HIVE-Build

> Make initializing dag names in SparkWork thread safe for parallel compilation 
> (HIVE-13512)
> --
>
> Key: HIVE-16590
> URL: https://issues.apache.org/jira/browse/HIVE-16590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16590.1.patch, HIVE-16590.2.patch
>
>
> In HIVE-13512 some modifications to {{TezWork}} were made to avoid generating 
> duplicate dag ids during parallel compilation. We should do the equivalent 
> for {{SparkWork}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16598) LlapServiceDriver - create directories and warn of errors

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16598:

   Resolution: Fixed
Fix Version/s: 2.4.0
   3.0.0
   Status: Resolved  (was: Patch Available)

Committed to branches.

> LlapServiceDriver - create directories and warn of errors
> -
>
> Key: HIVE-16598
> URL: https://issues.apache.org/jira/browse/HIVE-16598
> Project: Hive
>  Issue Type: Bug
>Reporter: Kavan Suresh
>Assignee: Sergey Shelukhin
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-16598.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-16605) Enforce NOT NULL constraints

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001386#comment-16001386
 ] 

Sergey Shelukhin edited comment on HIVE-16605 at 5/8/17 7:30 PM:
-

This, and many other such improvements would probably require new table type 
(e.g. MM tables ;)), or only work for ACID because of the general MR/Hive 
approach of treating a table as set of directories with unknown files. 
First, users can (and apparently often do) copy files directly into table 
directories, bypassing any checks. We could also have a read-time check making 
all queries (or all queries where not null is relevant) fail on such tables, 
and advising that the user use external tables if they want to load via copy.
Second, there are also import and load commands; they cannot enforce the 
constraints (without a huge cost increase).


was (Author: sershe):
This, and many other schemes would probably require new table type (e.g. MM 
tables ;)), or only work for ACID because of the general MR/Hive approach of 
treating a table as set of directories with unknown files. 
First, users can (and apparently often do) copy files directly into table 
directories, bypassing any checks. We could also have a read-time check making 
all queries (or all queries where not null is relevant) fail on such tables, 
and advising that the user use external tables if they want to load via copy.
Second, there are also import and load commands; they cannot enforce the 
constraints (without a huge cost increase).

> Enforce NOT NULL constraints
> 
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
>  Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>
> Since NOT NULL is so common it would be great to have tables start to enforce 
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add 
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass 
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-16605) Enforce NOT NULL constraints

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001386#comment-16001386
 ] 

Sergey Shelukhin edited comment on HIVE-16605 at 5/8/17 7:26 PM:
-

This, and many other schemes would probably require new table type (e.g. MM 
tables ;)), or only work for ACID because of the general MR/Hive approach of 
treating a table as set of directories with unknown files. 
First, users can (and apparently often do) copy files directly into table 
directories, bypassing any checks. We could also have a read-time check making 
all queries (or all queries where not null is relevant) fail on such tables, 
and advising that the user use external tables if they want to load via copy.
Second, there are also import and load commands; they cannot enforce the 
constraints (without a huge cost increase).


was (Author: sershe):
This, and many other schemes would probably require new table type (e.g. MM 
tables ;)), or only work for ACID because of the general MR/Hive approach of 
treating a table as set of directories with unknown files. 
First, users can (and apparently often do) copy files directly into table 
directories, bypassing any checks. We could also have a read-time check making 
all queries (or all queries where not null is relevant) fail on such tables, 
and advising that the user use external tables if they want to load via copy.
Second, there are also import and load commands, which cannot enforce the 
constraints.

> Enforce NOT NULL constraints
> 
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
>  Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>
> Since NOT NULL is so common it would be great to have tables start to enforce 
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add 
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass 
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16605) Enforce NOT NULL constraints

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001386#comment-16001386
 ] 

Sergey Shelukhin commented on HIVE-16605:
-

This, and many other schemes would probably require new table type (e.g. MM 
tables ;)), or only work for ACID because of the general MR/Hive approach of 
treating a table as set of directories with unknown files. 
First, users can (and apparently often do) copy files directly into table 
directories, bypassing any checks. We could also have a read-time check making 
all queries (or all queries where not null is relevant) fail on such tables, 
and advising that the user use external tables if they want to load via copy.
Second, there are also import and load commands, which cannot enforce the 
constraints.

> Enforce NOT NULL constraints
> 
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
>  Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>
> Since NOT NULL is so common it would be great to have tables start to enforce 
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add 
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass 
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16555) Add a new thrift API call for get_metastore_uuid

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001384#comment-16001384
 ] 

Sahil Takiar commented on HIVE-16555:
-

[~vihangk1] is this ready for review? Can you create an RB?

> Add a new thrift API call for get_metastore_uuid
> 
>
> Key: HIVE-16555
> URL: https://issues.apache.org/jira/browse/HIVE-16555
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16555.01.patch, HIVE-16555.02.patch, 
> HIVE-16555.03.patch
>
>
> Sub-task of the main JIRA to add the new thrift API



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16556) Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES table

[Naveen Gangam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001377#comment-16001377
 ] 

Naveen Gangam commented on HIVE-16556:
--

Just the schema changes look good to me. So +1 for me on the changes. what jira 
will have code that accesses this table? The HMS APIs, the JDO mapping changes 
and others. Please CC me on that jira as well. Thanks



> Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES 
> table
> 
>
> Key: HIVE-16556
> URL: https://issues.apache.org/jira/browse/HIVE-16556
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16556.01.patch, HIVE-16556.02.patch, 
> HIVE-16556.03.patch, HIVE-16556.04.patch, HIVE-16556.05.patch
>
>
> sub-task to modify schema tool and its related changes so that the new table 
> is added to the schema when schematool initializes or upgrades the schema.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16592) Vectorization: Long hashes use hash64shift and not hash6432shift to generate int hashCodes

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001358#comment-16001358
 ] 

Sergey Shelukhin commented on HIVE-16592:
-

+1

> Vectorization: Long hashes use hash64shift and not hash6432shift to generate 
> int hashCodes
> --
>
> Key: HIVE-16592
> URL: https://issues.apache.org/jira/browse/HIVE-16592
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-16592.1.patch
>
>
> {code}
> public static int calculateLongHashCode(long key) {
> key = (~key) + (key << 21); // key = (key << 21) - key - 1;
> key = key ^ (key >>> 24);
> key = (key + (key << 3)) + (key << 8); // key * 265
> key = key ^ (key >>> 14);
> key = (key + (key << 2)) + (key << 4); // key * 21
> key = key ^ (key >>> 28);
> key = key + (key << 31);
> return (int) key;
>   }
> {code}
> Does not mix enough bits into the lower 32 bits, which are used for the 
> bucket probes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001310#comment-16001310
 ] 

Hive QA commented on HIVE-14412:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866903/HIVE-14412.12.patch

{color:green}SUCCESS:{color} +1 due to 16 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10663 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
 (batchId=237)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5115/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5115/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5115/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866903 - PreCommit-HIVE-Build

> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.10.patch, HIVE-14412.11.patch, 
> HIVE-14412.12.patch, HIVE-14412.1.patch, HIVE-14412.2.patch, 
> HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, 
> HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch, HIVE-14412.9.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by 
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or 
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16604) Use [NOT] ENFORCED for column constraint characteristics

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001268#comment-16001268
 ] 

Ashutosh Chauhan commented on HIVE-16604:
-

Supporting both syntax will be good to have, since people migrating from Oracle 
will not need to change their queries in those cases. However, we need to make 
sure this doesn't add complexity in our parser which will make future changes 
in syntaxes harder to do.

> Use [NOT] ENFORCED for column constraint characteristics
> 
>
> Key: HIVE-16604
> URL: https://issues.apache.org/jira/browse/HIVE-16604
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Jesus Camacho Rodriguez
>
> From HIVE-16575:
> {quote}
> SQL:2011 spec, 10.8, uses [ NOT ] ENFORCED for column constraint 
> characteristics.
> I think Oracle invented DISABLE and NORELY and maybe others have copied it 
> over time.
> A quick check I see DB2 uses the [ NOT ] ENFORCED. Teradata uses something 
> else entirely.
> Personally I think there are advantages to adopting the standard approach.
> There is no standard approach for hinting the optimizer so we're on our own 
> there.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16577) Syntax error in the metastore init scripts for mssql

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001217#comment-16001217
 ] 

Vihang Karajgaonkar commented on HIVE-16577:


Thanks [~aihuaxu]

> Syntax error in the metastore init scripts for mssql
> 
>
> Key: HIVE-16577
> URL: https://issues.apache.org/jira/browse/HIVE-16577
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.2.0, 2.3.0, 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Blocker
> Fix For: 2.3.0, 3.0.0
>
> Attachments: HIVE-16577.01.patch
>
>
> HIVE-10562 introduced a new column to {{NOTIFICATION_LOG}} table. The mssql 
> init scripts which were modified have a syntax error and they fail to 
> initialize metastore schema from 2.2.0 onwards.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16577) Syntax error in the metastore init scripts for mssql

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001200#comment-16001200
 ] 

Aihua Xu commented on HIVE-16577:
-

Just pushed to branch-2.3. 

> Syntax error in the metastore init scripts for mssql
> 
>
> Key: HIVE-16577
> URL: https://issues.apache.org/jira/browse/HIVE-16577
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.2.0, 2.3.0, 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Blocker
> Fix For: 2.3.0, 3.0.0
>
> Attachments: HIVE-16577.01.patch
>
>
> HIVE-10562 introduced a new column to {{NOTIFICATION_LOG}} table. The mssql 
> init scripts which were modified have a syntax error and they fail to 
> initialize metastore schema from 2.2.0 onwards.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001199#comment-16001199
 ] 

Ashutosh Chauhan commented on HIVE-16113:
-

[~rusanu] You were working on this part of code recently.  Would you like to 
take a look at this one?

> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153  WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main] 
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}} 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16577) Syntax error in the metastore init scripts for mssql

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001190#comment-16001190
 ] 

Vihang Karajgaonkar commented on HIVE-16577:


oops sorry I didn't realize that the branch-2.3 is already cut out of branch-2. 
[~aihuaxu] Can you please commit this in branch-2.3 as well since we decided 
this is a must fix before releasing 2.3.0. Thanks [~leftylev] for pointing this 
out.

> Syntax error in the metastore init scripts for mssql
> 
>
> Key: HIVE-16577
> URL: https://issues.apache.org/jira/browse/HIVE-16577
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.2.0, 2.3.0, 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Blocker
> Fix For: 2.3.0, 3.0.0
>
> Attachments: HIVE-16577.01.patch
>
>
> HIVE-10562 introduced a new column to {{NOTIFICATION_LOG}} table. The mssql 
> init scripts which were modified have a syntax error and they fail to 
> initialize metastore schema from 2.2.0 onwards.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16607) ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-16607:
--
Attachment: HIVE-16607.patch

Regenerating the queryId in QueryStates only when created in the Driver 
constructors. Running a check to see if there is any test error caused by this.

Wanted to add a new boolean attribute to the constructor to explicitly request 
a queryId generation, but the multiple boolean parameters of the constructor 
made it hard to read the code. Decided to use a builder instead of the 
constructor to make the code more readable.

> ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID
> -
>
> Key: HIVE-16607
> URL: https://issues.apache.org/jira/browse/HIVE-16607
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, Logging
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16607.patch
>
>
> Creating a new {{QueryState}} object regenerates the HIVEQUERYID stored in 
> the {{HiveConf}}.
> In HiveServer logs it makes hard to follow the life of the query since a new 
> queryid is assigned to the query during the execution.
> Since BeeLine is showing the operation logs based on the queryid, only the 
> first several line of the logs is showed in BeeLine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16607) ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-16607:
--
Status: Patch Available  (was: Open)

> ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID
> -
>
> Key: HIVE-16607
> URL: https://issues.apache.org/jira/browse/HIVE-16607
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, Logging
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16607.patch
>
>
> Creating a new {{QueryState}} object regenerates the HIVEQUERYID stored in 
> the {{HiveConf}}.
> In HiveServer logs it makes hard to follow the life of the query since a new 
> queryid is assigned to the query during the execution.
> Since BeeLine is showing the operation logs based on the queryid, only the 
> first several line of the logs is showed in BeeLine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16559) Parquet schema evolution for partitioned tables may break if table and partition serdes differ

[Barna Zsombor Klara (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001148#comment-16001148
 ] 

Barna Zsombor Klara commented on HIVE-16559:


I ran the same test as in the Jira description but with ORC as the file format 
and HIVE_SCHEMA_EVOLUTION set to false and ended up with:{{Error: 
java.io.IOException: java.lang.ClassCastException: 
org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text 
(state=,code=0)}} when trying to select from the altered table. 
I don't think ORC supports this either.
But then again I don't really understand that check for ORC, how does setting 
HIVE_SCHEMA_EVOLUTION to false end up with a "supported" mode for schema 
evolution?

> Parquet schema evolution for partitioned tables may break if table and 
> partition serdes differ
> --
>
> Key: HIVE-16559
> URL: https://issues.apache.org/jira/browse/HIVE-16559
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Fix For: 3.0.0
>
> Attachments: HIVE-16559.01.patch
>
>
> Parquet schema evolution should make it possible to have partitions/tables 
>  backed by files with different schemas. Hive should match the table columns 
> with file columns based on the column name if possible.
> However if the serde for a table is missing columns from the serde of a 
> partition Hive fails to match the columns together.
> Steps to reproduce:
> {code}
> CREATE TABLE myparquettable_parted
> (
>   name string,
>   favnumber int,
>   favcolor string,
>   age int,
>   favpet string
> )
> PARTITIONED BY (day string)
> STORED AS PARQUET;
> INSERT OVERWRITE TABLE myparquettable_parted
> PARTITION(day='2017-04-04')
> SELECT
>'mary' as name,
>5 AS favnumber,
>'blue' AS favcolor,
>35 AS age,
>'dog' AS favpet;
> alter table myparquettable_parted
> REPLACE COLUMNS
> (
> favnumber int,
> age int
> );   

[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)

[Zhiyuan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhiyuan Yang updated HIVE-14731:

Status: Open  (was: Patch Available)

Cancelling patch since TEZ-3708 isn't committed and Jenkins won't be able to 
compile.

> Use Tez cartesian product edge in Hive (unpartitioned case only)
> 
>
> Key: HIVE-14731
> URL: https://issues.apache.org/jira/browse/HIVE-14731
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, 
> HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, 
> HIVE-14731.15.patch, HIVE-14731.1.patch, HIVE-14731.2.patch, 
> HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, 
> HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch
>
>
> Given cartesian product edge is available in Tez now (see TEZ-3230), let's 
> integrate it into Hive on Tez. This allows us to have more than one reducer 
> in cross product queries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16590) Make initializing dag names in SparkWork thread safe for parallel compilation (HIVE-13512)

[Yongzhi Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001114#comment-16001114
 ] 

Yongzhi Chen commented on HIVE-16590:
-

PATCH 2 looks good.  +1 pending tests. 

> Make initializing dag names in SparkWork thread safe for parallel compilation 
> (HIVE-13512)
> --
>
> Key: HIVE-16590
> URL: https://issues.apache.org/jira/browse/HIVE-16590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16590.1.patch, HIVE-16590.2.patch
>
>
> In HIVE-13512 some modifications to {{TezWork}} were made to avoid generating 
> duplicate dag ids during parallel compilation. We should do the equivalent 
> for {{SparkWork}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)

[Zhiyuan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhiyuan Yang updated HIVE-14731:

Attachment: HIVE-14731.15.patch

Upload new patch that use new unpartitioned cross product edge implemented in 
TEZ-3708. 

Key features:
1. allow arbitrary parallelism by partitioning source output
2. estimate the workload based on #record
3. group by #cross-product-operation to evenly distribute the workload

CC [~hagleitn]

> Use Tez cartesian product edge in Hive (unpartitioned case only)
> 
>
> Key: HIVE-14731
> URL: https://issues.apache.org/jira/browse/HIVE-14731
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, 
> HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, 
> HIVE-14731.15.patch, HIVE-14731.1.patch, HIVE-14731.2.patch, 
> HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, 
> HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch
>
>
> Given cartesian product edge is available in Tez now (see TEZ-3230), let's 
> integrate it into Hive on Tez. This allows us to have more than one reducer 
> in cross product queries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16575) Support for 'UNIQUE' and 'NOT NULL' constraints

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001097#comment-16001097
 ] 

Hive QA commented on HIVE-16575:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866874/HIVE-16575.02.patch

{color:green}SUCCESS:{color} +1 due to 19 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10659 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=144)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5114/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5114/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5114/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866874 - PreCommit-HIVE-Build

> Support for 'UNIQUE' and 'NOT NULL' constraints
> ---
>
> Key: HIVE-16575
> URL: https://issues.apache.org/jira/browse/HIVE-16575
> Project: Hive
>  Issue Type: New Feature
>  Components: CBO, Logical Optimizer, Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16575.01.patch, HIVE-16575.02.patch, 
> HIVE-16575.patch
>
>
> Follow-up on HIVE-13076.
> This issue add support for SQL 'UNIQUE' and 'NOT NULL' constraints when we 
> create a table / alter a table 
> (https://www.postgresql.org/docs/9.6/static/sql-createtable.html).
> As with PK and FK constraints, currently we do not enforce them; thus, the 
> constraints need to use the DISABLE option, but they will be stored and can 
> be enabled for rewriting/optimization using RELY.
> This patch also adds support for inlining the constraints next to the column 
> type definition, i.e., 'column constraints'.
> Some examples of the extension to the syntax included in the patch:
> {code:sql}
> CREATE TABLE table3 (x string NOT NULL DISABLE, PRIMARY KEY (x) DISABLE, 
> CONSTRAINT fk1 FOREIGN KEY (x) REFERENCES table2(a) DISABLE); 
> CREATE TABLE table4 (x string CONSTRAINT nn4_1 NOT NULL DISABLE, y string 
> CONSTRAINT nn4_2 NOT NULL DISABLE, UNIQUE (x) DISABLE, CONSTRAINT fk2 FOREIGN 
> KEY (x) REFERENCES table2(a) DISABLE, 
> CONSTRAINT fk3 FOREIGN KEY (y) REFERENCES table2(a) DISABLE);
> CREATE TABLE table12 (a STRING CONSTRAINT nn12_1 NOT NULL DISABLE NORELY, b 
> STRING);
> CREATE TABLE table13 (a STRING NOT NULL DISABLE RELY, b STRING);
> CREATE TABLE table14 (a STRING CONSTRAINT nn14_1 NOT NULL DISABLE RELY, b 
> STRING);
> CREATE TABLE table15 (a STRING REFERENCES table4(x) DISABLE, b STRING);
> CREATE TABLE table16 (a STRING CONSTRAINT nn16_1 REFERENCES table4(x) DISABLE 
> RELY, b STRING);
> ALTER TABLE table16 CHANGE a a STRING REFERENCES table4(x) DISABLE NOVALIDATE;
> ALTER TABLE table12 CHANGE COLUMN b b STRING CONSTRAINT nn12_2 NOT NULL 
> DISABLE NOVALIDATE;
> ALTER TABLE table13 CHANGE b b STRING NOT NULL DISABLE NOVALIDATE;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16539) Add PTF tests for blobstores

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001093#comment-16001093
 ] 

Juan Rodríguez Hortalá commented on HIVE-16539:
---

Hi Sergio, thanks for taking a look to the code review. The problem with this 
patch is that I have added a ORC file, and patches created with `git diff 
--no-prefix` following 
https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-CreatingaPatch
 don't work ok with binary files. I sent an email to d...@hive.apache.org about 
this, and [~owen.omalley] suggested to use ` git format-patch --stdout`, which 
works ok when you apply the patch with `git apply` (HIVE-16539.3.patch), but 
from the message above it looks like Hive QA is not able to process that 
because it is trying to use `path -p0` or `path -p1` or `path -p2`. Can you 
please provide instructions to build a patch that includes a binary file? 

Thanks!

> Add PTF tests for blobstores
> 
>
> Key: HIVE-16539
> URL: https://issues.apache.org/jira/browse/HIVE-16539
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Juan Rodríguez Hortalá
>Assignee: Juan Rodríguez Hortalá
> Fix For: 2.2.0
>
> Attachments: HIVE-16539.2.patch, HIVE-16539.3.patch, HIVE-16539.patch
>
>
> This patch covers 6 tests that check the integration of several Partitioned 
> Table Functions with blobstores



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16469) Parquet timestamp table property is not always taken into account

[Barna Zsombor Klara (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001073#comment-16001073
 ] 

Barna Zsombor Klara commented on HIVE-16469:


Thanks for the review(s) [~spena], [~Ferd]. I've been struggling with this 
timestamp fix for quite some time, so the help is very much appreciated.
The patch is updated on the review board as well if you want to take another 
look.

> Parquet timestamp table property is not always taken into account
> -
>
> Key: HIVE-16469
> URL: https://issues.apache.org/jira/browse/HIVE-16469
> Project: Hive
>  Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-16469.01.patch, HIVE-16469.02.patch, 
> HIVE-16469.03.patch, HIVE-16469.04.patch, HIVE-16469.05.patch
>
>
> The parquet timestamp timezone property is currently copied over into the 
> JobConf in the FetchOperator, but this may be too late for some execution 
> paths.
> We should:
> 1 - copy the property over earlier
> 2 - set the default value on the JobConf if no property is set, and fail in 
> the ParquetRecordReader if the property is missing from the JobConf
> We should add extra validations for the cases when:
> - the property was not set by accident on the JobConf (unexpected execution 
> path)
> - an incorrect/invalid timezone id is being set on the table



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16559) Parquet schema evolution for partitioned tables may break if table and partition serdes differ

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-16559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001064#comment-16001064
 ] 

Sergio Peña commented on HIVE-16559:


I see that ORC doesn't support this only when ACID tables are found, and 
HIVE_SCHEMA_EVOLUTION is enabled. Otherwise, it is supported. Shouldn't we 
support this for PARQUET as well?

> Parquet schema evolution for partitioned tables may break if table and 
> partition serdes differ
> --
>
> Key: HIVE-16559
> URL: https://issues.apache.org/jira/browse/HIVE-16559
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Fix For: 3.0.0
>
> Attachments: HIVE-16559.01.patch
>
>
> Parquet schema evolution should make it possible to have partitions/tables 
>  backed by files with different schemas. Hive should match the table columns 
> with file columns based on the column name if possible.
> However if the serde for a table is missing columns from the serde of a 
> partition Hive fails to match the columns together.
> Steps to reproduce:
> {code}
> CREATE TABLE myparquettable_parted
> (
>   name string,
>   favnumber int,
>   favcolor string,
>   age int,
>   favpet string
> )
> PARTITIONED BY (day string)
> STORED AS PARQUET;
> INSERT OVERWRITE TABLE myparquettable_parted
> PARTITION(day='2017-04-04')
> SELECT
>'mary' as name,
>5 AS favnumber,
>'blue' AS favcolor,
>35 AS age,
>'dog' AS favpet;
> alter table myparquettable_parted
> REPLACE COLUMNS
> (
> favnumber int,
> age int
> );   

[jira] [Updated] (HIVE-16590) Make initializing dag names in SparkWork thread safe for parallel compilation (HIVE-13512)

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-16590:

Attachment: (was: HIVE-16590.2.patch)

> Make initializing dag names in SparkWork thread safe for parallel compilation 
> (HIVE-13512)
> --
>
> Key: HIVE-16590
> URL: https://issues.apache.org/jira/browse/HIVE-16590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16590.1.patch, HIVE-16590.2.patch
>
>
> In HIVE-13512 some modifications to {{TezWork}} were made to avoid generating 
> duplicate dag ids during parallel compilation. We should do the equivalent 
> for {{SparkWork}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16590) Make initializing dag names in SparkWork thread safe for parallel compilation (HIVE-13512)

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-16590:

Attachment: HIVE-16590.2.patch

> Make initializing dag names in SparkWork thread safe for parallel compilation 
> (HIVE-13512)
> --
>
> Key: HIVE-16590
> URL: https://issues.apache.org/jira/browse/HIVE-16590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16590.1.patch, HIVE-16590.2.patch
>
>
> In HIVE-13512 some modifications to {{TezWork}} were made to avoid generating 
> duplicate dag ids during parallel compilation. We should do the equivalent 
> for {{SparkWork}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16590) Make initializing dag names in SparkWork thread safe for parallel compilation (HIVE-13512)

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-16590:

Attachment: HIVE-16590.2.patch

> Make initializing dag names in SparkWork thread safe for parallel compilation 
> (HIVE-13512)
> --
>
> Key: HIVE-16590
> URL: https://issues.apache.org/jira/browse/HIVE-16590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16590.1.patch, HIVE-16590.2.patch
>
>
> In HIVE-13512 some modifications to {{TezWork}} were made to avoid generating 
> duplicate dag ids during parallel compilation. We should do the equivalent 
> for {{SparkWork}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16469) Parquet timestamp table property is not always taken into account

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-16469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001040#comment-16001040
 ] 

Sergio Peña commented on HIVE-16469:


The patch looks good.
+1

> Parquet timestamp table property is not always taken into account
> -
>
> Key: HIVE-16469
> URL: https://issues.apache.org/jira/browse/HIVE-16469
> Project: Hive
>  Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-16469.01.patch, HIVE-16469.02.patch, 
> HIVE-16469.03.patch, HIVE-16469.04.patch, HIVE-16469.05.patch
>
>
> The parquet timestamp timezone property is currently copied over into the 
> JobConf in the FetchOperator, but this may be too late for some execution 
> paths.
> We should:
> 1 - copy the property over earlier
> 2 - set the default value on the JobConf if no property is set, and fail in 
> the ParquetRecordReader if the property is missing from the JobConf
> We should add extra validations for the cases when:
> - the property was not set by accident on the JobConf (unexpected execution 
> path)
> - an incorrect/invalid timezone id is being set on the table



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15184) Add the possibility to separate recoverable and not recoverable errors in DelegationTokenStore

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-15184:
--
Summary: Add the possibility to separate recoverable and not recoverable 
errors in DelegationTokenStore  (was: Add the possibility to separate 
recoverable and not recoverable errors in DeletagionTokenStore)

> Add the possibility to separate recoverable and not recoverable errors in 
> DelegationTokenStore
> --
>
> Key: HIVE-15184
> URL: https://issues.apache.org/jira/browse/HIVE-15184
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Metastore
>Affects Versions: 1.3.0, 2.0.1, 2.1.0, 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-15184.patch
>
>
> After HIVE-15090 committed we discussed it with [~thejas] and agreed, that it 
> would be even better if the DelegateTokenStore implementation could decide if 
> an error is recoverable or not. Since the DelegationTokenStore is not 
> mentioned as a Hive API it is possible to change the interface, so we could 
> change the implementations shipped with Hive to use the new functionality, 
> and if someone uses it's own implementation of DelegationTokenStore, then he 
> should do the matching changes himself



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-10865) Beeline needs to support DELIMITER command

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001037#comment-16001037
 ] 

Sergio Peña commented on HIVE-10865:


[~stakiar] I left a few comments on the RB link.

> Beeline needs to support DELIMITER command
> --
>
> Key: HIVE-10865
> URL: https://issues.apache.org/jira/browse/HIVE-10865
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Sahil Takiar
> Attachments: HIVE-10865.1.patch, HIVE-10865.2.patch, 
> HIVE-10865.3.patch, HIVE-10865.4.patch, HIVE-10865.5.patch
>
>
> MySQL Client provides a DELIMITER command to set statement delimiter.
> Beeline needs to support a similar command to allow commands having 
> semi-colon as non-statement delimiter (as with MySQL stored procedures). This 
> is a follow-up jira for HIVE-10659



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14979) Removing stale Zookeeper locks at HiveServer2 initialization

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-14979:
--
Status: Open  (was: Patch Available)

> Removing stale Zookeeper locks at HiveServer2 initialization
> 
>
> Key: HIVE-14979
> URL: https://issues.apache.org/jira/browse/HIVE-14979
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-14979.3.patch, HIVE-14979.4.patch, 
> HIVE-14979.5.patch, HIVE-14979.patch
>
>
> HiveServer2 could use Zookeeper to store token that indicate that particular 
> tables are locked with the creation of persistent Zookeeper objects. 
> A problem can occur when a HiveServer2 instance creates a lock on a table and 
> the HiveServer2 instances crashes ("Out of Memory" for example) and the locks 
> are not released in Zookeeper. This lock will then remain until it is 
> manually cleared by an admin.
> There should be a way to remove stale locks at HiveServer2 initialization, 
> helping the admins life.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16269) enable incremental function dump to be loaded via repl load

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001003#comment-16001003
 ] 

Hive QA commented on HIVE-16269:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866872/HIVE-16269.1.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 10655 tests passed

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5113/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5113/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5113/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866872 - PreCommit-HIVE-Build

> enable incremental function dump to be loaded via repl load 
> 
>
> Key: HIVE-16269
> URL: https://issues.apache.org/jira/browse/HIVE-16269
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Attachments: HIVE-16269.1.patch
>
>
> depends if there is additional spec elements we put out as part of HIVE-16268



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16539) Add PTF tests for blobstores

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000944#comment-16000944
 ] 

Sergio Peña commented on HIVE-16539:


[~juanrh] Can you submit another patch to see if HiveQA works this time?
Btw, the tests are good.. +1

> Add PTF tests for blobstores
> 
>
> Key: HIVE-16539
> URL: https://issues.apache.org/jira/browse/HIVE-16539
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Juan Rodríguez Hortalá
>Assignee: Juan Rodríguez Hortalá
> Fix For: 2.2.0
>
> Attachments: HIVE-16539.2.patch, HIVE-16539.3.patch, HIVE-16539.patch
>
>
> This patch covers 6 tests that check the integration of several Partitioned 
> Table Functions with blobstores



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16469) Parquet timestamp table property is not always taken into account

[Barna Zsombor Klara (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000945#comment-16000945
 ] 

Barna Zsombor Klara commented on HIVE-16469:


The failing test is known snowflake:
- HIVE-15289

> Parquet timestamp table property is not always taken into account
> -
>
> Key: HIVE-16469
> URL: https://issues.apache.org/jira/browse/HIVE-16469
> Project: Hive
>  Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-16469.01.patch, HIVE-16469.02.patch, 
> HIVE-16469.03.patch, HIVE-16469.04.patch, HIVE-16469.05.patch
>
>
> The parquet timestamp timezone property is currently copied over into the 
> JobConf in the FetchOperator, but this may be too late for some execution 
> paths.
> We should:
> 1 - copy the property over earlier
> 2 - set the default value on the JobConf if no property is set, and fail in 
> the ParquetRecordReader if the property is missing from the JobConf
> We should add extra validations for the cases when:
> - the property was not set by accident on the JobConf (unexpected execution 
> path)
> - an incorrect/invalid timezone id is being set on the table



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16507) Hive Explain User-Level may print out "Vertex dependency in root stage" twice

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000938#comment-16000938
 ] 

Sahil Takiar commented on HIVE-16507:
-

[~pxiong] could you take a look? RB: https://reviews.apache.org/r/59055/

> Hive Explain User-Level may print out "Vertex dependency in root stage" twice
> -
>
> Key: HIVE-16507
> URL: https://issues.apache.org/jira/browse/HIVE-16507
> Project: Hive
>  Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16507.1.patch, HIVE-16507.2.patch
>
>
> User-level explain plans have a section titled {{Vertex dependency in root 
> stage}} - which (according to the name) prints out the dependencies between 
> all vertices that are in the root stage.
> This logic is controlled by {{DagJsonParser#print}} and it may print out 
> {{Vertex dependency in root stage}} twice.
> The logic in this method first extracts all stages and plans. It then 
> iterates over all the stages, and if the stage contains any edges, it prints 
> them out.
> If we want to be consistent with the statement {{Vertex dependency in root 
> stage}} then we should add a check to see if the stage we are processing 
> during the iteration is the root stage or not.
> Alternatively, we could print out the edges for each stage and change the 
> line from {{Vertex dependency in root stage}} to {{Vertex dependency in 
> [stage-id]}}
> I'm not sure if its possible for Hive-on-Tez to create a plan with a non-root 
> stage that contains edges, but it is possible for Hive-on-Spark (support 
> added for HoS in HIVE-11133).
> Example for HoS:
> {code}
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.spark.dynamic.partition.pruning=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.strict.checks.cartesian.product=false;
> set hive.spark.explain.user=true;
> set hive.spark.dynamic.partition.pruning=true;
> EXPLAIN select count(*) from srcpart where srcpart.ds in (select 
> max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart);
> {code}
> Prints
> {code}
> Plan optimized by CBO.
> Vertex dependency in root stage
> Reducer 10 <- Map 9 (GROUP)
> Reducer 11 <- Reducer 10 (GROUP), Reducer 13 (GROUP)
> Reducer 13 <- Map 12 (GROUP)
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (PARTITION-LEVEL SORT), Reducer 6 (PARTITION-LEVEL SORT)
> Reducer 3 <- Reducer 2 (GROUP)
> Reducer 5 <- Map 4 (GROUP)
> Reducer 6 <- Reducer 5 (GROUP), Reducer 8 (GROUP)
> Reducer 8 <- Map 7 (GROUP)
> Stage-0
>   Fetch Operator
> limit:-1
> Stage-1
>   Reducer 3
>   File Output Operator [FS_34]
> Group By Operator [GBY_32] (rows=1 width=8)
>   Output:["_col0"],aggregations:["count(VALUE._col0)"]
> <-Reducer 2 [GROUP]
>   GROUP [RS_31]
> Group By Operator [GBY_30] (rows=1 width=8)
>   Output:["_col0"],aggregations:["count()"]
>   Join Operator [JOIN_28] (rows=2200 width=10)
> condition 
> map:[{"":"{\"type\":\"Inner\",\"left\":0,\"right\":1}"}],keys:{"0":"_col0","1":"_col0"}
>   <-Map 1 [PARTITION-LEVEL SORT]
> PARTITION-LEVEL SORT [RS_26]
>   PartitionCols:_col0
>   Select Operator [SEL_2] (rows=2000 width=10)
> Output:["_col0"]
> TableScan [TS_0] (rows=2000 width=10)
>   default@srcpart,srcpart,Tbl:COMPLETE,Col:NONE
>   <-Reducer 6 [PARTITION-LEVEL SORT]
> PARTITION-LEVEL SORT [RS_27]
>   PartitionCols:_col0
>   Group By Operator [GBY_24] (rows=1 width=184)
> Output:["_col0"],keys:KEY._col0
>   <-Reducer 5 [GROUP]
> GROUP [RS_23]
>   PartitionCols:_col0
>   Group By Operator [GBY_22] (rows=2 width=184)
> Output:["_col0"],keys:_col0
> Filter Operator [FIL_9] (rows=1 width=184)
>   predicate:_col0 is not null
>   Group By Operator [GBY_7] (rows=1 width=184)
> Output:["_col0"],aggregations:["max(VALUE._col0)"]
>   <-Map 4 [GROUP]
> GROUP [RS_6]
>   Group By Operator [GBY_5] (rows=1 width=184)
> Output:["_col0"],aggregations:["max(ds)"]
> Select Operator [SEL_4] (rows=2000 width=10)
>   Output:["ds"]
>   TableScan [TS_3] (rows=2000 width=10)
> 

[jira] [Commented] (HIVE-16590) Make initializing dag names in SparkWork thread safe for parallel compilation (HIVE-13512)

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000936#comment-16000936
 ] 

Sahil Takiar commented on HIVE-16590:
-

[~aihuaxu] could you take a look? Quick fix for parallel compilation for HoS. 
https://reviews.apache.org/r/59054/

CC: [~ychena]

> Make initializing dag names in SparkWork thread safe for parallel compilation 
> (HIVE-13512)
> --
>
> Key: HIVE-16590
> URL: https://issues.apache.org/jira/browse/HIVE-16590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16590.1.patch
>
>
> In HIVE-13512 some modifications to {{TezWork}} were made to avoid generating 
> duplicate dag ids during parallel compilation. We should do the equivalent 
> for {{SparkWork}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000924#comment-16000924
 ] 

Sahil Takiar commented on HIVE-16484:
-

[~xuefuz] could you take a look? https://reviews.apache.org/r/58684/

> Investigate SparkLauncher for HoS as alternative to bin/spark-submit
> 
>
> Key: HIVE-16484
> URL: https://issues.apache.org/jira/browse/HIVE-16484
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16484.1.patch, HIVE-16484.2.patch, 
> HIVE-16484.3.patch, HIVE-16484.4.patch, HIVE-16484.5.patch, 
> HIVE-16484.6.patch, HIVE-16484.7.patch
>
>
> The {{SparkClientImpl#startDriver}} currently looks for the {{SPARK_HOME}} 
> directory and invokes the {{bin/spark-submit}} script, which spawns a 
> separate process to run the Spark application.
> {{SparkLauncher}} was added in SPARK-4924 and is a programatic way to launch 
> Spark applications.
> I see a few advantages:
> * No need to spawn a separate process to launch a HoS --> lower startup time
> * Simplifies the code in {{SparkClientImpl}} --> easier to debug
> * {{SparkLauncher#startApplication}} returns a {{SparkAppHandle}} which 
> contains some useful utilities for querying the state of the Spark job
> ** It also allows the launcher to specify a list of job listeners



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-10865) Beeline needs to support DELIMITER command

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000916#comment-16000916
 ] 

Sahil Takiar commented on HIVE-10865:
-

[~spena] could you take a look?

> Beeline needs to support DELIMITER command
> --
>
> Key: HIVE-10865
> URL: https://issues.apache.org/jira/browse/HIVE-10865
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Sahil Takiar
> Attachments: HIVE-10865.1.patch, HIVE-10865.2.patch, 
> HIVE-10865.3.patch, HIVE-10865.4.patch, HIVE-10865.5.patch
>
>
> MySQL Client provides a DELIMITER command to set statement delimiter.
> Beeline needs to support a similar command to allow commands having 
> semi-colon as non-statement delimiter (as with MySQL stored procedures). This 
> is a follow-up jira for HIVE-10659



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16474) Upgrade Druid version to 0.10

[Nishant Bangarwa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000908#comment-16000908
 ] 

Nishant Bangarwa commented on HIVE-16474:
-

review link - https://reviews.apache.org/r/59052

> Upgrade Druid version to 0.10
> -
>
> Key: HIVE-16474
> URL: https://issues.apache.org/jira/browse/HIVE-16474
> Project: Hive
>  Issue Type: Task
>  Components: Druid integration
>Reporter: Ashutosh Chauhan
>Assignee: Nishant Bangarwa
> Attachments: HIVE-16474.01.patch, HIVE-16474.patch
>
>
> Druid 0.10 is out. We shall upgrade to it to take advantage of improvements 
> it brings.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16588) Resource leak by druid http client

[slim bouguerra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-16588:
--
Summary: Resource leak by druid http client  (was: Ressource leak by druid 
http client)

> Resource leak by druid http client
> --
>
> Key: HIVE-16588
> URL: https://issues.apache.org/jira/browse/HIVE-16588
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>  Labels: druid
> Fix For: 3.0.0
>
> Attachments: 16588.1.patch, 16588.patch
>
>
> Current implementation of druid storage handler does leak some resources if 
> the creation of the http client fails due to too many files exception.
> The reason this is leaking is the fact the cleaning hook is registered after 
> the client starts.
> In order to fix this will extract the creation of the HTTP client to become 
> static and reusable instead of per query creation.
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-14412:
--
Attachment: HIVE-14412.12.patch

Thanks [~xuefuz] and [~cartershanklin] for the review and testing. Update patch 
v12 to address your comments.
bq. there is no way to convert to a target time zone
We have the {{from_utc_timestamp}} UDF for this purpose. But since we don't 
store the time zone, I'm afraid it can't use the timestamptz type. You can find 
more discussions in HIVE-16418

> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.10.patch, HIVE-14412.11.patch, 
> HIVE-14412.12.patch, HIVE-14412.1.patch, HIVE-14412.2.patch, 
> HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, 
> HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch, HIVE-14412.9.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by 
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or 
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16588) Ressource leak by druid http client

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-16588:
---
Resolution: Fixed
  Assignee: slim bouguerra  (was: Nishant Bangarwa)
Status: Resolved  (was: Patch Available)

Pushed to master, thanks [~bslim]!

> Ressource leak by druid http client
> ---
>
> Key: HIVE-16588
> URL: https://issues.apache.org/jira/browse/HIVE-16588
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>  Labels: druid
> Fix For: 3.0.0
>
> Attachments: 16588.1.patch, 16588.patch
>
>
> Current implementation of druid storage handler does leak some resources if 
> the creation of the http client fails due to too many files exception.
> The reason this is leaking is the fact the cleaning hook is registered after 
> the client starts.
> In order to fix this will extract the creation of the HTTP client to become 
> static and reusable instead of per query creation.
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)