[jira] [Commented] (HIVE-8615) beeline csv,tsv outputformat needs backward compatibility mode

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186485#comment-14186485
 ] 

Vaibhav Gumashta commented on HIVE-8615:


+1. Thanks for adding MiniHS2 to the test! 

> beeline csv,tsv outputformat needs backward compatibility mode
> --
>
> Key: HIVE-8615
> URL: https://issues.apache.org/jira/browse/HIVE-8615
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 0.14.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8615.1.patch, HIVE-8615.2.patch
>
>
> Changes in HIVE-7390 break backward compatibility for beeline csv and tsv 
> formats.
> This can cause problems for users upgrading to hive 0.14, if they have code 
> for parsing the old output format. Instead of removing the old format in this 
> release, we should consider it deprecated and support it in a few releases 
> before removing it completely.
> Incompatible Changes in the tsv and csv formats-
> - Old format quoted every field. New format quotes only fields that contain a 
> delimiter or the quoteing char.
> - Old format quoted using single quotes, new format quotes using double quotes
> - Old format didn't escape quotes in a field (a bug). New format does escape 
> the quotes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8579) Guaranteed NPE in DDLSemanticAnalyzer

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186490#comment-14186490
 ] 

Vaibhav Gumashta commented on HIVE-8579:


+1

> Guaranteed NPE in DDLSemanticAnalyzer
> -
>
> Key: HIVE-8579
> URL: https://issues.apache.org/jira/browse/HIVE-8579
> Project: Hive
>  Issue Type: Bug
>Reporter: Lars Francke
>Assignee: Jason Dere
> Attachments: HIVE-8579.1.patch, HIVE-8579.1.patch
>
>
> This was added by [~jdere] in HIVE-8411. I don't fully understand the code 
> (i.e. what it means when desc is null) but I'm sure, Jason, you can fix it 
> without much trouble?
> {code}
> if (desc == null || 
> !AlterTableDesc.doesAlterTableTypeSupportPartialPartitionSpec(desc.getOp())) {
>   throw new SemanticException( 
> ErrorMsg.ALTER_TABLE_TYPE_PARTIAL_PARTITION_SPEC_NO_SUPPORTED, 
> desc.getOp().name());
> } else if (!conf.getBoolVar(HiveConf.ConfVars.DYNAMICPARTITIONING)) {
>   throw new SemanticException(ErrorMsg.DYNAMIC_PARTITION_DISABLED);
> }
> {code}
> You check for whether {{desc}} is null but then use it to do {{desc.getOp()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8544) Beeline SeparatedValuesOutputFormat (csv/tsv/dsv) wraps the results with extra double quotes

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-8544:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Beeline SeparatedValuesOutputFormat (csv/tsv/dsv) wraps the results with 
> extra double quotes  
> --
>
> Key: HIVE-8544
> URL: https://issues.apache.org/jira/browse/HIVE-8544
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, JDBC
>Affects Versions: 0.14.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-8544.1.patch
>
>
> Noticed additional double quotes in select *  output  
> while running it through HiveServer2/Beeline
> Actual
> {noformat}
> [18,19]   "{""col1"":""alice brown"",""col2"":18}""{""alice 
> brown"":18}"
> [18,19]   "{""col1"":""alice garcia"",""col2"":18}"   "{""alice 
> garcia"":18}"
> [18,19]   "{""col1"":""alice garcia"",""col2"":18}"   "{""alice 
> garcia"":18}"
> [18,19]   "{""col1"":""alice thompson"",""col2"":18}" "{""alice 
> thompson"":18}"
> [18,19]   "{""col1"":""alice xylophone"",""col2"":18}""{""alice 
> xylophone"":18}"
> [18,19]   "{""col1"":""bob hernandez"",""col2"":18}"  "{""bob 
> hernandez"":18}"
> [18,19]   "{""col1"":""bob ichabod"",""col2"":18}""{""bob 
> ichabod"":18}"
> {noformat}
> Expected
> {noformat}
> [18,19]   {"col1":"alice brown","col2":18}{"alice brown":18}
> [18,19]   {"col1":"alice garcia","col2":18}   {"alice garcia":18}
> [18,19]   {"col1":"alice garcia","col2":18}   {"alice garcia":18}
> [18,19]   {"col1":"alice thompson","col2":18} {"alice thompson":18}
> [18,19]   {"col1":"alice xylophone","col2":18}{"alice xylophone":18}
> [18,19]   {"col1":"bob hernandez","col2":18}  {"bob hernandez":18}
> [18,19]   {"col1":"bob ichabod","col2":18}{"bob ichabod":18}
> {noformat}
> Run these queries through HiveServer2 to reproduce the issue:
> {noformat}
> create table insert_9 (a array, b struct, c 
> map);
>  insert overwrite table insert_9 select array(age, age+1), struct(name, age), 
> map(name, age) from studenttab10k;
> select * from insert_9;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8544) Beeline SeparatedValuesOutputFormat (csv/tsv/dsv) wraps the results with extra double quotes

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186495#comment-14186495
 ] 

Vaibhav Gumashta commented on HIVE-8544:


Resolving this as HIVE-8615 supersedes this.

> Beeline SeparatedValuesOutputFormat (csv/tsv/dsv) wraps the results with 
> extra double quotes  
> --
>
> Key: HIVE-8544
> URL: https://issues.apache.org/jira/browse/HIVE-8544
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, JDBC
>Affects Versions: 0.14.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-8544.1.patch
>
>
> Noticed additional double quotes in select *  output  
> while running it through HiveServer2/Beeline
> Actual
> {noformat}
> [18,19]   "{""col1"":""alice brown"",""col2"":18}""{""alice 
> brown"":18}"
> [18,19]   "{""col1"":""alice garcia"",""col2"":18}"   "{""alice 
> garcia"":18}"
> [18,19]   "{""col1"":""alice garcia"",""col2"":18}"   "{""alice 
> garcia"":18}"
> [18,19]   "{""col1"":""alice thompson"",""col2"":18}" "{""alice 
> thompson"":18}"
> [18,19]   "{""col1"":""alice xylophone"",""col2"":18}""{""alice 
> xylophone"":18}"
> [18,19]   "{""col1"":""bob hernandez"",""col2"":18}"  "{""bob 
> hernandez"":18}"
> [18,19]   "{""col1"":""bob ichabod"",""col2"":18}""{""bob 
> ichabod"":18}"
> {noformat}
> Expected
> {noformat}
> [18,19]   {"col1":"alice brown","col2":18}{"alice brown":18}
> [18,19]   {"col1":"alice garcia","col2":18}   {"alice garcia":18}
> [18,19]   {"col1":"alice garcia","col2":18}   {"alice garcia":18}
> [18,19]   {"col1":"alice thompson","col2":18} {"alice thompson":18}
> [18,19]   {"col1":"alice xylophone","col2":18}{"alice xylophone":18}
> [18,19]   {"col1":"bob hernandez","col2":18}  {"bob hernandez":18}
> [18,19]   {"col1":"bob ichabod","col2":18}{"bob ichabod":18}
> {noformat}
> Run these queries through HiveServer2 to reproduce the issue:
> {noformat}
> create table insert_9 (a array, b struct, c 
> map);
>  insert overwrite table insert_9 select array(age, age+1), struct(name, age), 
> map(name, age) from studenttab10k;
> select * from insert_9;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HIVE-8544) Beeline SeparatedValuesOutputFormat (csv/tsv/dsv) wraps the results with extra double quotes

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair reopened HIVE-8544:
-

> Beeline SeparatedValuesOutputFormat (csv/tsv/dsv) wraps the results with 
> extra double quotes  
> --
>
> Key: HIVE-8544
> URL: https://issues.apache.org/jira/browse/HIVE-8544
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, JDBC
>Affects Versions: 0.14.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-8544.1.patch
>
>
> Noticed additional double quotes in select *  output  
> while running it through HiveServer2/Beeline
> Actual
> {noformat}
> [18,19]   "{""col1"":""alice brown"",""col2"":18}""{""alice 
> brown"":18}"
> [18,19]   "{""col1"":""alice garcia"",""col2"":18}"   "{""alice 
> garcia"":18}"
> [18,19]   "{""col1"":""alice garcia"",""col2"":18}"   "{""alice 
> garcia"":18}"
> [18,19]   "{""col1"":""alice thompson"",""col2"":18}" "{""alice 
> thompson"":18}"
> [18,19]   "{""col1"":""alice xylophone"",""col2"":18}""{""alice 
> xylophone"":18}"
> [18,19]   "{""col1"":""bob hernandez"",""col2"":18}"  "{""bob 
> hernandez"":18}"
> [18,19]   "{""col1"":""bob ichabod"",""col2"":18}""{""bob 
> ichabod"":18}"
> {noformat}
> Expected
> {noformat}
> [18,19]   {"col1":"alice brown","col2":18}{"alice brown":18}
> [18,19]   {"col1":"alice garcia","col2":18}   {"alice garcia":18}
> [18,19]   {"col1":"alice garcia","col2":18}   {"alice garcia":18}
> [18,19]   {"col1":"alice thompson","col2":18} {"alice thompson":18}
> [18,19]   {"col1":"alice xylophone","col2":18}{"alice xylophone":18}
> [18,19]   {"col1":"bob hernandez","col2":18}  {"bob hernandez":18}
> [18,19]   {"col1":"bob ichabod","col2":18}{"bob ichabod":18}
> {noformat}
> Run these queries through HiveServer2 to reproduce the issue:
> {noformat}
> create table insert_9 (a array, b struct, c 
> map);
>  insert overwrite table insert_9 select array(age, age+1), struct(name, age), 
> map(name, age) from studenttab10k;
> select * from insert_9;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7111) Extend join transitivity PPD to non-column expressions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186500#comment-14186500
 ] 

Hive QA commented on HIVE-7111:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677492/HIVE-7111.4.patch.txt

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 6579 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_exists_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sum_expr_with_order
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1494/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1494/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1494/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677492 - PreCommit-HIVE-TRUNK-Build

> Extend join transitivity PPD to non-column expressions
> --
>
> Key: HIVE-7111
> URL: https://issues.apache.org/jira/browse/HIVE-7111
> Project: Hive
>  Issue Type: Task
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-7111.1.patch.txt, HIVE-7111.2.patch.txt, 
> HIVE-7111.3.patch.txt, HIVE-7111.4.patch.txt
>
>
> Join transitive in PPD only supports column expressions, but it's possible to 
> extend this to generic expressions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-8544) Beeline SeparatedValuesOutputFormat (csv/tsv/dsv) wraps the results with extra double quotes

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair resolved HIVE-8544.
-
Resolution: Done

> Beeline SeparatedValuesOutputFormat (csv/tsv/dsv) wraps the results with 
> extra double quotes  
> --
>
> Key: HIVE-8544
> URL: https://issues.apache.org/jira/browse/HIVE-8544
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, JDBC
>Affects Versions: 0.14.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-8544.1.patch
>
>
> Noticed additional double quotes in select *  output  
> while running it through HiveServer2/Beeline
> Actual
> {noformat}
> [18,19]   "{""col1"":""alice brown"",""col2"":18}""{""alice 
> brown"":18}"
> [18,19]   "{""col1"":""alice garcia"",""col2"":18}"   "{""alice 
> garcia"":18}"
> [18,19]   "{""col1"":""alice garcia"",""col2"":18}"   "{""alice 
> garcia"":18}"
> [18,19]   "{""col1"":""alice thompson"",""col2"":18}" "{""alice 
> thompson"":18}"
> [18,19]   "{""col1"":""alice xylophone"",""col2"":18}""{""alice 
> xylophone"":18}"
> [18,19]   "{""col1"":""bob hernandez"",""col2"":18}"  "{""bob 
> hernandez"":18}"
> [18,19]   "{""col1"":""bob ichabod"",""col2"":18}""{""bob 
> ichabod"":18}"
> {noformat}
> Expected
> {noformat}
> [18,19]   {"col1":"alice brown","col2":18}{"alice brown":18}
> [18,19]   {"col1":"alice garcia","col2":18}   {"alice garcia":18}
> [18,19]   {"col1":"alice garcia","col2":18}   {"alice garcia":18}
> [18,19]   {"col1":"alice thompson","col2":18} {"alice thompson":18}
> [18,19]   {"col1":"alice xylophone","col2":18}{"alice xylophone":18}
> [18,19]   {"col1":"bob hernandez","col2":18}  {"bob hernandez":18}
> [18,19]   {"col1":"bob ichabod","col2":18}{"bob ichabod":18}
> {noformat}
> Run these queries through HiveServer2 to reproduce the issue:
> {noformat}
> create table insert_9 (a array, b struct, c 
> map);
>  insert overwrite table insert_9 select array(age, age+1), struct(name, age), 
> map(name, age) from studenttab10k;
> select * from insert_9;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8557) automatically setup ZooKeeperTokenStore to use kerberos authentication when kerberos is enabled

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186501#comment-14186501
 ] 

Lefty Leverenz commented on HIVE-8557:
--

+1 for doc issues

> automatically setup ZooKeeperTokenStore to use kerberos authentication when 
> kerberos is enabled
> ---
>
> Key: HIVE-8557
> URL: https://issues.apache.org/jira/browse/HIVE-8557
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.14.0
>
> Attachments: HIVE-8557.1.patch, HIVE-8557.2.patch
>
>
> ZooKeeperTokenStore does not automatically setup the zookeeper client to use 
> kerberos authentication to talk to zookeeper, it requires additional 
> configuration.
> If kerberos is enabled in the configuration, it makes sense to do this 
> configuration for zookeeper client automatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8630) CombineHiveInputFormat splits subclass of FileInputFormat which returns false for isSplitable()

[Navis (JIRA)]

Navis created HIVE-8630:
---

 Summary: CombineHiveInputFormat splits subclass of FileInputFormat 
which returns false for isSplitable()
 Key: HIVE-8630
 URL: https://issues.apache.org/jira/browse/HIVE-8630
 Project: Hive
  Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Minor


CombineFileInputFormat in hadoop does not check FileInputFormat.isSplitable and 
tries to split input file. Happened on custom SerDe which handles multi-lined 
json (return false always for isSplitable).

Can be bypass by setting it HiveInputFormat for the query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8630) CombineHiveInputFormat splits subclass of FileInputFormat which returns false for isSplitable()

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-8630:

Status: Patch Available  (was: Open)

> CombineHiveInputFormat splits subclass of FileInputFormat which returns false 
> for isSplitable()
> ---
>
> Key: HIVE-8630
> URL: https://issues.apache.org/jira/browse/HIVE-8630
> Project: Hive
>  Issue Type: Bug
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-8630.1.patch.txt
>
>
> CombineFileInputFormat in hadoop does not check FileInputFormat.isSplitable 
> and tries to split input file. Happened on custom SerDe which handles 
> multi-lined json (return false always for isSplitable).
> Can be bypass by setting it HiveInputFormat for the query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8630) CombineHiveInputFormat splits subclass of FileInputFormat which returns false for isSplitable()

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-8630:

Attachment: HIVE-8630.1.patch.txt

> CombineHiveInputFormat splits subclass of FileInputFormat which returns false 
> for isSplitable()
> ---
>
> Key: HIVE-8630
> URL: https://issues.apache.org/jira/browse/HIVE-8630
> Project: Hive
>  Issue Type: Bug
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-8630.1.patch.txt
>
>
> CombineFileInputFormat in hadoop does not check FileInputFormat.isSplitable 
> and tries to split input file. Happened on custom SerDe which handles 
> multi-lined json (return false always for isSplitable).
> Can be bypass by setting it HiveInputFormat for the query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7111) Extend join transitivity PPD to non-column expressions

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-7111:

Attachment: HIVE-7111.2.patch.txt

Fixed others. Cannot sure the change of vector_mapjoin_reduce.q is right or not.

> Extend join transitivity PPD to non-column expressions
> --
>
> Key: HIVE-7111
> URL: https://issues.apache.org/jira/browse/HIVE-7111
> Project: Hive
>  Issue Type: Task
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-7111.1.patch.txt, HIVE-7111.2.patch.txt, 
> HIVE-7111.2.patch.txt, HIVE-7111.3.patch.txt, HIVE-7111.4.patch.txt
>
>
> Join transitive in PPD only supports column expressions, but it's possible to 
> extend this to generic expressions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8495) Add progress bar for Hive on Tez queries

[Mostafa Mokhtar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mostafa Mokhtar updated HIVE-8495:
--
Attachment: HIVE-8495.2.patch

> Add progress bar for Hive on Tez queries
> 
>
> Key: HIVE-8495
> URL: https://issues.apache.org/jira/browse/HIVE-8495
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Mostafa Mokhtar
> Fix For: 0.14.0
>
> Attachments: HIVE-8495.1.patch, HIVE-8495.2.patch, Screen Shot 
> 2014-10-16 at 9.35.26 PM.png, Screen Shot 2014-10-22 at 11.48.57 AM.png
>
>
> Build a "Progress bar" to provide overall progress on running tasks.
> Progress is calculated as : 
>  (Completed tasks) / (Total number of tasks)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186553#comment-14186553
 ] 

Hive QA commented on HIVE-5718:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677495/HIVE-5718.11.patch.txt

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 6578 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch_threshold
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_select_dummy_source
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1495/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1495/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1495/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677495 - PreCommit-HIVE-TRUNK-Build

> Support direct fetch for lateral views, sub queries, etc.
> -
>
> Key: HIVE-5718
> URL: https://issues.apache.org/jira/browse/HIVE-5718
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13857.1.patch, D13857.2.patch, D13857.3.patch, 
> HIVE-5718.10.patch.txt, HIVE-5718.11.patch.txt, HIVE-5718.4.patch.txt, 
> HIVE-5718.5.patch.txt, HIVE-5718.6.patch.txt, HIVE-5718.7.patch.txt, 
> HIVE-5718.8.patch.txt, HIVE-5718.9.patch.txt
>
>
> Extend HIVE-2925 with LV and SubQ.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8591) hive.default.fileformat should accept all formats described by StorageFormatDescriptor

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186572#comment-14186572
 ] 

Lefty Leverenz commented on HIVE-8591:
--

+1 for the typo fix 

> hive.default.fileformat should accept all formats described by 
> StorageFormatDescriptor
> --
>
> Key: HIVE-8591
> URL: https://issues.apache.org/jira/browse/HIVE-8591
> Project: Hive
>  Issue Type: Task
>  Components: Configuration
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-8591.1.patch.txt, HIVE-8591.2.patch.txt
>
>
> NO PRECOMMIT TESTS
> FileFormats are described by StorageFormatDescriptor, which is added in 
> HIVE-5976. Validator for FileFormats should reflect that also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8329) Enable postgres for storing stats

[Damien Carol (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186605#comment-14186605
 ] 

Damien Carol commented on HIVE-8329:


[~hagleitn] I don't know. I'm still working on it.

> Enable postgres for storing stats
> -
>
> Key: HIVE-8329
> URL: https://issues.apache.org/jira/browse/HIVE-8329
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.14.0
>Reporter: Damien Carol
>Assignee: Damien Carol
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8329.1.patch, HIVE-8329.1.patch, HIVE-8329.1.patch
>
>
> Simple patch to enable postgresql as JDBC publisher for statistics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8329) Enable postgres for storing stats

[Damien Carol (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186608#comment-14186608
 ] 

Damien Carol commented on HIVE-8329:


Tests pass locally with :
{code}
 mvn -B -o test -Phadoop-2 -Dtest=TestCliDriver
{code}

> Enable postgres for storing stats
> -
>
> Key: HIVE-8329
> URL: https://issues.apache.org/jira/browse/HIVE-8329
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.14.0
>Reporter: Damien Carol
>Assignee: Damien Carol
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8329.1.patch, HIVE-8329.1.patch, HIVE-8329.1.patch
>
>
> Simple patch to enable postgresql as JDBC publisher for statistics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8628) NPE in case of shuffle join in tez

[Damien Carol (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damien Carol updated HIVE-8628:
---
Description: 
test throws NullPointerException:
{noformat}
Vertex failed, vertexName=Reducer 2, vertexId=vertex_1413774081318_0803_5_03, 
diagnostics=[Task failed, taskId=task_1413774081318_0803_5_03_00, 
diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running 
task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error 
while closing operators: null
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing 
operators: null
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:218)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:178)
... 13 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:368)
at 
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:310)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:582)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:200)
... 14 more
], TaskAttempt 1 failed, info=[Error: Failure while running 
task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error 
while closing operators: null
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing 
operators: null
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:218)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:178)
... 13 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:368)
at 
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:310)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:582)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:200)
... 14 more
], TaskAttempt 2 failed, info=[Error: Failure while running 
task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error 
while closing

[jira] [Commented] (HIVE-8628) NPE in case of shuffle join in tez

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186623#comment-14186623
 ] 

Hive QA commented on HIVE-8628:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677499/HIVE-8628.1.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6578 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1496/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1496/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1496/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677499 - PreCommit-HIVE-TRUNK-Build

> NPE in case of shuffle join in tez
> --
>
> Key: HIVE-8628
> URL: https://issues.apache.org/jira/browse/HIVE-8628
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8628.1.patch
>
>
> test throws NullPointerException:
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1413774081318_0803_5_03, 
> diagnostics=[Task failed, taskId=task_1413774081318_0803_5_03_00, 
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running 
> task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime 
> Error while closing operators: null
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing 
> operators: null
>   at 
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:218)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:178)
>   ... 13 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:368)
>   at 
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:310)
>   at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:582)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:200)
>   ... 14 more
> ], TaskAttempt 1 failed, info=[Error: Failure while running 
> task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime 
> Error while closing operators: null
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.jav

[jira] [Commented] (HIVE-8392) HiveServer2 Operation.close fails on windows

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186654#comment-14186654
 ] 

Vaibhav Gumashta commented on HIVE-8392:


[~hagleitn] I'll upload one in next couple of days. 

> HiveServer2 Operation.close fails on windows
> 
>
> Key: HIVE-8392
> URL: https://issues.apache.org/jira/browse/HIVE-8392
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.14.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Critical
> Fix For: 0.14.0
>
>
> {code}
> java.io.IOException: Unable to delete file: 
> C:\Users\HADOOP~1.ONP\AppData\Local\Temp\hadoop\operation_logs\ac7d4f51-d9b9-4189-b248-6e8d5e3102af\4b1f1153-5c0c-4741-8f53-1f1b6ed9b190
>   at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
>   at 
> org.apache.hive.service.cli.operation.OperationLog$LogFile.remove(OperationLog.java:131)
>   at 
> org.apache.hive.service.cli.operation.OperationLog.close(OperationLog.java:95)
>   at 
> org.apache.hive.service.cli.operation.Operation.cleanupOperationLog(Operation.java:268)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.close(SQLOperation.java:307)
>   at 
> org.apache.hive.service.cli.operation.OperationManager.closeOperation(OperationManager.java:215)
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.closeOperation(HiveSessionImpl.java:640)
>   at 
> org.apache.hive.service.cli.CLIService.closeOperation(CLIService.java:392)
>   at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.CloseOperation(ThriftCLIService.java:573)
>   at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1513)
>   at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1498)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> On windows, close needs to be called before delete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8614) Upgrade hive to use tez version 0.5.2-SNAPSHOT

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186670#comment-14186670
 ] 

Hive QA commented on HIVE-8614:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677506/HIVE-8614.3.patch

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6580 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1497/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1497/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1497/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677506 - PreCommit-HIVE-TRUNK-Build

> Upgrade hive to use tez version 0.5.2-SNAPSHOT
> --
>
> Key: HIVE-8614
> URL: https://issues.apache.org/jira/browse/HIVE-8614
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8614.1.patch, HIVE-8614.2.patch, HIVE-8614.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8629) Streaming / ACID : hive cli session creation takes too long and times out if execution engine is tez

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186702#comment-14186702
 ] 

Hive QA commented on HIVE-8629:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677512/HIVE-8629.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6579 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1498/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1498/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1498/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677512 - PreCommit-HIVE-TRUNK-Build

> Streaming / ACID : hive cli session creation takes too long and times out if 
> execution engine is tez
> 
>
> Key: HIVE-8629
> URL: https://issues.apache.org/jira/browse/HIVE-8629
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.14.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>  Labels: ACID, Streaming
> Attachments: HIVE-8629.patch
>
>
> When creating a hive session to run basic "alter table create partition"  
> queries, the session creation takes too long (more than 5 sec)  if the hive 
> execution engine is set to tez.
> Since the streaming clients dont care about Tez , it can explicitly override 
> the setting to mr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8461) Make Vectorized Decimal query results match Non-Vectorized query results with respect to trailing zeroes... .0000

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-8461:
---
Status: Patch Available  (was: Open)

> Make Vectorized Decimal query results match Non-Vectorized query results with 
> respect to trailing zeroes... .
> -
>
> Key: HIVE-8461
> URL: https://issues.apache.org/jira/browse/HIVE-8461
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8461.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8461) Make Vectorized Decimal query results match Non-Vectorized query results with respect to trailing zeroes... .0000

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-8461:
---
Attachment: HIVE-8461.01.patch

> Make Vectorized Decimal query results match Non-Vectorized query results with 
> respect to trailing zeroes... .
> -
>
> Key: HIVE-8461
> URL: https://issues.apache.org/jira/browse/HIVE-8461
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8461.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8557) automatically setup ZooKeeperTokenStore to use kerberos authentication when kerberos is enabled

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186744#comment-14186744
 ] 

Hive QA commented on HIVE-8557:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677539/HIVE-8557.2.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6579 tests executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1499/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1499/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1499/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677539 - PreCommit-HIVE-TRUNK-Build

> automatically setup ZooKeeperTokenStore to use kerberos authentication when 
> kerberos is enabled
> ---
>
> Key: HIVE-8557
> URL: https://issues.apache.org/jira/browse/HIVE-8557
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.14.0
>
> Attachments: HIVE-8557.1.patch, HIVE-8557.2.patch
>
>
> ZooKeeperTokenStore does not automatically setup the zookeeper client to use 
> kerberos authentication to talk to zookeeper, it requires additional 
> configuration.
> If kerberos is enabled in the configuration, it makes sense to do this 
> configuration for zookeeper client automatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8630) CombineHiveInputFormat splits subclass of FileInputFormat which returns false for isSplitable()

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186788#comment-14186788
 ] 

Hive QA commented on HIVE-8630:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677546/HIVE-8630.1.patch.txt

{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 6579 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_neg_float
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_const_type
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_partition_metadataonly
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partInit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_date2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_varchar1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_count
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_percentile
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testCombinationInputFormat
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testCombinationInputFormatWithAcid
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1500/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1500/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1500/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677546 - PreCommit-HIVE-TRUNK-Build

> CombineHiveInputFormat splits subclass of FileInputFormat which returns false 
> for isSplitable()
> ---
>
> Key: HIVE-8630
> URL: https://issues.apache.org/jira/browse/HIVE-8630
> Project: Hive
>  Issue Type: Bug
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-8630.1.patch.txt
>
>
> CombineFileInputFormat in hadoop does not check FileInputFormat.isSplitable 
> and tries to split input file. Happened on custom SerDe which handles 
> multi-lined json (return false always for isSplitable).
> Can be bypass by setting it HiveInputFormat for the query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8610) Compile time skew join optimization doesn't work with auto map join

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-8610:
-
Status: Patch Available  (was: Open)

> Compile time skew join optimization doesn't work with auto map join
> ---
>
> Key: HIVE-8610
> URL: https://issues.apache.org/jira/browse/HIVE-8610
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-8610.1.patch
>
>
> NPE is thrown if both {{hive.optimize.skewjoin.compiletime}} and 
> {{hive.auto.convert.join}} are enabled:
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.convertMapJoin(MapJoinProcessor.java:329)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:236)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:181)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:463)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:275)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:223)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10028)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:415)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1068)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1130)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1005)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:995)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:246)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:198)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> A simple way to reproduce this issue is to add {{set 
> hive.auto.convert.join=true}} to one of the skew join qfile, e.g. 
> {{skewjoinopt2.q}}.
> While reduce side join can produce correct results, we kind of lost the point 
> of skew join optimization - join skewed data via a map join to avoid one 
> reducer getting too much records.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8610) Compile time skew join optimization doesn't work with auto map join

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-8610:
-
Attachment: HIVE-8610.1.patch

This patch adds QBJoinTree and colExprMap for the cloned join operator tree in 
SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned 
join to map join.
The added tests are copied from skewjoinopt*.q, except that auto map join is 
enabled.

> Compile time skew join optimization doesn't work with auto map join
> ---
>
> Key: HIVE-8610
> URL: https://issues.apache.org/jira/browse/HIVE-8610
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-8610.1.patch
>
>
> NPE is thrown if both {{hive.optimize.skewjoin.compiletime}} and 
> {{hive.auto.convert.join}} are enabled:
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.convertMapJoin(MapJoinProcessor.java:329)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:236)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:181)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:463)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:275)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:223)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10028)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:415)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1068)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1130)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1005)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:995)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:246)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:198)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> A simple way to reproduce this issue is to add {{set 
> hive.auto.convert.join=true}} to one of the skew join qfile, e.g. 
> {{skewjoinopt2.q}}.
> While reduce side join can produce correct results, we kind of lost the point 
> of skew join optimization - join skewed data via a map join to avoid one 
> reducer getting too much records.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8610) Compile time skew join optimization doesn't work with auto map join

[Rui Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186853#comment-14186853
 ] 

Rui Li commented on HIVE-8610:
--

Hi [~xuefuz], do I have to add something to {{testconfiguration.properties}} to 
make hive automatically run the new tests for MR?

> Compile time skew join optimization doesn't work with auto map join
> ---
>
> Key: HIVE-8610
> URL: https://issues.apache.org/jira/browse/HIVE-8610
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-8610.1.patch
>
>
> NPE is thrown if both {{hive.optimize.skewjoin.compiletime}} and 
> {{hive.auto.convert.join}} are enabled:
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.convertMapJoin(MapJoinProcessor.java:329)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:236)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:181)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:463)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:275)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:223)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10028)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:415)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1068)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1130)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1005)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:995)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:246)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:198)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> A simple way to reproduce this issue is to add {{set 
> hive.auto.convert.join=true}} to one of the skew join qfile, e.g. 
> {{skewjoinopt2.q}}.
> While reduce side join can produce correct results, we kind of lost the point 
> of skew join optimization - join skewed data via a map join to avoid one 
> reducer getting too much records.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 27283: Compile time skew join optimization doesn't work with auto map join

[Rui Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27283/
---

Review request for hive, Szehon Ho and Xuefu Zhang.


Bugs: HIVE-8610
https://issues.apache.org/jira/browse/HIVE-8610


Repository: hive-git


Description
---

This patch adds QBJoinTree and colExprMap for the cloned join operator tree in 
SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned 
join to map join.
The added tests are copied from skewjoinopt*.q, except that auto map join is 
enabled.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java 
e87c41b 
  ql/src/test/queries/clientpositive/skewjoin_mapjoin1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/skewjoin_mapjoin10.q PRE-CREATION 
  ql/src/test/queries/clientpositive/skewjoin_mapjoin11.q PRE-CREATION 
  ql/src/test/queries/clientpositive/skewjoin_mapjoin2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/skewjoin_mapjoin3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/skewjoin_mapjoin4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/skewjoin_mapjoin5.q PRE-CREATION 
  ql/src/test/queries/clientpositive/skewjoin_mapjoin6.q PRE-CREATION 
  ql/src/test/queries/clientpositive/skewjoin_mapjoin7.q PRE-CREATION 
  ql/src/test/queries/clientpositive/skewjoin_mapjoin8.q PRE-CREATION 
  ql/src/test/queries/clientpositive/skewjoin_mapjoin9.q PRE-CREATION 
  ql/src/test/results/clientpositive/skewjoin_mapjoin1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/skewjoin_mapjoin10.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/skewjoin_mapjoin2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/skewjoin_mapjoin3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/skewjoin_mapjoin4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/skewjoin_mapjoin6.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/skewjoin_mapjoin8.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/skewjoin_mapjoin9.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/27283/diff/


Testing
---


Thanks,

Rui Li

[jira] [Commented] (HIVE-7111) Extend join transitivity PPD to non-column expressions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186861#comment-14186861
 ] 

Hive QA commented on HIVE-7111:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677553/HIVE-7111.2.patch.txt

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6580 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1501/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1501/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1501/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677553 - PreCommit-HIVE-TRUNK-Build

> Extend join transitivity PPD to non-column expressions
> --
>
> Key: HIVE-7111
> URL: https://issues.apache.org/jira/browse/HIVE-7111
> Project: Hive
>  Issue Type: Task
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-7111.1.patch.txt, HIVE-7111.2.patch.txt, 
> HIVE-7111.2.patch.txt, HIVE-7111.3.patch.txt, HIVE-7111.4.patch.txt
>
>
> Join transitive in PPD only supports column expressions, but it's possible to 
> extend this to generic expressions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7292) Hive on Spark

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186862#comment-14186862
 ] 

Paulo Motta commented on HIVE-7292:
---

Is the branch already usable in production?

> Hive on Spark
> -
>
> Key: HIVE-7292
> URL: https://issues.apache.org/jira/browse/HIVE-7292
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>  Labels: Spark-M1, Spark-M2, Spark-M3, Spark-M4, Spark-M5
> Attachments: Hive-on-Spark.pdf
>
>
> Spark as an open-source data analytics cluster computing framework has gained 
> significant momentum recently. Many Hive users already have Spark installed 
> as their computing backbone. To take advantages of Hive, they still need to 
> have either MapReduce or Tez on their cluster. This initiative will provide 
> user a new alternative so that those user can consolidate their backend. 
> Secondly, providing such an alternative further increases Hive's adoption as 
> it exposes Spark users  to a viable, feature-rich de facto standard SQL tools 
> on Hadoop.
> Finally, allowing Hive to run on Spark also has performance benefits. Hive 
> queries, especially those involving multiple reducer stages, will run faster, 
> thus improving user experience as Tez does.
> This is an umbrella JIRA which will cover many coming subtask. Design doc 
> will be attached here shortly, and will be on the wiki as well. Feedback from 
> the community is greatly appreciated!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7292) Hive on Spark

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186891#comment-14186891
 ] 

Xuefu Zhang commented on HIVE-7292:
---

[~pauloricardomg], thanks for your interest. I think the branch is ready for 
propective users to try out, but I'd recommend for production you wait for a 
formal release.

> Hive on Spark
> -
>
> Key: HIVE-7292
> URL: https://issues.apache.org/jira/browse/HIVE-7292
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>  Labels: Spark-M1, Spark-M2, Spark-M3, Spark-M4, Spark-M5
> Attachments: Hive-on-Spark.pdf
>
>
> Spark as an open-source data analytics cluster computing framework has gained 
> significant momentum recently. Many Hive users already have Spark installed 
> as their computing backbone. To take advantages of Hive, they still need to 
> have either MapReduce or Tez on their cluster. This initiative will provide 
> user a new alternative so that those user can consolidate their backend. 
> Secondly, providing such an alternative further increases Hive's adoption as 
> it exposes Spark users  to a viable, feature-rich de facto standard SQL tools 
> on Hadoop.
> Finally, allowing Hive to run on Spark also has performance benefits. Hive 
> queries, especially those involving multiple reducer stages, will run faster, 
> thus improving user experience as Tez does.
> This is an umbrella JIRA which will cover many coming subtask. Design doc 
> will be attached here shortly, and will be on the wiki as well. Feedback from 
> the community is greatly appreciated!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8461) Make Vectorized Decimal query results match Non-Vectorized query results with respect to trailing zeroes... .0000

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186922#comment-14186922
 ] 

Hive QA commented on HIVE-8461:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677588/HIVE-8461.01.patch

{color:red}ERROR:{color} -1 due to 36 failed/errored test(s), 6589 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testAvgDecimal
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testAvgDecimalNegative
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testCountDecimal
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testDecimalKeyTypeAggregate
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testMaxDecimal
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testMinDecimal
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testStdPopDecimal
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testStdSampDecimal
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testSumDecimal
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testSumDecimalHive6508
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testVarSampDecimal
org.apache.hadoop.hive.ql.exec.vector.TestVectorGroupByOperator.testVarianceDecimal
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalColAddDecimalColumn
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalColAddDecimalScalar
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalColDivideDecimalColumn
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalColDivideDecimalScalar
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalColModuloDecimalColumn
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalColModuloDecimalScalar
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalColMultiplyDecimalColumn
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalColMultiplyDecimalScalar
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalColSubtractDecimalColumn
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalColSubtractDecimalScalar
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalScalarAddDecimalColumn
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalScalarDivideDecimalColumn
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalScalarModuloDecimalColumn
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalScalarMultiplyDecimalColumn
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorArithmeticExpressions.testDecimalScalarSubtractDecimalColumn
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTypeCasts.testCastDecimalToDecimal
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTypeCasts.testCastDecimalToString
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTypeCasts.testCastDoubleToDecimal
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTypeCasts.testCastLongToDecimal
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTypeCasts.testCastStringToDecimal
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTypeCasts.testCastTimestampToDecimal
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorizationWithAcid
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1502/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1502/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1502/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 36 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677588 - PreCommit-HIVE-TRUNK-Build

> Make Vectorized Decimal query results match Non-Vectorized query results with 
> respect to trailing zeroes... .
> ---

Re: How to run TestCliDriver Unit Test

[Gordon Wang]

Nice!

Thanks to Xuefu and Serigo for your reply!
It works now!

On Tue, Oct 28, 2014 at 1:21 PM, Sergio Pena 
wrote:

> Hey Gordon,
>
> Now it is done in two steps.
>
> 1) Compile the main project first
> trunk# mvn clean compile -Phadoop-2   -DskipTests
>
> 2) Compile the itests/ directory second
> trunk# cd itests
> itests# mvn clean compile -Phadoop-2  -DskipTests
>
> Then, you can start the test you want (let's say you want to run
> avro_joins.q).
>
> itests# mvn test -Phadoop-2 -Dtest=TestCliDriver -Dqfile=avro_joins.q
>
> The test log will be written to itests/qtest/target/surefire-reports/
> directory.
>
> You can replace TestCliDriver with TestNegativeCliDriver if you want to run
> negative tests.
>
> If you want to overwrite tests output results (avro_joins.q.out), then you
> can append -Dtest.output.overwrite=true to the mvn command.
>
> Here's more information about the new tests command:
>
> https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeveloperFAQ-Whereisthelogoutputofatest
> ?
>
> Sergio
>
> On Mon, Oct 27, 2014 at 10:22 PM, Gordon Wang  wrote:
>
> > Hi all,
> >
> > I am working on a hive issue and I want to add a Unit test for hive query
> > parser.
> >
> > Before hive 0.13, it is built with ant. And the query parser test cases
> > were ran with TestCliDriver.
> >
> > In current trunk, hive is built with maven. Could you give me some guide
> > about how to run TestCliDriver unit test?
> >
> > I tried " mvn clean test -Phadoop-2  -Dtest=TestCliDriver", but it didn't
> > work. Seems that TestCliDriver.java is not genrated yet.
> >
> > Many thanks.
> >
> > --
> > Regards
> > Gordon Wang
> >
>



-- 
Regards
Gordon Wang

Re: [ANNOUNCE] New Hive PMC Member - Alan Gates

[Devopam Mittra]

Congratulations Alan !


regards
Dev

On Tue, Oct 28, 2014 at 4:08 AM, Carl Steinbach  wrote:

> I am pleased to announce that Alan Gates has been elected to the Hive
> Project Management Committee. Please join me in congratulating Alan!
>
> Thanks.
>
> - Carl
>



-- 
Devopam Mittra
Life and Relations are not binary

[jira] [Updated] (HIVE-8539) Enable collect table statistics based on SparkCounter[Spark Branch]

[Chengxiang Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chengxiang Li updated HIVE-8539:

Attachment: Hive on Spark table stats collection on counter.docx

> Enable collect table statistics based on SparkCounter[Spark Branch]
> ---
>
> Key: HIVE-8539
> URL: https://issues.apache.org/jira/browse/HIVE-8539
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Chengxiang Li
>Assignee: Chengxiang Li
>  Labels: Spark-M3
> Attachments: Hive on Spark table stats collection on counter.docx
>
>
> Hive support collect table statistics based on Counters/TezCounters in MR/Tez 
>  mode, we should enable this in Spark mode as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27265: Support SMB Join for Hive on Spark [Spark Branch]

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27265/#review58809
---



ql/src/java/org/apache/hadoop/hive/ql/lib/TypeRule.java


stack.peek()?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java


Remove int pos = 0; define pos here?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java


This should be always true. Are you checking if pos != -1?


- Jimmy Xiang


On Oct. 28, 2014, 2:20 a.m., Szehon Ho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27265/
> ---
> 
> (Updated Oct. 28, 2014, 2:20 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This change re-uses the SMBJoinOperator for Spark.  Background: the logical 
> layer already converts joins to SMB Joins.  This changes just introduces a 
> class called "SparkSortMergeJoinFactory" on the Spark-compile path which 
> attaches the data structures (like local work, bucket info) to the MapWork 
> for the SMBJoinOperator to consume.  It is largely-based on the MapReduce 
> class "MapJoinFactory".
> 
> However, in spark-path, it is activated only for SMBJoin and not map-joins, 
> as we have another strategy for map-joins.  That is why there's a new 
> optimizer-rule called "TypeRule", so this processor is only run on 
> SMBJoinOperators (which share same name with MapJoinOperators, which is 
> needed for logical-optimizers dealing with hints).
> 
> One major assumption around the whole SMB concept is that both tables have 
> corresponding buckets.  I found during testing of large numbers of buckets 
> (like auto_sortmerge_join_16) that "insert" into a bucketed table wasn't 
> putting the same keys in corresponding buckets.  I activated MR-style shuffle 
> (hash-shuffle instead of total-order shuffle), and that seemed to solve the 
> issue.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 00c9f4d 
>   ql/src/java/org/apache/hadoop/hive/ql/lib/TypeRule.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java ae1d1ab 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 
> ed88c60 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> 8e28887 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java 4f5feca 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 
> 1c663c4 
>   ql/src/test/results/clientpositive/spark/auto_join0.q.out 76ff63d 
>   ql/src/test/results/clientpositive/spark/auto_join10.q.out 05a5912 
>   ql/src/test/results/clientpositive/spark/auto_join11.q.out 998c28b 
>   ql/src/test/results/clientpositive/spark/auto_join12.q.out d2b7993 
>   ql/src/test/results/clientpositive/spark/auto_join13.q.out 78aa01e 
>   ql/src/test/results/clientpositive/spark/auto_join15.q.out 5916070 
>   ql/src/test/results/clientpositive/spark/auto_join16.q.out 0b6807d 
>   ql/src/test/results/clientpositive/spark/auto_join18.q.out 6083b38 
>   ql/src/test/results/clientpositive/spark/auto_join18_multi_distinct.q.out 
> 01c8f0a 
>   ql/src/test/results/clientpositive/spark/auto_join20.q.out a8f2b9a 
>   ql/src/test/results/clientpositive/spark/auto_join21.q.out f9ac35d 
>   ql/src/test/results/clientpositive/spark/auto_join22.q.out 516322c 
>   ql/src/test/results/clientpositive/spark/auto_join23.q.out ce5a670 
>   ql/src/test/results/clientpositive/spark/auto_join24.q.out 15b 
>   ql/src/test/results/clientpositive/spark/auto_join27.q.out 67f5739 
>   ql/src/test/results/clientpositive/spark/auto_join28.q.out b979661 
>   ql/src/test/results/clientpositive/spark/auto_join29.q.out 0951b8d 
>   ql/src/test/results/clientpositive/spark/auto_join30.q.out 98b3974 
>   ql/src/test/results/clientpositive/spark/auto_join31.q.out df502c8 
>   ql/src/test/results/clientpositive/spark/auto_join32.q.out 8d83188 
>   ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out e64d4fb 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out 
> 9158d65 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_10.q.out 
> f608cc5 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_11.q.out 
> 3c26363 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out 
> 65e496f 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_13.q.out 
> a5a281b 

[jira] [Updated] (HIVE-8495) Add progress bar for Hive on Tez queries

[Mostafa Mokhtar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mostafa Mokhtar updated HIVE-8495:
--
Attachment: (was: HIVE-8495.2.patch)

> Add progress bar for Hive on Tez queries
> 
>
> Key: HIVE-8495
> URL: https://issues.apache.org/jira/browse/HIVE-8495
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Mostafa Mokhtar
> Fix For: 0.14.0
>
> Attachments: HIVE-8495.1.patch, Screen Shot 2014-10-16 at 9.35.26 
> PM.png, Screen Shot 2014-10-22 at 11.48.57 AM.png
>
>
> Build a "Progress bar" to provide overall progress on running tasks.
> Progress is calculated as : 
>  (Completed tasks) / (Total number of tasks)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8495) Add progress bar for Hive on Tez queries

[Mostafa Mokhtar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mostafa Mokhtar updated HIVE-8495:
--
Attachment: HIVE-8495.2.patch

> Add progress bar for Hive on Tez queries
> 
>
> Key: HIVE-8495
> URL: https://issues.apache.org/jira/browse/HIVE-8495
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Mostafa Mokhtar
> Fix For: 0.14.0
>
> Attachments: HIVE-8495.1.patch, HIVE-8495.2.patch, Screen Shot 
> 2014-10-16 at 9.35.26 PM.png, Screen Shot 2014-10-22 at 11.48.57 AM.png
>
>
> Build a "Progress bar" to provide overall progress on running tasks.
> Progress is calculated as : 
>  (Completed tasks) / (Total number of tasks)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New Hive PMC Member - Alan Gates

[Alan Gates]

Thanks everyone for the congratulations.  I look forward to being able 
to serve Hive as part of the PMC.


Alan.


Carl Steinbach 
October 27, 2014 at 15:38
I am pleased to announce that Alan Gates has been elected to the Hive 
Project Management Committee. Please join me in congratulating Alan!


Thanks.

- Carl


--
Sent with Postbox 

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Commented] (HIVE-7111) Extend join transitivity PPD to non-column expressions

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186956#comment-14186956
 ] 

Ashutosh Chauhan commented on HIVE-7111:


I think changes in vector_mapjoin_reduce are correct. After filter got pushed, 
constant propagate optimized select expression to constant which looks good to 
me. [~navis] Can you update RB entry for this ?

> Extend join transitivity PPD to non-column expressions
> --
>
> Key: HIVE-7111
> URL: https://issues.apache.org/jira/browse/HIVE-7111
> Project: Hive
>  Issue Type: Task
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-7111.1.patch.txt, HIVE-7111.2.patch.txt, 
> HIVE-7111.2.patch.txt, HIVE-7111.3.patch.txt, HIVE-7111.4.patch.txt
>
>
> Join transitive in PPD only supports column expressions, but it's possible to 
> extend this to generic expressions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8321) Fix serialization of TypeInfo for qualified types

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186961#comment-14186961
 ] 

Ashutosh Chauhan commented on HIVE-8321:


yeah, it can go to 0.14 

> Fix serialization of TypeInfo for qualified types
> -
>
> Key: HIVE-8321
> URL: https://issues.apache.org/jira/browse/HIVE-8321
> Project: Hive
>  Issue Type: Bug
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.15.0
>
> Attachments: HIVE-8321.1.patch, HIVE-8321.2.patch, HIVE-8321.3.patch
>
>
> TypeInfos for decimal/char/varchar don't appear to be serializing properly 
> with javaXML.
> Decimal needed proper getters/setters for precision/scale.
> Also disabling setTypeInfo since for decimal/char/varchar the proper type 
> name should already be set by the constructor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New Hive PMC Member - Alan Gates

[Suhas Gogate]

Alan, Congratulations! Being so experienced in this field, this is
definitely a very good sign for Hive project!   --Suhas

On Tue, Oct 28, 2014 at 7:22 AM, Alan Gates  wrote:

> Thanks everyone for the congratulations.  I look forward to being able to
> serve Hive as part of the PMC.
>
> Alan.
>
>   Carl Steinbach 
>  October 27, 2014 at 15:38
> I am pleased to announce that Alan Gates has been elected to the Hive
> Project Management Committee. Please join me in congratulating Alan!
>
> Thanks.
>
> - Carl
>
>
> --
> Sent with Postbox 
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

[jira] [Commented] (HIVE-8629) Streaming / ACID : hive cli session creation takes too long and times out if execution engine is tez

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186997#comment-14186997
 ] 

Alan Gates commented on HIVE-8629:
--

bq.  Does this config change affect compaction jobs triggered by ACID?
No.  Compactions are done by the metastore service, not the clients.  And they 
are always done using MR, regardless of what execution engine is configured.

> Streaming / ACID : hive cli session creation takes too long and times out if 
> execution engine is tez
> 
>
> Key: HIVE-8629
> URL: https://issues.apache.org/jira/browse/HIVE-8629
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.14.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>  Labels: ACID, Streaming
> Attachments: HIVE-8629.patch
>
>
> When creating a hive session to run basic "alter table create partition"  
> queries, the session creation takes too long (more than 5 sec)  if the hive 
> execution engine is set to tez.
> Since the streaming clients dont care about Tez , it can explicitly override 
> the setting to mr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8610) Compile time skew join optimization doesn't work with auto map join

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186998#comment-14186998
 ] 

Hive QA commented on HIVE-8610:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677603/HIVE-8610.1.patch

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6589 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1503/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1503/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1503/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677603 - PreCommit-HIVE-TRUNK-Build

> Compile time skew join optimization doesn't work with auto map join
> ---
>
> Key: HIVE-8610
> URL: https://issues.apache.org/jira/browse/HIVE-8610
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-8610.1.patch
>
>
> NPE is thrown if both {{hive.optimize.skewjoin.compiletime}} and 
> {{hive.auto.convert.join}} are enabled:
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.convertMapJoin(MapJoinProcessor.java:329)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:236)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:181)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:463)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:275)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:223)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10028)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:415)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1068)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1130)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1005)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:995)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:246)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:198)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect

[jira] [Commented] (HIVE-8629) Streaming / ACID : hive cli session creation takes too long and times out if execution engine is tez

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187002#comment-14187002
 ] 

Alan Gates commented on HIVE-8629:
--

Why is it necessary to set METASTORE_EXECUTE_SET_UGI?  As far as I can tell 
this is only used by the metastore, and so has no affect on the client.

I think you should add a LOG.info() in overrideConfSettings so that users know 
you are overriding some values in their configuration.

> Streaming / ACID : hive cli session creation takes too long and times out if 
> execution engine is tez
> 
>
> Key: HIVE-8629
> URL: https://issues.apache.org/jira/browse/HIVE-8629
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.14.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>  Labels: ACID, Streaming
> Attachments: HIVE-8629.patch
>
>
> When creating a hive session to run basic "alter table create partition"  
> queries, the session creation takes too long (more than 5 sec)  if the hive 
> execution engine is set to tez.
> Since the streaming clients dont care about Tez , it can explicitly override 
> the setting to mr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8610) Compile time skew join optimization doesn't work with auto map join

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187040#comment-14187040
 ] 

Xuefu Zhang commented on HIVE-8610:
---

{quote}
do I have to add something to testconfiguration.properties to make hive 
automatically run the new tests for MR?
{quote}

No. You should not need to, unless you have some specific needs.

BTW, some test failures above seems relevant to your patch.

> Compile time skew join optimization doesn't work with auto map join
> ---
>
> Key: HIVE-8610
> URL: https://issues.apache.org/jira/browse/HIVE-8610
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-8610.1.patch
>
>
> NPE is thrown if both {{hive.optimize.skewjoin.compiletime}} and 
> {{hive.auto.convert.join}} are enabled:
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.convertMapJoin(MapJoinProcessor.java:329)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:236)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:181)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:463)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:275)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:223)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10028)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:415)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1068)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1130)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1005)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:995)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:246)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:198)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> A simple way to reproduce this issue is to add {{set 
> hive.auto.convert.join=true}} to one of the skew join qfile, e.g. 
> {{skewjoinopt2.q}}.
> While reduce side join can produce correct results, we kind of lost the point 
> of skew join optimization - join skewed data via a map join to avoid one 
> reducer getting too much records.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8435) Add identity project remover optimization

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesús Camacho Rodríguez updated HIVE-8435:
--
Attachment: HIVE-8435.05.patch

I think the following fix is sound, although maybe it is not complete (it does 
not cover all cases). It is based on some aux functions that were also 
introduced in https://issues.apache.org/jira/browse/HIVE-8625 .

> Add identity project remover optimization
> -
>
> Key: HIVE-8435
> URL: https://issues.apache.org/jira/browse/HIVE-8435
> Project: Hive
>  Issue Type: New Feature
>  Components: Logical Optimizer
>Affects Versions: 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8435.02.patch, HIVE-8435.03.patch, 
> HIVE-8435.03.patch, HIVE-8435.04.patch, HIVE-8435.05.patch, 
> HIVE-8435.1.patch, HIVE-8435.patch
>
>
> In some cases there is an identity project in plan which is useless. Better 
> to optimize it away to avoid evaluating it without any benefit at runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8631) Compressed transaction list cannot be parsed in job.xml

[Alan Gates (JIRA)]

Alan Gates created HIVE-8631:


 Summary: Compressed transaction list cannot be parsed in job.xml
 Key: HIVE-8631
 URL: https://issues.apache.org/jira/browse/HIVE-8631
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 0.14.0
Reporter: Alan Gates
Assignee: Alan Gates
Priority: Blocker
 Fix For: 0.14.0


HIVE-8341 added code to compress the transaction list in the JobConf when it 
reaches a certain size.  This breaks when the JobConf is converted to job.xml 
and sent to MR.  The special characters are not correctly escaped.  

The proposed fix is to back out the compression (not the changes to 
ScriptOperator) for now and look later at compressing/encoding it in a way that 
works with job.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8561) Expose Hive optiq operator tree to be able to support other sql on hadoop query engines

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187069#comment-14187069
 ] 

Brock Noland commented on HIVE-8561:


I am not familiar with the COB code so it's possible we want to take a 
different approach. However, in principle I don't see an issue with the idea of 
this change. Additionally, any API that we open to limited group of downstream 
projects, such as Drill, should be marked with annotations. For example 
{{LimitedPrivate("Apache Hive, Apache Drill (Incubating)")}} and then either 
{{Unstable}} or {{Evolving}}.

https://github.com/apache/hive/blob/trunk/common/src/java/org/apache/hadoop/hive/common/classification/InterfaceAudience.java#L34
https://github.com/apache/hive/blob/trunk/common/src/java/org/apache/hadoop/hive/common/classification/InterfaceStability.java#L34

We should do that for any change which opens API's for Drill. Additionally, I 
see we marked some members from java private to java public. As opposed to 
doing that we should add getters for those member variables.

> Expose Hive optiq operator tree to be able to support other sql on hadoop 
> query engines
> ---
>
> Key: HIVE-8561
> URL: https://issues.apache.org/jira/browse/HIVE-8561
> Project: Hive
>  Issue Type: Task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Na Yang
>Assignee: Na Yang
> Attachments: HIVE-8561.2.patch, HIVE-8561.patch
>
>
> Hive-0.14 added cost based optimization and optiq operator tree is created 
> for select queries. However, the optiq operator tree is not visible from 
> outside and hard to be used by other Sql on Hadoop query engine such as 
> apache Drill. To be able to allow drill to access the hive optiq operator 
> tree, we need to add a public api to return the hive optiq operator tree.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8631) Compressed transaction list cannot be parsed in job.xml

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8631:
-
Attachment: HIVE-8631.patch

> Compressed transaction list cannot be parsed in job.xml
> ---
>
> Key: HIVE-8631
> URL: https://issues.apache.org/jira/browse/HIVE-8631
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8631.patch
>
>
> HIVE-8341 added code to compress the transaction list in the JobConf when it 
> reaches a certain size.  This breaks when the JobConf is converted to job.xml 
> and sent to MR.  The special characters are not correctly escaped.  
> The proposed fix is to back out the compression (not the changes to 
> ScriptOperator) for now and look later at compressing/encoding it in a way 
> that works with job.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8631) Compressed transaction list cannot be parsed in job.xml

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8631:
-
Status: Patch Available  (was: Open)

> Compressed transaction list cannot be parsed in job.xml
> ---
>
> Key: HIVE-8631
> URL: https://issues.apache.org/jira/browse/HIVE-8631
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8631.patch
>
>
> HIVE-8341 added code to compress the transaction list in the JobConf when it 
> reaches a certain size.  This breaks when the JobConf is converted to job.xml 
> and sent to MR.  The special characters are not correctly escaped.  
> The proposed fix is to back out the compression (not the changes to 
> ScriptOperator) for now and look later at compressing/encoding it in a way 
> that works with job.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8632) VectorKeyHashWrapper::clone allocates too many zero sized arrays

[Gopal V (JIRA)]

Gopal V created HIVE-8632:
-

 Summary: VectorKeyHashWrapper::clone allocates too many zero sized 
arrays
 Key: HIVE-8632
 URL: https://issues.apache.org/jira/browse/HIVE-8632
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Gopal V


VectorHashKeyWrapper::clone() should not make copies of zero sized typed arrays.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8605) HIVE-5799 breaks backward compatibility for time values in config

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8605:
-
Status: Open  (was: Patch Available)

Posting a new patch that removes 'f' as an option, just supports 'l' as a now 
ignored modifier for time units.

> HIVE-5799 breaks backward compatibility for time values in config
> -
>
> Key: HIVE-8605
> URL: https://issues.apache.org/jira/browse/HIVE-8605
> Project: Hive
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8605.patch
>
>
> It is legal for long values in the config file to have an L or for float 
> values to have an f.  For example, the default value for 
> hive.compactor.check.interval was 300L.  As part of HIVE-5799, many long 
> values were converted to TimeUnit.  Attempts to read these values now throw 
> "java.lang.IllegalArgumentException: Invalid time unit l"
> We need to change this to ignore the L or f, so that users existing config 
> files don't break.  I propose to do this by changing HiveConf.unitFor to 
> detect the L or f and interpret it to mean the default time unit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8605) HIVE-5799 breaks backward compatibility for time values in config

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8605:
-
Status: Patch Available  (was: Open)

> HIVE-5799 breaks backward compatibility for time values in config
> -
>
> Key: HIVE-8605
> URL: https://issues.apache.org/jira/browse/HIVE-8605
> Project: Hive
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8605.2.patch, HIVE-8605.patch
>
>
> It is legal for long values in the config file to have an L or for float 
> values to have an f.  For example, the default value for 
> hive.compactor.check.interval was 300L.  As part of HIVE-5799, many long 
> values were converted to TimeUnit.  Attempts to read these values now throw 
> "java.lang.IllegalArgumentException: Invalid time unit l"
> We need to change this to ignore the L or f, so that users existing config 
> files don't break.  I propose to do this by changing HiveConf.unitFor to 
> detect the L or f and interpret it to mean the default time unit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8605) HIVE-5799 breaks backward compatibility for time values in config

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8605:
-
Attachment: HIVE-8605.2.patch

> HIVE-5799 breaks backward compatibility for time values in config
> -
>
> Key: HIVE-8605
> URL: https://issues.apache.org/jira/browse/HIVE-8605
> Project: Hive
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8605.2.patch, HIVE-8605.patch
>
>
> It is legal for long values in the config file to have an L or for float 
> values to have an f.  For example, the default value for 
> hive.compactor.check.interval was 300L.  As part of HIVE-5799, many long 
> values were converted to TimeUnit.  Attempts to read these values now throw 
> "java.lang.IllegalArgumentException: Invalid time unit l"
> We need to change this to ignore the L or f, so that users existing config 
> files don't break.  I propose to do this by changing HiveConf.unitFor to 
> detect the L or f and interpret it to mean the default time unit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-6669) sourcing txn-script from schema script results in failure for mysql & oracle

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187103#comment-14187103
 ] 

Alan Gates commented on HIVE-6669:
--

Patch 2 committed to trunk and branch 0.14.  Thanks Ashutosh for the review and 
Damien for your help on the patch.

> sourcing txn-script from schema script results in failure for mysql & oracle
> 
>
> Key: HIVE-6669
> URL: https://issues.apache.org/jira/browse/HIVE-6669
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Prasad Mujumdar
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-6669.2.patch, HIVE-6669.patch
>
>
> This issues is addressed in 0.13 by in-lining the the transaction schema 
> statements in the schema initialization script (HIVE-6559)
> The 0.14 schema initialization is not fixed. This is the followup ticket for 
> to address the problem in 0.14. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-6669) sourcing txn-script from schema script results in failure for mysql & oracle

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-6669:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> sourcing txn-script from schema script results in failure for mysql & oracle
> 
>
> Key: HIVE-6669
> URL: https://issues.apache.org/jira/browse/HIVE-6669
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Prasad Mujumdar
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-6669.2.patch, HIVE-6669.patch
>
>
> This issues is addressed in 0.13 by in-lining the the transaction schema 
> statements in the schema initialization script (HIVE-6559)
> The 0.14 schema initialization is not fixed. This is the followup ticket for 
> to address the problem in 0.14. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8628) NPE in case of shuffle join in tez

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187123#comment-14187123
 ] 

Gunther Hagleitner commented on HIVE-8628:
--

failures are unrelated.

> NPE in case of shuffle join in tez
> --
>
> Key: HIVE-8628
> URL: https://issues.apache.org/jira/browse/HIVE-8628
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8628.1.patch
>
>
> test throws NullPointerException:
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1413774081318_0803_5_03, 
> diagnostics=[Task failed, taskId=task_1413774081318_0803_5_03_00, 
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running 
> task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime 
> Error while closing operators: null
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing 
> operators: null
>   at 
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:218)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:178)
>   ... 13 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:368)
>   at 
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:310)
>   at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:582)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:200)
>   ... 14 more
> ], TaskAttempt 1 failed, info=[Error: Failure while running 
> task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime 
> Error while closing operators: null
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing 
> operators: null
>   at 
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:218)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:178)
>   ... 13 more
> Caused by: java

[jira] [Created] (HIVE-8633) Move Alan Gates from committer list to PMC list on website

[Alan Gates (JIRA)]

Alan Gates created HIVE-8633:


 Summary: Move Alan Gates from committer list to PMC list on website
 Key: HIVE-8633
 URL: https://issues.apache.org/jira/browse/HIVE-8633
 Project: Hive
  Issue Type: Task
  Components: Website
Reporter: Alan Gates
Assignee: Alan Gates






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8561) Expose Hive optiq operator tree to be able to support other sql on hadoop query engines

[Na Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187125#comment-14187125
 ] 

Na Yang commented on HIVE-8561:
---

[~brocknoland], thank you for the suggestion. If the new public APIs are 
acceptable, I will made the change as you suggested and upload a new patch.  
Thanks.

> Expose Hive optiq operator tree to be able to support other sql on hadoop 
> query engines
> ---
>
> Key: HIVE-8561
> URL: https://issues.apache.org/jira/browse/HIVE-8561
> Project: Hive
>  Issue Type: Task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Na Yang
>Assignee: Na Yang
> Attachments: HIVE-8561.2.patch, HIVE-8561.patch
>
>
> Hive-0.14 added cost based optimization and optiq operator tree is created 
> for select queries. However, the optiq operator tree is not visible from 
> outside and hard to be used by other Sql on Hadoop query engine such as 
> apache Drill. To be able to allow drill to access the hive optiq operator 
> tree, we need to add a public api to return the hive optiq operator tree.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8628) NPE in case of shuffle join in tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8628:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk and branch.

> NPE in case of shuffle join in tez
> --
>
> Key: HIVE-8628
> URL: https://issues.apache.org/jira/browse/HIVE-8628
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8628.1.patch
>
>
> test throws NullPointerException:
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1413774081318_0803_5_03, 
> diagnostics=[Task failed, taskId=task_1413774081318_0803_5_03_00, 
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running 
> task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime 
> Error while closing operators: null
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing 
> operators: null
>   at 
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:218)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:178)
>   ... 13 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:368)
>   at 
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:310)
>   at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:582)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:200)
>   ... 14 more
> ], TaskAttempt 1 failed, info=[Error: Failure while running 
> task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime 
> Error while closing operators: null
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing 
> operators: null
>   at 
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:218)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:178)
>   ... 13 m

[jira] [Commented] (HIVE-8631) Compressed transaction list cannot be parsed in job.xml

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187135#comment-14187135
 ] 

Gunther Hagleitner commented on HIVE-8631:
--

we've added an option to send the compressed plan via conf to tez. The way we 
did it there was to base 64 the compressed string. That should work here too.

Anyways: +1 for the patch to back it out.

> Compressed transaction list cannot be parsed in job.xml
> ---
>
> Key: HIVE-8631
> URL: https://issues.apache.org/jira/browse/HIVE-8631
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8631.patch
>
>
> HIVE-8341 added code to compress the transaction list in the JobConf when it 
> reaches a certain size.  This breaks when the JobConf is converted to job.xml 
> and sent to MR.  The special characters are not correctly escaped.  
> The proposed fix is to back out the compression (not the changes to 
> ScriptOperator) for now and look later at compressing/encoding it in a way 
> that works with job.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HIVE-8623) Implement SparkHashTableLoader for map-join broadcast variable read [Spark Branch]

[Jimmy Xiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-8623 started by Jimmy Xiang.
-
> Implement SparkHashTableLoader for map-join broadcast variable read [Spark 
> Branch]
> --
>
> Key: HIVE-8623
> URL: https://issues.apache.org/jira/browse/HIVE-8623
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Suhas Satish
>Assignee: Jimmy Xiang
>
> This is a sub-task of map-join for spark 
> https://issues.apache.org/jira/browse/HIVE-7613
> This can use the baseline patch for map-join
> https://issues.apache.org/jira/browse/HIVE-8616



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8634) HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly

[Prasad Mujumdar (JIRA)]

Prasad Mujumdar created HIVE-8634:
-

 Summary: HiveServer2 fair scheduler queue mapping doesn't handle 
the secondary groups rules correctly
 Key: HIVE-8634
 URL: https://issues.apache.org/jira/browse/HIVE-8634
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 0.14.0
Reporter: Prasad Mujumdar
Assignee: Prasad Mujumdar
 Fix For: 0.14.0, 0.15.0


The fair scheduler queue refresh in HiveServer2 (for non-impersonation mode), 
doesn't handle the primary/secondary queue mappings correctly. It's not reading 
primary and secondary rules from the scheduler rule file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8631) Compressed transaction list cannot be parsed in job.xml

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187140#comment-14187140
 ] 

Alan Gates commented on HIVE-8631:
--

I agree we should do that.  I just wanted to back it out for now since it's 
getting late and we need to quit adding features.  

> Compressed transaction list cannot be parsed in job.xml
> ---
>
> Key: HIVE-8631
> URL: https://issues.apache.org/jira/browse/HIVE-8631
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8631.patch
>
>
> HIVE-8341 added code to compress the transaction list in the JobConf when it 
> reaches a certain size.  This breaks when the JobConf is converted to job.xml 
> and sent to MR.  The special characters are not correctly escaped.  
> The proposed fix is to back out the compression (not the changes to 
> ScriptOperator) for now and look later at compressing/encoding it in a way 
> that works with job.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8321) Fix serialization of TypeInfo for qualified types

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-8321:
-
Fix Version/s: 0.14.0

> Fix serialization of TypeInfo for qualified types
> -
>
> Key: HIVE-8321
> URL: https://issues.apache.org/jira/browse/HIVE-8321
> Project: Hive
>  Issue Type: Bug
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.14.0, 0.15.0
>
> Attachments: HIVE-8321.1.patch, HIVE-8321.2.patch, HIVE-8321.3.patch
>
>
> TypeInfos for decimal/char/varchar don't appear to be serializing properly 
> with javaXML.
> Decimal needed proper getters/setters for precision/scale.
> Also disabling setTypeInfo since for decimal/char/varchar the proper type 
> name should already be set by the constructor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8321) Fix serialization of TypeInfo for qualified types

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187169#comment-14187169
 ] 

Jason Dere commented on HIVE-8321:
--

Committed to 0.14 branch

> Fix serialization of TypeInfo for qualified types
> -
>
> Key: HIVE-8321
> URL: https://issues.apache.org/jira/browse/HIVE-8321
> Project: Hive
>  Issue Type: Bug
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.14.0
>
> Attachments: HIVE-8321.1.patch, HIVE-8321.2.patch, HIVE-8321.3.patch
>
>
> TypeInfos for decimal/char/varchar don't appear to be serializing properly 
> with javaXML.
> Decimal needed proper getters/setters for precision/scale.
> Also disabling setTypeInfo since for decimal/char/varchar the proper type 
> name should already be set by the constructor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8321) Fix serialization of TypeInfo for qualified types

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-8321:
-
Fix Version/s: (was: 0.15.0)

> Fix serialization of TypeInfo for qualified types
> -
>
> Key: HIVE-8321
> URL: https://issues.apache.org/jira/browse/HIVE-8321
> Project: Hive
>  Issue Type: Bug
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.14.0
>
> Attachments: HIVE-8321.1.patch, HIVE-8321.2.patch, HIVE-8321.3.patch
>
>
> TypeInfos for decimal/char/varchar don't appear to be serializing properly 
> with javaXML.
> Decimal needed proper getters/setters for precision/scale.
> Also disabling setTypeInfo since for decimal/char/varchar the proper type 
> name should already be set by the constructor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 27292: HIVE-8634: HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly

[Prasad Mujumdar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27292/
---

Review request for hive, Brock Noland and Mohit Sabharwal.


Bugs: HIVE-8634
https://issues.apache.org/jira/browse/HIVE-8634


Repository: hive-git


Description
---

The fair scheduler queue refresh in HiveServer2 (for non-impersonation mode), 
doesn't handle the primary/secondary queue mappings correctly. It's not reading 
primary and secondary rules from the scheduler rule file.
The patch involves refreshing the queue policy allocation to read the 
fair-scheduler.xml file correctly.


Diffs
-

  data/conf/fair-scheduler-test.xml PRE-CREATION 
  
itests/hive-unit-hadoop2/src/test/java/org/apache/hive/jdbc/TestSchedulerQueue.java
 79878ba 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
6125714 

Diff: https://reviews.apache.org/r/27292/diff/


Testing
---

Added test cases to for various queue allocation rules.


Thanks,

Prasad Mujumdar

[jira] [Updated] (HIVE-8634) HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-8634:
--
Status: Patch Available  (was: Open)

> HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups 
> rules correctly
> 
>
> Key: HIVE-8634
> URL: https://issues.apache.org/jira/browse/HIVE-8634
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.14.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Fix For: 0.14.0, 0.15.0
>
> Attachments: HIVE-8634.1.patch
>
>
> The fair scheduler queue refresh in HiveServer2 (for non-impersonation mode), 
> doesn't handle the primary/secondary queue mappings correctly. It's not 
> reading primary and secondary rules from the scheduler rule file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8634) HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-8634:
--
Attachment: HIVE-8634.1.patch

> HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups 
> rules correctly
> 
>
> Key: HIVE-8634
> URL: https://issues.apache.org/jira/browse/HIVE-8634
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.14.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Fix For: 0.14.0, 0.15.0
>
> Attachments: HIVE-8634.1.patch
>
>
> The fair scheduler queue refresh in HiveServer2 (for non-impersonation mode), 
> doesn't handle the primary/secondary queue mappings correctly. It's not 
> reading primary and secondary rules from the scheduler rule file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27247: convert joinOp to MapJoinOp and generate MapWorks only

[Xuefu Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27247/#review58833
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkConvertJoinMapJoin.java


Nit: Let's keep order of the modifies in convention: private static final



ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkConvertJoinMapJoin.java


1. prviate vs public?
2. some comments on this method would be nice.



ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java


I don't see this method is called anywhere.


- Xuefu Zhang


On Oct. 27, 2014, 9:53 p.m., Suhas Satish wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27247/
> ---
> 
> (Updated Oct. 27, 2014, 9:53 p.m.)
> 
> 
> Review request for hive, Rui Li, Szehon Ho, and Xuefu Zhang.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This addresses jira HIVE-8616
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
> 46dcfaf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkConvertJoinMapJoin.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 
> ed88c60 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> 8e28887 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java 4f5feca 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 
> 1c663c4 
> 
> Diff: https://reviews.apache.org/r/27247/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Suhas Satish
> 
>

Re: Review Request 27065: Expose Hive optiq operator tree to be able to support other sql on hadoop query engines

[Na Yang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27065/
---

(Updated Oct. 28, 2014, 6:10 p.m.)


Review request for hive.


Changes
---

Made change according to Brock's suggestion


Bugs: Hive-8561
https://issues.apache.org/jira/browse/Hive-8561


Repository: hive-git


Description
---

Expose Hive optiq operator tree to be able to support other sql on hadoop query 
engines such as apache Drill


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java e254505 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/reloperators/HiveLimitRel.java
 f8755d0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d615aa3 

Diff: https://reviews.apache.org/r/27065/diff/


Testing
---


Thanks,

Na Yang

[jira] [Updated] (HIVE-8561) Expose Hive optiq operator tree to be able to support other sql on hadoop query engines

[Na Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Yang updated HIVE-8561:
--
Attachment: HIVE-8561.3.patch

Uploaded a new patch according to Brock's suggestion.

> Expose Hive optiq operator tree to be able to support other sql on hadoop 
> query engines
> ---
>
> Key: HIVE-8561
> URL: https://issues.apache.org/jira/browse/HIVE-8561
> Project: Hive
>  Issue Type: Task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Na Yang
>Assignee: Na Yang
> Attachments: HIVE-8561.2.patch, HIVE-8561.3.patch, HIVE-8561.patch
>
>
> Hive-0.14 added cost based optimization and optiq operator tree is created 
> for select queries. However, the optiq operator tree is not visible from 
> outside and hard to be used by other Sql on Hadoop query engine such as 
> apache Drill. To be able to allow drill to access the hive optiq operator 
> tree, we need to add a public api to return the hive optiq operator tree.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8435) Add identity project remover optimization

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187214#comment-14187214
 ] 

Hive QA commented on HIVE-8435:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677632/HIVE-8435.05.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1504/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1504/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1504/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-1504/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/0.20/target shims/0.20S/target 
shims/0.23/target shims/aggregator/target shims/common/target 
shims/common-secure/target hbase-handler/target jdbc/target metastore/target 
accumulo-handler/target common/target common/src/gen service/target 
contrib/target serde/target beeline/target cli/target 
ql/dependency-reduced-pom.xml ql/target
+ svn update
Uql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java
Umetastore/scripts/upgrade/postgres/hive-schema-0.14.0.postgres.sql
Ametastore/scripts/upgrade/postgres/hive-txn-schema-0.14.0.postgres.sql
Ametastore/scripts/upgrade/mssql/hive-txn-schema-0.14.0.mssql.sql
Ametastore/scripts/upgrade/derby/hive-txn-schema-0.14.0.derby.sql
Umetastore/scripts/upgrade/derby/hive-schema-0.14.0.derby.sql
Umetastore/scripts/upgrade/mysql/hive-schema-0.14.0.mysql.sql
Ametastore/scripts/upgrade/mysql/hive-txn-schema-0.14.0.mysql.sql
Umetastore/scripts/upgrade/oracle/hive-schema-0.14.0.oracle.sql
Ametastore/scripts/upgrade/oracle/hive-txn-schema-0.14.0.oracle.sql

Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1634938.

Updated to revision 1634938.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
patch:  malformed patch at line 400:  

patch:  malformed patch at line 400:  

patch:  malformed patch at line 400:  

The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677632 - PreCommit-HIVE-TRUNK-Build

> Add identity project remover optimization
> -
>
> Key: HIVE-8435
> URL: https://issues.apache.org/jira/browse/HIVE-8435
> Project: Hive
>  Issue Type: New Feature
>  Components: Logical Optimizer
>Affects Versions: 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8435.02.patch, HIVE-8435.03.patch, 
> HIVE-8435.03.patch, HIVE-8435.04.patch, HIVE-8435.05.patch, 
> HIVE-8435.1.patch, HIVE-8435.patch
>
>
> In some cases there is an identity project in plan which is useless. Better 
> to optimize it away to avoid evaluating it without any benefit at runtime.



--
This message was sent by Atlassian 

[jira] [Updated] (HIVE-8435) Add identity project remover optimization

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesús Camacho Rodríguez updated HIVE-8435:
--
Attachment: HIVE-8435.05.patch

> Add identity project remover optimization
> -
>
> Key: HIVE-8435
> URL: https://issues.apache.org/jira/browse/HIVE-8435
> Project: Hive
>  Issue Type: New Feature
>  Components: Logical Optimizer
>Affects Versions: 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8435.02.patch, HIVE-8435.03.patch, 
> HIVE-8435.03.patch, HIVE-8435.04.patch, HIVE-8435.05.patch, 
> HIVE-8435.05.patch, HIVE-8435.1.patch, HIVE-8435.patch
>
>
> In some cases there is an identity project in plan which is useless. Better 
> to optimize it away to avoid evaluating it without any benefit at runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8435) Add identity project remover optimization

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesús Camacho Rodríguez updated HIVE-8435:
--
Attachment: (was: HIVE-8435.05.patch)

> Add identity project remover optimization
> -
>
> Key: HIVE-8435
> URL: https://issues.apache.org/jira/browse/HIVE-8435
> Project: Hive
>  Issue Type: New Feature
>  Components: Logical Optimizer
>Affects Versions: 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8435.02.patch, HIVE-8435.03.patch, 
> HIVE-8435.03.patch, HIVE-8435.04.patch, HIVE-8435.05.patch, 
> HIVE-8435.1.patch, HIVE-8435.patch
>
>
> In some cases there is an identity project in plan which is useless. Better 
> to optimize it away to avoid evaluating it without any benefit at runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8435) Add identity project remover optimization

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesús Camacho Rodríguez updated HIVE-8435:
--
Attachment: HIVE-8435.05.patch

Fixed the patch.

> Add identity project remover optimization
> -
>
> Key: HIVE-8435
> URL: https://issues.apache.org/jira/browse/HIVE-8435
> Project: Hive
>  Issue Type: New Feature
>  Components: Logical Optimizer
>Affects Versions: 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8435.02.patch, HIVE-8435.03.patch, 
> HIVE-8435.03.patch, HIVE-8435.04.patch, HIVE-8435.05.patch, 
> HIVE-8435.05.patch, HIVE-8435.1.patch, HIVE-8435.patch
>
>
> In some cases there is an identity project in plan which is useless. Better 
> to optimize it away to avoid evaluating it without any benefit at runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8620) CBO: HIVE-8433 RowResolver check is too stringent

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187273#comment-14187273
 ] 

Sergey Shelukhin commented on HIVE-8620:


The patch is no good... it causes queries that CBO cannot handle to lose 
columns again, because they add same column twice because it is mentioned twice 
and it now works. I will look at another fix... probably the best long term fix 
is to exterminate the horror that is RowResolver and write something sane 
instead, where duplicates would never be allowed so there will be no 
programming-by-coincidence ambiguity

> CBO: HIVE-8433 RowResolver check is too stringent
> -
>
> Key: HIVE-8620
> URL: https://issues.apache.org/jira/browse/HIVE-8620
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-8620.patch
>
>
> Looks like it causes some valid queries to fail in CBO.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8618) Add SORT_QUERY_RESULTS for test that doesn't guarantee order #3

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-8618:
--
   Resolution: Fixed
Fix Version/s: 0.15.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Also merged to Spark branch. Thanks to Chao for the 
contribution.

> Add SORT_QUERY_RESULTS for test that doesn't guarantee order #3
> ---
>
> Key: HIVE-8618
> URL: https://issues.apache.org/jira/browse/HIVE-8618
> Project: Hive
>  Issue Type: Test
>Reporter: Chao
>Assignee: Chao
>Priority: Minor
> Fix For: 0.15.0
>
> Attachments: HIVE-8618.1.patch
>
>
> We need to add {{SORT_QUERY_RESULTS}} to a few more tests:
> {noformat}
> auto_join26
> date_join1
> join40
> vector_decimal_mapjoin
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8619) CBO causes some more type problems

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187284#comment-14187284
 ] 

Ashutosh Chauhan commented on HIVE-8619:


[~jpullokkaran] This is caused by HIVE-8530. Once, I removed that patch from my 
tree (and HIVE-8575) these above tests passes. Initial investigation seems to 
suggest that in these cases some literals which are coming in as double to Hive 
are now parsed as int after CBO. 
Would you like to take a look? 

cc: [~sershe]

> CBO causes some more type problems
> --
>
> Key: HIVE-8619
> URL: https://issues.apache.org/jira/browse/HIVE-8619
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Ashutosh Chauhan
>Priority: Critical
>
> On recent rerun, tests such as char_cast, decimal_2, literal_double, 
> vector_elt (incl. in Tez), udf4, vector_decimal_math_funcs have changes in 
> results it looks like something is broken with types again; ".0" suffix 
> disappeared in some cases (in char_cast it's probably the simplest to see), 
> number formats changed when casts are involved.
> Also, windowing_expressions and couple more tests fail due to attempts to 
> cast Double to Int writables, might be the same issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8619) CBO causes some more type problems

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-8619:
---
Assignee: Laljo John Pullokkaran  (was: Ashutosh Chauhan)

> CBO causes some more type problems
> --
>
> Key: HIVE-8619
> URL: https://issues.apache.org/jira/browse/HIVE-8619
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Laljo John Pullokkaran
>Priority: Critical
>
> On recent rerun, tests such as char_cast, decimal_2, literal_double, 
> vector_elt (incl. in Tez), udf4, vector_decimal_math_funcs have changes in 
> results it looks like something is broken with types again; ".0" suffix 
> disappeared in some cases (in char_cast it's probably the simplest to see), 
> number formats changed when casts are involved.
> Also, windowing_expressions and couple more tests fail due to attempts to 
> cast Double to Int writables, might be the same issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8635) CBO: ambiguous_col negative test no longer fails

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-8635:
--

 Summary: CBO: ambiguous_col negative test no longer fails
 Key: HIVE-8635
 URL: https://issues.apache.org/jira/browse/HIVE-8635
 Project: Hive
  Issue Type: Bug
  Components: CBO
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8614) Upgrade hive to use tez version 0.5.2-SNAPSHOT

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187303#comment-14187303
 ] 

Gunther Hagleitner commented on HIVE-8614:
--

Ran the failing test (sortmerge_join_5) after applying HIVE-8628. Now it 
produces NPE with this patch:

{noformat}
ava.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while 
closing operators
  at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
  at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
  at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
  at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:394)
  at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
  at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
  at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  at java.lang.Thread.run(Thread.java:695)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing 
operators
  at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:333)
  at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:177)
  ... 14 more
Caused by: java.lang.NullPointerException
  at 
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:309)
  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
  at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:312)
  ... 15 more
{noformat}


> Upgrade hive to use tez version 0.5.2-SNAPSHOT
> --
>
> Key: HIVE-8614
> URL: https://issues.apache.org/jira/browse/HIVE-8614
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8614.1.patch, HIVE-8614.2.patch, HIVE-8614.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8635) CBO: ambiguous_col negative test no longer fails

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8635:
---
Status: Patch Available  (was: Open)

> CBO: ambiguous_col negative test no longer fails
> 
>
> Key: HIVE-8635
> URL: https://issues.apache.org/jira/browse/HIVE-8635
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-8635.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8635) CBO: ambiguous_col negative test no longer fails

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8635:
---
Attachment: HIVE-8635.patch

> CBO: ambiguous_col negative test no longer fails
> 
>
> Key: HIVE-8635
> URL: https://issues.apache.org/jira/browse/HIVE-8635
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-8635.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8635) CBO: ambiguous_col negative test no longer fails

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187313#comment-14187313
 ] 

Sergey Shelukhin commented on HIVE-8635:


[~ashutoshc] [~jpullokkaran] tiny patch, can you guys check?
In this method, when getColsByRegex is called, it passes "true" as the last 
argument, to ensure unique columns.
This check was not extended to the clause of the if that doesn't call 
getCols... but just adds the thing directly.
Now it's being extended

> CBO: ambiguous_col negative test no longer fails
> 
>
> Key: HIVE-8635
> URL: https://issues.apache.org/jira/browse/HIVE-8635
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-8635.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8605) HIVE-5799 breaks backward compatibility for time values in config

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187321#comment-14187321
 ] 

Hive QA commented on HIVE-8605:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677637/HIVE-8605.2.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6580 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1505/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1505/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1505/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677637 - PreCommit-HIVE-TRUNK-Build

> HIVE-5799 breaks backward compatibility for time values in config
> -
>
> Key: HIVE-8605
> URL: https://issues.apache.org/jira/browse/HIVE-8605
> Project: Hive
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8605.2.patch, HIVE-8605.patch
>
>
> It is legal for long values in the config file to have an L or for float 
> values to have an f.  For example, the default value for 
> hive.compactor.check.interval was 300L.  As part of HIVE-5799, many long 
> values were converted to TimeUnit.  Attempts to read these values now throw 
> "java.lang.IllegalArgumentException: Invalid time unit l"
> We need to change this to ignore the L or f, so that users existing config 
> files don't break.  I propose to do this by changing HiveConf.unitFor to 
> detect the L or f and interpret it to mean the default time unit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8629) Streaming / ACID : hive cli session creation takes too long and times out if execution engine is tez

[Roshan Naik (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187329#comment-14187329
 ] 

Roshan Naik commented on HIVE-8629:
---

[~alangates]  the without setugi, the directories created by the metastore 
during add partition etc are done as hive user instead of the client user of 
the metastore process, consequently leading to incorrect permissions and later 
failure to stream to those directories.

WRT Log.info(): Since this done each time a new connection is created, which 
occurs multiple times over the duration of a long running streaming process, to 
reduce noise in the log output .. i am wondering if we should document this 
instead of log.info()  ? Either I am fine. Let me know what you think.

> Streaming / ACID : hive cli session creation takes too long and times out if 
> execution engine is tez
> 
>
> Key: HIVE-8629
> URL: https://issues.apache.org/jira/browse/HIVE-8629
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.14.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>  Labels: ACID, Streaming
> Attachments: HIVE-8629.patch
>
>
> When creating a hive session to run basic "alter table create partition"  
> queries, the session creation takes too long (more than 5 sec)  if the hive 
> execution engine is set to tez.
> Since the streaming clients dont care about Tez , it can explicitly override 
> the setting to mr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7111) Extend join transitivity PPD to non-column expressions

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187331#comment-14187331
 ] 

Ashutosh Chauhan commented on HIVE-7111:


I took a variant of test I added in HIVE-8598 and ran with this patch applied :
{code}
select * from src s1, src s2 where 1 = 2 and s1.value = s2.value;
{code}

My hope was that filter 1 = 2 will be pushed to TS after this patch and than 
constant folding will fold this to where false and than nullscan optimizer will 
optimize it. But seems like filter was not pushed past Join, as a result, null 
scan optimizer didnt kick in. 

Is expectation here to able to push such filters as well? This is more of a 
question if this patch is targeting such filter expressions. We need not to 
necessarily handle this within this jira, can be taken as follow-on.

> Extend join transitivity PPD to non-column expressions
> --
>
> Key: HIVE-7111
> URL: https://issues.apache.org/jira/browse/HIVE-7111
> Project: Hive
>  Issue Type: Task
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-7111.1.patch.txt, HIVE-7111.2.patch.txt, 
> HIVE-7111.2.patch.txt, HIVE-7111.3.patch.txt, HIVE-7111.4.patch.txt
>
>
> Join transitive in PPD only supports column expressions, but it's possible to 
> extend this to generic expressions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8636) CBO: split cbo_correctness test

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187343#comment-14187343
 ] 

Sergey Shelukhin commented on HIVE-8636:


[~jpullokkaran] [~ashutoshc] fyi

> CBO: split cbo_correctness test
> ---
>
> Key: HIVE-8636
> URL: https://issues.apache.org/jira/browse/HIVE-8636
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> CBO correctness test is extremely annoying - it runs forever, if anything 
> fails it's hard to debug due to the volume of logs from all the stuff, also 
> it doesn't run further so if multiple things fail they can only be discovered 
> one by one; also SORT_QUERY_RESULTS cannot be used, because some queries 
> presumably use sorting.
> It should be split into separate tests, the numbers in there now may be good 
> as boundaries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8636) CBO: split cbo_correctness test

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-8636:
--

 Summary: CBO: split cbo_correctness test
 Key: HIVE-8636
 URL: https://issues.apache.org/jira/browse/HIVE-8636
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


CBO correctness test is extremely annoying - it runs forever, if anything fails 
it's hard to debug due to the volume of logs from all the stuff, also it 
doesn't run further so if multiple things fail they can only be discovered one 
by one; also SORT_QUERY_RESULTS cannot be used, because some queries presumably 
use sorting.
It should be split into separate tests, the numbers in there now may be good as 
boundaries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8620) CBO: HIVE-8433 RowResolver check is too stringent

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8620:
---
Attachment: HIVE-8620.01.patch

updated patch. Order changed in some tests, select_same_col needs HIVE-8635 to 
wotk; that is temporarily included in this patch

> CBO: HIVE-8433 RowResolver check is too stringent
> -
>
> Key: HIVE-8620
> URL: https://issues.apache.org/jira/browse/HIVE-8620
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-8620.01.patch, HIVE-8620.patch
>
>
> Looks like it causes some valid queries to fail in CBO.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8394) HIVE-7803 doesn't handle Pig MultiQuery, can cause data-loss.

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187361#comment-14187361
 ] 

Sushanth Sowmyan commented on HIVE-8394:


I'll admit to the same distaste to using a Singleton to store state like this - 
we've had similar problems with HCatContext in the past, but I agree with your 
assertion that that seems to be the only real way to handle this issue.

Now, that said, the attached file HIVE-8394.1.patch includes a 
TaskCommitterContextRegistry.discardCleanupFor and does not ever call it. I 
assume that's what you mean with your comment on needing to add a finally 
block? Also, yes, the patch in the current form has an issue with multiple 
HCatStorers - do you have an updated patch with both these issues resolved?


> HIVE-7803 doesn't handle Pig MultiQuery, can cause data-loss.
> -
>
> Key: HIVE-8394
> URL: https://issues.apache.org/jira/browse/HIVE-8394
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0, 0.14.0, 0.13.1
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>Priority: Critical
> Attachments: HIVE-8394.1.patch
>
>
> We've found situations in production where Pig queries using {{HCatStorer}}, 
> dynamic partitioning and {{opt.multiquery=true}} that produce partitions in 
> the output table, but the corresponding directories have no data files (in 
> spite of Pig reporting non-zero records written to HDFS). I don't yet have a 
> distilled test-case for this.
> Here's the code from FileOutputCommitterContainer after HIVE-7803:
> {code:java|title=FileOutputCommitterContainer.java|borderStyle=dashed|titleBGColor=#F7D6C1|bgColor=#CE}
>   @Override
>   public void commitTask(TaskAttemptContext context) throws IOException {
> String jobInfoStr = 
> context.getConfiguration().get(FileRecordWriterContainer.DYN_JOBINFO);
> if (!dynamicPartitioningUsed) {
>  //See HCATALOG-499
>   FileOutputFormatContainer.setWorkOutputPath(context);
>   
> getBaseOutputCommitter().commitTask(HCatMapRedUtil.createTaskAttemptContext(context));
> } else if (jobInfoStr != null) {
>   ArrayList jobInfoList = 
> (ArrayList)HCatUtil.deserialize(jobInfoStr);
>   org.apache.hadoop.mapred.TaskAttemptContext currTaskContext = 
> HCatMapRedUtil.createTaskAttemptContext(context);
>   for (String jobStr : jobInfoList) {
>   OutputJobInfo localJobInfo = 
> (OutputJobInfo)HCatUtil.deserialize(jobStr);
>   FileOutputCommitter committer = new FileOutputCommitter(new 
> Path(localJobInfo.getLocation()), currTaskContext);
>   committer.commitTask(currTaskContext);
>   }
> }
>   }
> {code}
> The serialized jobInfoList can't be retrieved, and hence the commit never 
> completes. This is because Pig's MapReducePOStoreImpl deliberately clones 
> both the TaskAttemptContext and the contained Configuration instance, thus 
> separating the Configuration instances passed to 
> {{FileOutputCommitterContainer::commitTask()}} and 
> {{FileRecordWriterContainer::close()}}. Anything set by the RecordWriter is 
> unavailable to the Committer.
> One approach would have been to store state in the FileOutputFormatContainer. 
> But that won't work since this is constructed via reflection in 
> HCatOutputFormat (itself constructed via reflection by PigOutputFormat via 
> HCatStorer). There's no guarantee that the instance is preserved.
> My only recourse seems to be to use a Singleton to store shared state. I'm 
> loath to indulge in this brand of shenanigans. (Statics and container-reuse 
> in Tez might not play well together, for instance.) It might work if we're 
> careful about tearing down the singleton.
> Any other ideas? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8394) HIVE-7803 doesn't handle Pig MultiQuery, can cause data-loss.

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187363#comment-14187363
 ] 

Sushanth Sowmyan commented on HIVE-8394:


(Also, slight warning - I'll likely be applying your patch post HIVE-4329, and 
that might wind up requiring a rebase - the two should be compatible, I think, 
but I wanted to point it out if you think it makes a difference)

> HIVE-7803 doesn't handle Pig MultiQuery, can cause data-loss.
> -
>
> Key: HIVE-8394
> URL: https://issues.apache.org/jira/browse/HIVE-8394
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0, 0.14.0, 0.13.1
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>Priority: Critical
> Attachments: HIVE-8394.1.patch
>
>
> We've found situations in production where Pig queries using {{HCatStorer}}, 
> dynamic partitioning and {{opt.multiquery=true}} that produce partitions in 
> the output table, but the corresponding directories have no data files (in 
> spite of Pig reporting non-zero records written to HDFS). I don't yet have a 
> distilled test-case for this.
> Here's the code from FileOutputCommitterContainer after HIVE-7803:
> {code:java|title=FileOutputCommitterContainer.java|borderStyle=dashed|titleBGColor=#F7D6C1|bgColor=#CE}
>   @Override
>   public void commitTask(TaskAttemptContext context) throws IOException {
> String jobInfoStr = 
> context.getConfiguration().get(FileRecordWriterContainer.DYN_JOBINFO);
> if (!dynamicPartitioningUsed) {
>  //See HCATALOG-499
>   FileOutputFormatContainer.setWorkOutputPath(context);
>   
> getBaseOutputCommitter().commitTask(HCatMapRedUtil.createTaskAttemptContext(context));
> } else if (jobInfoStr != null) {
>   ArrayList jobInfoList = 
> (ArrayList)HCatUtil.deserialize(jobInfoStr);
>   org.apache.hadoop.mapred.TaskAttemptContext currTaskContext = 
> HCatMapRedUtil.createTaskAttemptContext(context);
>   for (String jobStr : jobInfoList) {
>   OutputJobInfo localJobInfo = 
> (OutputJobInfo)HCatUtil.deserialize(jobStr);
>   FileOutputCommitter committer = new FileOutputCommitter(new 
> Path(localJobInfo.getLocation()), currTaskContext);
>   committer.commitTask(currTaskContext);
>   }
> }
>   }
> {code}
> The serialized jobInfoList can't be retrieved, and hence the commit never 
> completes. This is because Pig's MapReducePOStoreImpl deliberately clones 
> both the TaskAttemptContext and the contained Configuration instance, thus 
> separating the Configuration instances passed to 
> {{FileOutputCommitterContainer::commitTask()}} and 
> {{FileRecordWriterContainer::close()}}. Anything set by the RecordWriter is 
> unavailable to the Committer.
> One approach would have been to store state in the FileOutputFormatContainer. 
> But that won't work since this is constructed via reflection in 
> HCatOutputFormat (itself constructed via reflection by PigOutputFormat via 
> HCatStorer). There's no guarantee that the instance is preserved.
> My only recourse seems to be to use a Singleton to store shared state. I'm 
> loath to indulge in this brand of shenanigans. (Statics and container-reuse 
> in Tez might not play well together, for instance.) It might work if we're 
> careful about tearing down the singleton.
> Any other ideas? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8614) Upgrade hive to use tez version 0.5.2-SNAPSHOT

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8614:
-
Attachment: HIVE-8614.4.patch

Fixes issue with the failing test.

> Upgrade hive to use tez version 0.5.2-SNAPSHOT
> --
>
> Key: HIVE-8614
> URL: https://issues.apache.org/jira/browse/HIVE-8614
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8614.1.patch, HIVE-8614.2.patch, HIVE-8614.3.patch, 
> HIVE-8614.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8631) Compressed transaction list cannot be parsed in job.xml

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187379#comment-14187379
 ] 

Hive QA commented on HIVE-8631:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12677635/HIVE-8631.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6579 tests executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1506/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1506/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1506/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12677635 - PreCommit-HIVE-TRUNK-Build

> Compressed transaction list cannot be parsed in job.xml
> ---
>
> Key: HIVE-8631
> URL: https://issues.apache.org/jira/browse/HIVE-8631
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8631.patch
>
>
> HIVE-8341 added code to compress the transaction list in the JobConf when it 
> reaches a certain size.  This breaks when the JobConf is converted to job.xml 
> and sent to MR.  The special characters are not correctly escaped.  
> The proposed fix is to back out the compression (not the changes to 
> ScriptOperator) for now and look later at compressing/encoding it in a way 
> that works with job.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8394) HIVE-7803 doesn't handle Pig MultiQuery, can cause data-loss.

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-8394:
---
Status: Open  (was: Patch Available)

Yes, actually, I do have to update the patch. Will do shortly.

> HIVE-7803 doesn't handle Pig MultiQuery, can cause data-loss.
> -
>
> Key: HIVE-8394
> URL: https://issues.apache.org/jira/browse/HIVE-8394
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.13.1, 0.12.0, 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>Priority: Critical
> Attachments: HIVE-8394.1.patch
>
>
> We've found situations in production where Pig queries using {{HCatStorer}}, 
> dynamic partitioning and {{opt.multiquery=true}} that produce partitions in 
> the output table, but the corresponding directories have no data files (in 
> spite of Pig reporting non-zero records written to HDFS). I don't yet have a 
> distilled test-case for this.
> Here's the code from FileOutputCommitterContainer after HIVE-7803:
> {code:java|title=FileOutputCommitterContainer.java|borderStyle=dashed|titleBGColor=#F7D6C1|bgColor=#CE}
>   @Override
>   public void commitTask(TaskAttemptContext context) throws IOException {
> String jobInfoStr = 
> context.getConfiguration().get(FileRecordWriterContainer.DYN_JOBINFO);
> if (!dynamicPartitioningUsed) {
>  //See HCATALOG-499
>   FileOutputFormatContainer.setWorkOutputPath(context);
>   
> getBaseOutputCommitter().commitTask(HCatMapRedUtil.createTaskAttemptContext(context));
> } else if (jobInfoStr != null) {
>   ArrayList jobInfoList = 
> (ArrayList)HCatUtil.deserialize(jobInfoStr);
>   org.apache.hadoop.mapred.TaskAttemptContext currTaskContext = 
> HCatMapRedUtil.createTaskAttemptContext(context);
>   for (String jobStr : jobInfoList) {
>   OutputJobInfo localJobInfo = 
> (OutputJobInfo)HCatUtil.deserialize(jobStr);
>   FileOutputCommitter committer = new FileOutputCommitter(new 
> Path(localJobInfo.getLocation()), currTaskContext);
>   committer.commitTask(currTaskContext);
>   }
> }
>   }
> {code}
> The serialized jobInfoList can't be retrieved, and hence the commit never 
> completes. This is because Pig's MapReducePOStoreImpl deliberately clones 
> both the TaskAttemptContext and the contained Configuration instance, thus 
> separating the Configuration instances passed to 
> {{FileOutputCommitterContainer::commitTask()}} and 
> {{FileRecordWriterContainer::close()}}. Anything set by the RecordWriter is 
> unavailable to the Committer.
> One approach would have been to store state in the FileOutputFormatContainer. 
> But that won't work since this is constructed via reflection in 
> HCatOutputFormat (itself constructed via reflection by PigOutputFormat via 
> HCatStorer). There's no guarantee that the instance is preserved.
> My only recourse seems to be to use a Singleton to store shared state. I'm 
> loath to indulge in this brand of shenanigans. (Statics and container-reuse 
> in Tez might not play well together, for instance.) It might work if we're 
> careful about tearing down the singleton.
> Any other ideas? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)