[jira] [Updated] (HIVE-14735) Build Infra: Spark artifacts download takes a long time

2016-09-11 Thread Vaibhav Gumashta (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-14735:

Description: 
In particular this command:
{{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz 
http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}}

  was:
In particular this command in build.sh:
{{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz 
http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}}


> Build Infra: Spark artifacts download takes a long time
> ---
>
> Key: HIVE-14735
> URL: https://issues.apache.org/jira/browse/HIVE-14735
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Reporter: Vaibhav Gumashta
>
> In particular this command:
> {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz 
> http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13878:

Status: Patch Available  (was: In Progress)

Avoid rescanning string/char/varchar to determine if there were escapes.

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, 
> HIVE-13878.09.patch, HIVE-13878.1.patch, HIVE-13878.2.patch, 
> HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13878:

Attachment: HIVE-13878.09.patch

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, 
> HIVE-13878.09.patch, HIVE-13878.1.patch, HIVE-13878.2.patch, 
> HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13878:

Status: In Progress  (was: Patch Available)

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, 
> HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-12812) Enable mapred.input.dir.recursive by default to support union with aggregate function

2016-09-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483002#comment-15483002
 ] 

Hive QA commented on HIVE-12812:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827953/HIVE-12812.patch

{color:green}SUCCESS:{color} +1 due to 54 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10546 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[root_dir_external_table]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[list_bucket_dml_2]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testTaskStatus
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1151/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1151/console
Test logs: 
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1151/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12827953 - PreCommit-HIVE-MASTER-Build

> Enable mapred.input.dir.recursive by default to support union with aggregate 
> function
> -
>
> Key: HIVE-12812
> URL: https://issues.apache.org/jira/browse/HIVE-12812
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-12812.patch, HIVE-12812.patch
>
>
> When union remove optimization is enabled, union query with aggregate 
> function writes its subquery intermediate results to subdirs which needs 
> mapred.input.dir.recursive to be enabled in order to be fetched. This 
> property is not defined by default in Hive and often ignored by user, which 
> causes the query failure and is hard to be debugged.
> So we need set mapred.input.dir.recursive to true whenever union remove 
> optimization is enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482932#comment-15482932
 ] 

Matt McCline commented on HIVE-13878:
-

Test failures are not related.

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, 
> HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

2016-09-11 Thread Rui Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482896#comment-15482896
 ] 

Rui Li commented on HIVE-14412:
---

The failure don't seem related.
[~xuefuz] could you take a look? I've put the v4 patch to RB. Thanks.

> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch, 
> HIVE-14412.3.patch, HIVE-14412.4.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by 
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or 
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-13589) beeline - support prompt for password with '-u' option

2016-09-11 Thread Ke Jia (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482879#comment-15482879
 ] 

Ke Jia commented on HIVE-13589:
---

Hi [~vihangk1], Thanks for your comments.  About this issue, I have some ideas 
and now am working on the patch. May I continue my work for this JIRA? The main 
idea is that we can add the "-- force" or some other "--" options in 
Beeline.java, which are defined in 
https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineCommands.
 You can share your thoughts once my patch is ready. Thanks again for your 
reviews and inputs.


> beeline - support prompt for password with '-u' option
> --
>
> Key: HIVE-13589
> URL: https://issues.apache.org/jira/browse/HIVE-13589
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Reporter: Thejas M Nair
>Assignee: Ke Jia
> Fix For: 2.2.0
>
> Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch, 
> HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch, HIVE-13589.6.patch
>
>
> Specifying connection string using commandline options in beeline is 
> convenient, as it gets saved in shell command history, and it is easy to 
> retrieve it from there.
> However, specifying the password in command prompt is not secure as it gets 
> displayed on screen and saved in the history.
> It should be possible to specify '-p' without an argument to make beeline 
> prompt for password.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-12812) Enable mapred.input.dir.recursive by default to support union with aggregate function

2016-09-11 Thread Chaoyu Tang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaoyu Tang updated HIVE-12812:
---
Attachment: HIVE-12812.patch

The failed tests seem not related to this patch. Reattach it to kick off 
another precommit test.

> Enable mapred.input.dir.recursive by default to support union with aggregate 
> function
> -
>
> Key: HIVE-12812
> URL: https://issues.apache.org/jira/browse/HIVE-12812
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-12812.patch, HIVE-12812.patch
>
>
> When union remove optimization is enabled, union query with aggregate 
> function writes its subquery intermediate results to subdirs which needs 
> mapred.input.dir.recursive to be enabled in order to be fetched. This 
> property is not defined by default in Hive and often ignored by user, which 
> causes the query failure and is hard to be debugged.
> So we need set mapred.input.dir.recursive to true whenever union remove 
> optimization is enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically

2016-09-11 Thread JIRA

[ 
https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482800#comment-15482800
 ] 

Sergio Peña commented on HIVE-14734:


Btw, the idea is to call {{jenkins-execute-build.sh}} from the 
{{PreCommit-HIVE-Build}} job. This job should run tests for any branch, and the 
branch name should appear on the build jenkins list.

For instance:
{noformat}
​150 Sep 12, 2016 8:18 PM
HIVE-13878  branch-2.1

​14​9 Sep 11, 2016 8:18 PM
HIVE-13878  master
{noformat}

> Allow jenkins ptest job to execute tests on branch dynamically
> --
>
> Key: HIVE-14734
> URL: https://issues.apache.org/jira/browse/HIVE-14734
> Project: Hive
>  Issue Type: Task
>  Components: Hive, Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14734
>
>
> NO PRECOMMIT TESTS
> Currently, to execute tests on a new branch, a manual process must be done:
> 1. Create a new Jenkins job with the new branch name
> 2. Create a patch to jenkins-submit-build.sh with the new branch
> 3. Create a profile properties file on the ptest master with the new branch
> This jira will attempt to automate steps 1 and 2 by detecting the branch 
> profile from a patch to test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482797#comment-15482797
 ] 

Hive QA commented on HIVE-13878:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827941/HIVE-13878.08.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10546 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1150/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1150/console
Test logs: 
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1150/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12827941 - PreCommit-HIVE-MASTER-Build

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, 
> HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically

2016-09-11 Thread JIRA

 [ 
https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-14734:
---
Status: Patch Available  (was: Open)

> Allow jenkins ptest job to execute tests on branch dynamically
> --
>
> Key: HIVE-14734
> URL: https://issues.apache.org/jira/browse/HIVE-14734
> Project: Hive
>  Issue Type: Task
>  Components: Hive, Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14734
>
>
> NO PRECOMMIT TESTS
> Currently, to execute tests on a new branch, a manual process must be done:
> 1. Create a new Jenkins job with the new branch name
> 2. Create a patch to jenkins-submit-build.sh with the new branch
> 3. Create a profile properties file on the ptest master with the new branch
> This jira will attempt to automate steps 1 and 2 by detecting the branch 
> profile from a patch to test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically

2016-09-11 Thread JIRA

[ 
https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482790#comment-15482790
 ] 

Sergio Peña commented on HIVE-14734:


[~sseth] [~prasanth_j] Can you help me review this patch?

The patch will detect the branch profile from the file attached to the JIRA, 
and then will call ptest with such profile. I also made some refactoring on the 
'jenkins-execute-build.sh' script.

> Allow jenkins ptest job to execute tests on branch dynamically
> --
>
> Key: HIVE-14734
> URL: https://issues.apache.org/jira/browse/HIVE-14734
> Project: Hive
>  Issue Type: Task
>  Components: Hive, Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14734
>
>
> NO PRECOMMIT TESTS
> Currently, to execute tests on a new branch, a manual process must be done:
> 1. Create a new Jenkins job with the new branch name
> 2. Create a patch to jenkins-submit-build.sh with the new branch
> 3. Create a profile properties file on the ptest master with the new branch
> This jira will attempt to automate steps 1 and 2 by detecting the branch 
> profile from a patch to test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically

2016-09-11 Thread JIRA

 [ 
https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-14734:
---
Description: 
NO PRECOMMIT TESTS

Currently, to execute tests on a new branch, a manual process must be done:

1. Create a new Jenkins job with the new branch name
2. Create a patch to jenkins-submit-build.sh with the new branch
3. Create a profile properties file on the ptest master with the new branch

This jira will attempt to automate steps 1 and 2 by detecting the branch 
profile from a patch to test.

  was:
Currently, to execute tests on a new branch, a manual process must be done:

1. Create a new Jenkins job with the new branch name
2. Create a patch to jenkins-submit-build.sh with the new branch
3. Create a profile properties file on the ptest master with the new branch

This jira will attempt to automate steps 1 and 2 by detecting the branch 
profile from a patch to test.


> Allow jenkins ptest job to execute tests on branch dynamically
> --
>
> Key: HIVE-14734
> URL: https://issues.apache.org/jira/browse/HIVE-14734
> Project: Hive
>  Issue Type: Task
>  Components: Hive, Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14734
>
>
> NO PRECOMMIT TESTS
> Currently, to execute tests on a new branch, a manual process must be done:
> 1. Create a new Jenkins job with the new branch name
> 2. Create a patch to jenkins-submit-build.sh with the new branch
> 3. Create a profile properties file on the ptest master with the new branch
> This jira will attempt to automate steps 1 and 2 by detecting the branch 
> profile from a patch to test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically

2016-09-11 Thread JIRA

 [ 
https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-14734:
---
Attachment: HIVE-14734

> Allow jenkins ptest job to execute tests on branch dynamically
> --
>
> Key: HIVE-14734
> URL: https://issues.apache.org/jira/browse/HIVE-14734
> Project: Hive
>  Issue Type: Task
>  Components: Hive, Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14734
>
>
> NO PRECOMMIT TESTS
> Currently, to execute tests on a new branch, a manual process must be done:
> 1. Create a new Jenkins job with the new branch name
> 2. Create a patch to jenkins-submit-build.sh with the new branch
> 3. Create a profile properties file on the ptest master with the new branch
> This jira will attempt to automate steps 1 and 2 by detecting the branch 
> profile from a patch to test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically

2016-09-11 Thread JIRA

 [ 
https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-14734:
---
Description: 
Currently, to execute tests on a new branch, a manual process must be done:

1. Create a new Jenkins job with the new branch name
2. Create a patch to jenkins-submit-build.sh with the new branch
3. Create a profile properties file on the ptest master with the new branch

This jira will attempt to automate steps 1 and 2 by detecting the branch 
profile from a patch to test.

  was:
Currently, to execute tests on a new branch, a manual process must be done to:

1. Create a new Jenkins job with the new branch name
2. Create a patch to jenkins-submit-build.sh with the new branch
3. Create a profile properties file on the ptest master with the new branch

This jira will attempt to automate steps 1 and 2 for any branch that is 
submitted for testing.


> Allow jenkins ptest job to execute tests on branch dynamically
> --
>
> Key: HIVE-14734
> URL: https://issues.apache.org/jira/browse/HIVE-14734
> Project: Hive
>  Issue Type: Task
>  Components: Hive, Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>
> Currently, to execute tests on a new branch, a manual process must be done:
> 1. Create a new Jenkins job with the new branch name
> 2. Create a patch to jenkins-submit-build.sh with the new branch
> 3. Create a profile properties file on the ptest master with the new branch
> This jira will attempt to automate steps 1 and 2 by detecting the branch 
> profile from a patch to test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-14626) Support Trash in Truncate Table

2016-09-11 Thread Chaoyu Tang (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482740#comment-15482740
 ] 

Chaoyu Tang commented on HIVE-14626:


Thanks, [~leftylev]. The patch is to enhance the Truncate with Trash support 
and there is not backward compatibility, so I do not think we need the 
explanation to the behavior before this. 

> Support Trash in Truncate Table
> ---
>
> Key: HIVE-14626
> URL: https://issues.apache.org/jira/browse/HIVE-14626
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14626.1.patch, HIVE-14626.patch
>
>
> Currently Truncate Table (or Partition) is implemented using 
> FileSystem.delete and then recreate the directory, so
> 1. it does not support HDFS Trash
> 2. if the table/partition directory is initially encryption protected, after 
> being deleted and recreated, it is no more protected.
> The new implementation is to clean the contents of directory using 
> multi-threaded trashFiles. If Trash is enabled and has a lower encryption 
> level than the data directory, the files under it will be deleted. Otherwise, 
> they will be Trashed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-14727) llap-server may case file descriptor leak in BuddyAllocator class

2016-09-11 Thread Yechao Chen (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482720#comment-15482720
 ] 

Yechao Chen commented on HIVE-14727:


There is no original failrue,I just read the llap-server code ,find this may 
case this suitation

> llap-server may case file descriptor leak in BuddyAllocator class
> -
>
> Key: HIVE-14727
> URL: https://issues.apache.org/jira/browse/HIVE-14727
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.2.0
>Reporter: Yechao Chen
>Assignee: Yechao Chen
> Fix For: 2.2.0
>
> Attachments: HIVE-14727.patch
>
>
> llap-server,the method preallocate(int) of  BuddyAllocator may case file 
> descriptor leak when FileChannel map allocate memory error.
> the code:
> //here if failed
>  ByteBuffer rwbuf = rwf.getChannel().map(MapMode.READ_WRITE, 0, 
> arenaSize);
> // A mapping, once established, is not dependent upon the file 
> channel that was used to
> // create it. delete file and hold onto the map
>//can not close() and delete file
> rwf.close();
> rf.delete();



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13878:

Status: Patch Available  (was: In Progress)

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, 
> HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13878:

Status: In Progress  (was: Patch Available)

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, 
> HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13878:

Attachment: HIVE-13878.08.patch

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, 
> HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482557#comment-15482557
 ] 

Matt McCline commented on HIVE-13878:
-

Test failure TestVectorSerDeRow *is* related.

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.1.patch, 
> HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482456#comment-15482456
 ] 

Hive QA commented on HIVE-13878:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827936/HIVE-13878.07.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10546 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.ql.exec.vector.TestVectorSerDeRow.testVectorSerDeRow
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1149/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1149/console
Test logs: 
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1149/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12827936 - PreCommit-HIVE-MASTER-Build

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.1.patch, 
> HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13878:

Status: Patch Available  (was: In Progress)

Optimize LazySimpleDeserializeRead.parse

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.1.patch, 
> HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13878:

Status: In Progress  (was: Patch Available)

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.1.patch, 
> HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13878:

Attachment: HIVE-13878.07.patch

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.1.patch, 
> HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"

2016-09-11 Thread Fan Yunbo (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481681#comment-15481681
 ] 

Fan Yunbo commented on HIVE-14686:
--

thanks for you review.

> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS 
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch, 
> HIVE-14686.3.patch, HIVE-14686.4.patch, HIVE-14686.5.patch
>
>
> See the query: 
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will 
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by 
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in 
> SemanticAnalyzer.java will return null and won't set the correct command type 
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
>   try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
>   return null;
> }
>   } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " + 
> e.getMessage(), e);
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Gopal V (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481172#comment-15481172
 ] 

Gopal V commented on HIVE-13878:


Thanks [~mmccline], I've added this to my build queue.

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.1.patch, HIVE-13878.2.patch, 
> HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Matt McCline (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481169#comment-15481169
 ] 

Matt McCline commented on HIVE-13878:
-

Test failures are unrelated.

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.1.patch, HIVE-13878.2.patch, 
> HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-09-11 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481161#comment-15481161
 ] 

Hive QA commented on HIVE-13878:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827916/HIVE-13878.06.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10544 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1148/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1148/console
Test logs: 
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1148/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12827916 - PreCommit-HIVE-MASTER-Build

> Vectorization: Column pruning for Text vectorization
> 
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, 
> HIVE-13878.06.patch, HIVE-13878.1.patch, HIVE-13878.2.patch, 
> HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe 
> settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-14626) Support Trash in Truncate Table

2016-09-11 Thread Lefty Leverenz (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481159#comment-15481159
 ] 

Lefty Leverenz commented on HIVE-14626:
---

Thanks for the doc and the jira for encryption, [~ctang.ma].  I had forgotten 
about HIVE-8065, which still needs to be documented in the wiki.

I added version information in the Truncate Table section, with a link to this 
issue.  Should we also explain what the behavior was before this patch?

> Support Trash in Truncate Table
> ---
>
> Key: HIVE-14626
> URL: https://issues.apache.org/jira/browse/HIVE-14626
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14626.1.patch, HIVE-14626.patch
>
>
> Currently Truncate Table (or Partition) is implemented using 
> FileSystem.delete and then recreate the directory, so
> 1. it does not support HDFS Trash
> 2. if the table/partition directory is initially encryption protected, after 
> being deleted and recreated, it is no more protected.
> The new implementation is to clean the contents of directory using 
> multi-threaded trashFiles. If Trash is enabled and has a lower encryption 
> level than the data directory, the files under it will be deleted. Otherwise, 
> they will be Trashed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8065) Support HDFS encryption functionality on Hive

2016-09-11 Thread Lefty Leverenz (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481156#comment-15481156
 ] 

Lefty Leverenz commented on HIVE-8065:
--

Nudge:  [~spena], could you please document HDFS encryption in the wiki?  Or if 
you don't have time, could you suggest someone else to do it?  Thanks.

> Support HDFS encryption functionality on Hive
> -
>
> Key: HIVE-8065
> URL: https://issues.apache.org/jira/browse/HIVE-8065
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.13.1
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>  Labels: TODOC15
>
> The new encryption support on HDFS makes Hive incompatible and unusable when 
> this feature is used.
> HDFS encryption is designed so that an user can configure different 
> encryption zones (or directories) for multi-tenant environments. An 
> encryption zone has an exclusive encryption key, such as AES-128 or AES-256. 
> Because of security compliance, the HDFS does not allow to move/rename files 
> between encryption zones. Renames are allowed only inside the same encryption 
> zone. A copy is allowed between encryption zones.
> See HDFS-6134 for more details about HDFS encryption design.
> Hive currently uses a scratch directory (like /tmp/$user/$random). This 
> scratch directory is used for the output of intermediate data (between MR 
> jobs) and for the final output of the hive query which is later moved to the 
> table directory location.
> If Hive tables are in different encryption zones than the scratch directory, 
> then Hive won't be able to renames those files/directories, and it will make 
> Hive unusable.
> To handle this problem, we can change the scratch directory of the 
> query/statement to be inside the same encryption zone of the table directory 
> location. This way, the renaming process will be successful. 
> Also, for statements that move files between encryption zones (i.e. LOAD 
> DATA), a copy may be executed instead of a rename. This will cause an 
> overhead when copying large data files, but it won't break the encryption on 
> Hive.
> Another security thing to consider is when using joins selects. If Hive joins 
> different tables with different encryption key strengths, then the results of 
> the select might break the security compliance of the tables. Let's say two 
> tables with 128 bits and 256 bits encryption are joined, then the temporary 
> results might be stored in the 128 bits encryption zone. This will conflict 
> with the table encrypted with 256 bits temporary.
> To fix this, Hive should be able to select the scratch directory that is more 
> secured/encrypted in order to save the intermediate data temporary with no 
> compliance issues.
> For instance:
> {noformat}
> SELECT * FROM table-aes128 t1 JOIN table-aes256 t2 WHERE t1.id == t2.id;
> {noformat}
> - This should use a scratch directory (or staging directory) inside the 
> table-aes256 table location.
> {noformat}
> INSERT OVERWRITE TABLE table-unencrypted SELECT * FROM table-aes1;
> {noformat}
> - This should use a scratch directory inside the table-aes1 location.
> {noformat}
> FROM table-unencrypted
> INSERT OVERWRITE TABLE table-aes128 SELECT id, name
> INSERT OVERWRITE TABLE table-aes256 SELECT id, name
> {noformat}
> - This should use a scratch directory on each of the tables locations.
> - The first SELECT will have its scratch directory on table-aes128 directory.
> - The second SELECT will have its scratch directory on table-aes256 directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)